Sensing data fusion method and device, controller, vehicle and product

Abstract

The embodiment of the invention relates to a sensing data fusion method and device, a controller, a vehicle and a product. The method includes forming a global grid map including a plurality of grid cells based on sensed data from a sensor on a vehicle. The method further includes identifying, in the global grid map, a target grid cell where a target sensing point of the target object is located and a predetermined number of neighbor grid cells adjacent to the target grid cell between the vehicle and the target object. The method further includes associating a target sensing point of the target object with a corresponding sensing point satisfying a predetermined condition in the target grid unit and the neighbor grid unit, where the target sensing point and the corresponding sensing point are from at least one of different sensors or different time instances. In this way, the computational efficiency in the data association process can be significantly improved to improve the performance of data fusion.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure generally relate to the field of driving, and in particular, to a method and apparatus for fusing sensing data, a controller, a vehicle and a product. BACKGROUND

[0002] Data fusion is a process of combining various different data and information, such as from different sources, different formats, etc., to provide more comprehensive, accurate and reliable data support to meet the needs of specific applications or tasks. In addition, by performing data fusion, redundant and repetitive data can be eliminated, and the quality and consistency of data can be improved.

[0003] The application of such data fusion technology in the field of driving can improve the perception ability of vehicles to the surrounding environment, which can help better understand the driving environment, so as to make safer and more effective decisions and take appropriate actions. For example, sensing data fusion technology promotes the performance improvement of various driving functionalities of vehicle driving systems, such as advanced driving assistance systems (ADAS) and autonomous driving (AD) systems, such as obstacle detection, road condition perception, etc. SUMMARY

[0004] Embodiments of the present disclosure provide a method and apparatus for fusing sensing data, a controller, a vehicle and a product.

[0005] According to a first aspect of the present disclosure, a method for fusing sensing data is provided. The method comprises forming, based on sensing data from sensors on a vehicle, a global grid map comprising a plurality of grid cells. The method further comprises identifying, in the global grid map, a target grid cell in which a target sensing point of a target object is located, and a predetermined number of neighbor grid cells adjacent to the target grid cell, between the vehicle and the target object. The method further comprises associating, in the target grid cell and the neighbor grid cells, the target sensing point of the target object with respective sensing points satisfying a predetermined condition, wherein the target sensing point and the respective sensing points are from at least one of: different sensors, or different time instances.

[0006] According to a second aspect of the disclosure, there is provided an apparatus for fusing sensing data. The apparatus comprises a coordinate forming module configured to form, based on sensing data from sensors on a vehicle, a global grid map comprising a plurality of grid cells. The apparatus further comprises a grid identification module configured to identify, in the global grid map, a target grid cell in which a target sensing point of a target object is located, and a predetermined number of neighbor grid cells adjacent to the target grid cell, between the vehicle and the target object. The apparatus further comprises a sensing point association module configured to associate, in the target grid cell and the neighbor grid cells, the target sensing point of the target object with respective sensing points satisfying a predetermined condition, wherein the target sensing point and the respective sensing points are from at least one of: different sensors, or different time instances.

[0007] According to a third aspect of the disclosure, there is provided a controller. The controller comprises at least one processor. The electronic device further comprises a memory coupled to the at least one processor and having instructions stored thereon that, when executed by the at least one processor, cause the device to perform the steps of the method in the first aspect of the disclosure.

[0008] According to a fourth aspect of the disclosure, there is provided a vehicle. The vehicle comprises the electronic device in the third aspect of the disclosure.

[0009] According to a fifth aspect of the disclosure, there is provided a computer program product tangibly stored on a computer readable medium and comprising computer executable instructions that, when executed by a processor of a computer, cause the computer to perform the steps of the method in the first aspect of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other objects, features and advantages of the disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which similar reference characters refer to similar elements throughout the several views. In the drawings:

[0011] Figure 1 FIG. 1 illustrates a schematic diagram of an example environment in which methods and / or devices according to embodiments of the disclosure can be implemented;

[0012] Figure 2 FIG. 2 illustrates a flowchart of a method for fusing sensing data according to embodiments of the disclosure;

[0013] Figure 3 FIG. 3 illustrates an example of a sensing point association process based on a global grid map according to embodiments of the disclosure;

[0014] Figure 4FIG. 1 illustrates an example flow of a data association process according to embodiments of the present disclosure;

[0015] Figure 5 An example implementation of the above-described data association process is illustrated schematically in FIG. 2;

[0016] Figure 6 FIG. 3 illustrates a diagram of a virtual projection of a ray according to embodiments of the present disclosure;

[0017] Figure 7A FIG. 4 illustrates an example effect of the virtual projection of a ray according to embodiments of the present disclosure;

[0018] Figure 7B is Figure 7A a close-up view of;

[0019] Figure 8 FIG. 5 illustrates an example flow of a data fusion process according to embodiments of the present disclosure;

[0020] Figure 9 FIG. 6 shows a schematic diagram of an apparatus for fusing sensing data according to embodiments of the present disclosure; and

[0021] Figure 10 FIG. 7 illustrates a schematic block diagram of an example device suitable for use in implementing embodiments of the present disclosure.

[0022] In the various drawings, like or corresponding elements are denoted by like or corresponding reference numerals. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While certain embodiments of the present disclosure will be shown in the drawings, it should be understood that the present disclosure can be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present disclosure to those skilled in the art. It should be understood that the drawings and the embodiments are only for illustrative purposes and not intended to limit the scope of protection of the present disclosure.

[0024] In the description of embodiments of the present disclosure, the term "including" and its variants are to be construed as open-ended, meaning that "including, but not limited to." The term "based on" is to be construed as "based at least in part on." The term "one embodiment" or "an embodiment" is to be construed as "at least one embodiment." The term "first," "second," and the like can refer to different or identical objects, unless explicitly indicated otherwise.

[0025] Data fusion is a data science technique that plays a key role in numerous use cases, such as in the fields of security, healthcare, transportation, and the like. As described above, it combines, correlates, and integrates various different data (such as from different sources, different formats, and the like) to provide more comprehensive, accurate, and reliable data support to meet the needs of specific applications or tasks. In addition, it can eliminate redundant and repetitive data, improving the quality and consistency of data.

[0026] In the field of driving, data fusion techniques help improve the performance of various driving functionalities of vehicle driving systems, such as advanced driver assistance systems (ADAS) and autonomous driving (AD) systems, for example, such as automatic parking. In the automatic parking functionality, correlating multi-frame, multi-source sensor sensing data is a key prerequisite for sensing data fusion and construction of drivable space around the vehicle. The sensing results of the surrounding environment (e.g., sensing results of obstacles around the vehicle) are usually characterized as a set of discrete points.

[0027] In related solutions, correlation is established for these points by calculating the Euclidean distance between each two points, which facilitates the determination of the nearest neighbor of each point. However, while the nearest neighbor correlation strategy based on Euclidean distance is intuitive, its computational efficiency is directly affected by the size of the point set, resulting in a high level of complexity (i.e., O(n^2)). It should be noted that in ADAS, for example, a large number of lidars and cameras are often used to obtain more abundant sensing information.

[0028] In addition, the surfaces of obstacles usually form a continuous and dense three-dimensional (3D) surface. While the sensing results from different sources, different formats, and the like are correlated by the nearest neighbor correlation, their respective ground truths can not correspond to the same point on the 3D surface of the obstacle. Such inconsistency causes data that should not be correlated together to be mis-correlated, which not only leads to the degradation of the performance of the corresponding functionality, but also incurs additional computational resources and the like to correct or compensate for the impact of such inconsistency of data.

[0029] To address at least the above concerns and others, embodiments of the present disclosure provide a scheme for fusing sensing data. The scheme for fusing sensing data according to embodiments of the present disclosure includes forming, based on sensing data from sensors on a vehicle, a global grid map including a plurality of grid cells. The scheme further includes identifying, in the global grid map, a target grid cell in which a target sensing point of a target object between the vehicle and the target object is located, and a predetermined number of neighbor grid cells adjacent to the target grid cell. The scheme further includes associating, in the target grid cell and the neighbor grid cells, the target sensing point of the target object with respective sensing points that satisfy a predetermined condition, where the target sensing point and the respective sensing points are from at least one of: different sensors, or different time instances.

[0030] In this way, based on the established grid map, sensing data from different sensors, different time instances, or both, can be associated together, provided that the sensing points fall within the same grid cell or adjacent grid cells, they are considered to be associated with each other, without the need to perform pairwise Euclidean distance calculation. In this way, the computation process of data association can be greatly simplified, and the computation efficiency of data association can be significantly improved, for improving the performance of data fusion. Moreover, even if there is certain noise or error in the sensing data to be associated, as long as these errors do not cause the sensing points to be wrongly mapped into completely irrelevant grid cells, these sensing points can still be correctly associated.

[0031] The basic principles and several example implementations of the present disclosure will be explained below with reference to Figures 1 to 10 It should be understood that these example embodiments are given for the purpose of better illustrating and thus enabling those skilled in the art to better implement embodiments of the present disclosure, but are not intended to limit the scope of the present disclosure in any way.

[0032] Figure 1 A schematic diagram of an example environment 100 in which methods and / or procedures according to embodiments of the present disclosure can be implemented is illustrated. For ease of understanding, the sensing data fusion in a traffic scenario will be exemplarily described in the following. It should be understood that this is not limiting, and the methods according to embodiments of the present disclosure can also be used for other different applications or tasks, such as security scenarios, medical scenarios, etc.

[0033] As shown in Figure 1 The example environment 100 can include a target object 101, a vehicle 110, sensing data 120, a computing device 130, and a storage device 140, and these components can be coupled to each other for interaction, as shown in Figure 1As shown in the illustration. It should be understood that a limited set of components is shown in the example environment 100 for implementing embodiments of the present disclosure for purposes of ease of understanding and illustration only, and embodiments of the present disclosure are not limited thereto. For example, the example environment 100 may also include a display (not shown) configured to display sensed data 120, various functional results, etc.

[0034] According to embodiments of this disclosure, vehicle 110 can be any type of motorized or non-motorized vehicle capable of carrying people and / or goods and being movable. Vehicle 110 typically includes one or more wheels, one or more seats, one or more load-bearing structures (such as a cabin, compartment, etc.), one or more power systems (such as an engine, electric motor, etc.), one or more control components (such as a steering wheel, accelerator pedal, etc.), and one or more safety components (such as seat belts, airbags, etc.), etc.

[0035] like Figure 1 As shown, vehicle 110 is illustrated as an automobile. However, this is merely exemplary and not limiting. By way of example and not limitation, vehicle 110 may include buses, trucks, motorcycles, etc. Furthermore, vehicle 110 may be based on fossil fuels, clean energy, or a combination thereof. Fossil fuel-based vehicles primarily refer to vehicles that use fossil fuels such as oil and natural gas as their power source, such as traditional gasoline vehicles and diesel vehicles. Clean energy-based vehicles refer to vehicles that use clean energy as their power source, such as electric vehicles, hydrogen fuel cell vehicles, and solar-powered vehicles.

[0036] According to embodiments of this disclosure, in order to achieve various vehicle functionalities (e.g., the numerous functionalities of ADAS and AD systems), vehicle 110 may include multiple sensors (not shown) configured to sense the driving environment in which vehicle 110 is located when it is driving or parked, the position, state, and behavior of objects and persons in that driving environment (such as other vehicles and pedestrians). By way of example and not limitation, the sensors on vehicle 110 may include ultrasonic radar, cameras, and lidar, etc. In some embodiments, the data type of the sensing data 120 captured by these sensors may include images, videos, distance data, speed data, etc.

[0037] like Figure 1 As shown, in the driving environment of vehicle 110, target object 101 may be located within a predetermined range of vehicle 110. In driving functionalities such as automatic parking, the construction of the drivable space around vehicle 110 requires accurate location information of target object 101, such as obstacles, for avoidance of target object 101. By way of example and not limitation, target object 101 may include other vehicles, pedestrians, terrain obstacles, etc., within the predetermined range of vehicle 110. It should be noted that...Figure 1 In the example, only one target object (i.e., the target object 101) is shown, but embodiments of the present disclosure can also be applied to an environment including more target objects.

[0038] According to embodiments of the present disclosure, the sensing of the driving environment, the target object 101 in the driving environment, via sensors on the vehicle 110 can generate the sensing data 120. The sensing data 120 can be stored in the storage device 140 and accessed by the computing device 130. As mentioned above, the sensors on the vehicle 110 can include, for example, ultrasonic radar, camera, and lidar, etc., and thus the sensing data 120 can include ultrasonic radar sensing data, camera sensing data, lidar sensing data, etc. Furthermore, the data types of the sensing data 120 captured via these sensors can include images, videos, distance data, speed data, etc. In some embodiments, a series of pre-processing operations can be performed on different types of sensing data from different sensors so that they can be uniformly acquired, further processed, etc. in subsequent processes. Examples of these processing can include, but are not limited to, format conversion, standardization, cropping, scaling, flipping rotation, brightness / contrast adjustment, etc.

[0039] According to embodiments of the present disclosure, the computing device 130 can include an in-vehicle computing device (which means that the computing device 130 can be located within the vehicle 100), an off-vehicle computing device, or a combination thereof, and can have computing capability suitable for performing the fusion of the sensing data. For example, the computing device 130 can be configured to perform the fusion of the sensing data for the target object 101 for acquiring accurate location information of the target object 101. During the execution of the sensing data fusion evaluation, the computing device 130 can process (such as, unify, integrate, etc.) the sensing data 120 from the sensors on the vehicle 100 and store the processed sensing data 120 in the storage device 140. It should be understood that the computing device 130 can be configured to perform the fusion of the sensing data for the target object 101 in real time, or in near real time, or in a batch manner, or in a combination thereof. Figure 1 In the example, only one computing device is shown, but this is only illustrative and non-limiting, and more numbers of computing devices can also exist in the example environment 100. In the following, the corresponding operations of the computing device 130 will be described in further detail.

[0040] By way of example and not limitation, the computing device 130 can include, but is not limited to, an in-vehicle computer, a personal computer, a laptop computer, a server computer, a mobile device (such as, a smartphone, a tablet computer, etc.), a wearable electronic device, a multimedia player, a personal digital assistant (PDA), a smart home device, a consumer electronic product, or a distributed computing environment including any one or more of the above devices, etc.

[0041] According to embodiments of the present disclosure, the storage device 140 can be configured to store the sensing data 120, parameters to be used for performing sensing data fusion on the computing device 130, data association results, etc. It should be appreciated that the storage device 140 is shown as one storage device in Figure 1 the example environment 100, but this is merely illustrative and not limiting, and there can be a larger number of storage devices in the example environment 100.

[0042] By way of example and not limitation, the storage device 140 can include, but is not limited to, a local storage device, a remote storage device, or a combination thereof. In some embodiments, a plurality of storage devices in the storage device 140 can include, but are not limited to, a hard disk drive (HDD), a solid state drive (SSD), a hybrid hard disk drive (SSHD), etc., and some of the plurality of storage devices can be arranged locally while others can be arranged remotely, e.g., coupled together via a wire or a network, etc.

[0043] The example environment 100 in which methods and / or processes according to embodiments of the present disclosure can be implemented is described above in connection with Figure 1 The flowchart of a method 200 for fusing sensing data according to embodiments of the present disclosure is described below in connection with Figure 2 With this method 200, based on the established global grid map, sensing data from different sensors, different time instances, or both, can be associated together. This method 200 can simplify and improve the efficiency of data association computation to optimize the data fusion performance. Even if the sensing data has noise or errors, as long as the sensing points are not mapped into completely irrelevant grid cells, these points can still be correctly associated.

[0044] Figure 2 The flowchart of a method 200 for fusing sensing data according to embodiments of the present disclosure is illustrated. At block 210, based on sensing data from sensors on a vehicle, a global grid map is formed, which includes a plurality of grid cells. Each of the grid cells can have a same predetermined size. According to embodiments of the present disclosure, based on the sensing data, a global grid map is established, which refers to a reference map that has a plurality of grid cells evenly distributed on a coordinate system of the vehicle itself as a reference point. With the global grid map, the relative position, pose, etc., between a target object in the environment around the vehicle and the vehicle itself can be described.

[0045] At block 220, in the global grid map, a target grid cell in which a target sensing point of the target object is located, and a predetermined number of neighbor grid cells adjacent to the target grid cell are identified between the vehicle and the target object. The sensing data for the target object is converted into a set of sensing points in the global grid map, the points to be associated in one association operation are referred to as target sensing points, which are to be associated with corresponding sensing points sensed by the same sensor at different time instances (or frames), or are to be associated with corresponding sensing points sensed by different sensors at the same time instance, or both. Here, the value of the predetermined number can be matched with the accuracy requirement, for example, inversely related. That is, the higher the accuracy requirement for detecting the target obstacle, the smaller the value of the predetermined number can be set, and the lower the accuracy requirement for detecting the target obstacle, the larger the value of the predetermined number can be set.

[0046] At block 230, in the identified target grid cell and neighbor grid cells, the target sensing point of the target object is associated with a corresponding sensing point that satisfies a predetermined condition, wherein the target sensing point and the corresponding sensing point are from at least one of different sensors or different time instances. According to embodiments of the present disclosure, the target sensing point is associated with the corresponding sensing point located within the same grid cell or adjacent grid cells that satisfies the predetermined condition, rather than calculating the Euclidean distance between each two points. In some embodiments, such a predetermined condition can include that in the identified target grid cell or neighbor grid cells, the corresponding sensing point and the target sensing point correspond to the same true value point of the target object, and the distance between the corresponding sensing point and the target sensing point is not greater than a predetermined distance threshold. It should be understood that the predetermined condition according to embodiments of the present disclosure is not limited thereto, for example, can also include that the corresponding sensing point and the target sensing point correspond to a similar true value point of the target object, etc.

[0047] Through the method for fusing sensing data according to embodiments of the present disclosure, based on the established grid map, the sensing data from different sensors, different time instances, or both can be associated together, if the sensing points fall within the same grid cell or adjacent grid cells, they are considered to be associated with each other, so that the pairwise Euclidean distance calculation is not required. In this way, the calculation process of data association can be greatly simplified, and the calculation efficiency of data association can be significantly improved, in order to improve the performance of data fusion. In addition, even if there is a certain noise or error in the sensing data to be associated, as long as these errors do not cause the sensing points to be incorrectly mapped into completely unrelated grid cells, these sensing points can still be correctly associated.

[0048] Figure 3 FIG. 1 illustrates an example of a sensing point association process 100 based on a global grid map according to embodiments of the present disclosure. As shown in FIG. 1, the sensing point association process 100 includes a vehicle 110, a target object 120, a first sensor 130, a second sensor 140, and a global grid map 150. Figure 3The coordinate system of the global grid map includes a horizontal axis (e.g., as shown in FIG. 3 as the x-axis) and a vertical axis (e.g., as shown in FIG. 3 as the y-axis) with respect to the current or some point in time position of the ego vehicle 310 (e.g., with the center of the rear axle of the vehicle 310 as the coordinate origin). Further, a plurality of grid cells can be distributed on the coordinate system of the global grid map, which can have the same predetermined size. It should be understood that the global grid map described herein is to cover and identify objects and objects in the environment where the vehicle is located as much as possible, and is not intended to limit the specific size of the grid map. In other words, the size of the grid map can be selected depending on the specific use needs, for example, and a local grid map is also possible. Figure 3 The coordinate system of the global grid map includes a horizontal axis (e.g., as shown in FIG. 3 as the x-axis) and a vertical axis (e.g., as shown in FIG. 3 as the y-axis) with respect to the current or some point in time position of the ego vehicle 310 (e.g., with the center of the rear axle of the vehicle 310 as the coordinate origin). Further, a plurality of grid cells can be distributed on the coordinate system of the global grid map, which can have the same predetermined size. It should be understood that the global grid map described herein is to cover and identify objects and objects in the environment where the vehicle is located as much as possible, and is not intended to limit the specific size of the grid map. In other words, the size of the grid map can be selected depending on the specific use needs, for example, and a local grid map is also possible. Figure 3

[0049] To obtain accurate position information of the vehicle 320 as the target object, a plurality of sensors of the vehicle 310 can sense the vehicle 320 to obtain sensing information. For example, the sensors on the vehicle 310 can include ultrasonic radar, camera and lidar, and the obtained sensing information can thus include ultrasonic radar sensing data, camera sensing data and lidar sensing data.

[0050] According to embodiments of the present disclosure, the sensing data for the vehicle 320 can be projected to the global grid map corresponding to the plane of the vehicle 310 to generate a plurality of sensing points of the vehicle 320. As described above, a corresponding sensing point to be associated corresponds to the same ground truth point of the target object as the target sensing point, and the distance between the corresponding sensing point and the target sensing point is not greater than a predetermined distance threshold. According to embodiments of the present disclosure, the corresponding sensing point to be associated is further sensed by the same sensor as the sensing target sensing point at different time instances, or further sensed by a different sensor from the sensing target sensing point at the same time instance, or both.

[0051] As shown in FIG. 3, the plurality of sensing points for the vehicle 320 edge embodying the outline of the vehicle 320 can come from the ultrasonic radar sensing data, camera sensing data or lidar sensing data at the same or different time instances. By way of example and not limitation, the target sensing point 301 is a sensing point sensed by a camera at a first time instance, and the grid cell in which the target sensing point 301 is located is identified as a target grid cell. In the identified target grid cell, the sensing point 302 corresponds to the same ground truth point of the vehicle 320 as the target sensing point 301, which is also a sensing point sensed by a lidar at the first time instance. Thus, in the target grid cell, the target sensing point 301 and the corresponding sensing point 302 can be associated for fusion. Figure 3

[0052] ​​Additionally, by way of example and not limitation, the target sensing point 303 is a sensing point sensed by the ultrasonic radar at a first time instance, and the grid cell in which the target sensing point 303 is located is identified as a target grid cell. In a neighbor grid cell adjacent to the target grid cell, the sensing point 304 corresponds to the same ground truth point of the vehicle 320 as the target sensing point 303, but it is a sensing point sensed by the ultrasonic radar at a second time instance different from the first time instance. For example, the second time instance is earlier than the first time instance, i.e., the sensing point 304 is a historical sensing point compared to the target sensing point 303, and vice versa. In response to a distance (e.g., Euclidean distance) between the target sensing point 303 and the sensing point 304 being less than or equal to a predetermined distance threshold, the target sensing point 303 and the corresponding sensing point 304 can be associated for fusion in the identified target grid cell and the neighbor grid cell. In some embodiments, the identified target grid cell and the neighbor grid cell form a strip shape. In the following, grid cell identification according to embodiments of the present disclosure will be described in further detail.

[0053] According to embodiments of the present disclosure, based on the sensing data corresponding to the target sensing point, the coordinate of the target sensing point in the global grid map can be determined. Then, based on the predetermined size of the grid cell, the horizontal coordinate value and the vertical coordinate value of the coordinate of the target sensing point, the first number of grid cells of the target sensing point from the vertical axis and the second number of grid cells of the target sensing point from the horizontal axis can be determined. Here, the numerical values of the first number and the second number can be the same or different.

[0054] Figure 4 An example flow of the data association process 400 according to embodiments of the present disclosure is illustrated. Blocks 410-440 indicate the input sensing data to be associated, respectively. The ultrasonic sensor system (USS) point cloud 410 can be generated by an ultrasonic radar transmitting and receiving ultrasonic waves, which utilizes the time-of-flight principle to determine the position of obstacles around a vehicle. The single-frame information is usually represented as a set of two-dimensional (2D) points.

[0055] According to embodiments of the present disclosure, the boundary information of the target object in the image can be obtained by performing spatial estimation based on the camera sensing data for the target object, and the depth information of the target object in the image can be obtained by performing depth estimation based on the camera sensing data for the target object. Such boundary information and depth information can be projected to the global grid map.

[0056] Free space 420 can be acquired by inputting camera images, e.g., of a single frame fisheye or pinhole camera, into a deep learning model for free space estimation. This provides the boundaries of obstacles around the vehicle in the image domain. These boundary details are then projected using a camera model to the coordinate system to derive the actual obstacle positions. Single frame information is typically also presented in the form of a set of 2D points. Depth image 430 involves inputting camera images, e.g., of a single frame fisheye or pinhole camera, into a monocular depth estimation deep learning model. This results in a depth value corresponding to each pixel in the image. By projecting these pixel depths into the coordinate system, a 3D point cloud of obstacles around the vehicle can be formed. Single frame information is typically represented by a single image, which corresponds to one frame of the 3D point cloud. Furthermore, lidar point cloud 440 can consist of point cloud data collected by a lidar sensor.

[0057] Blocks 450-470 indicate respective data association steps. At joint 450, the sensing data from various sensors, e.g., ultrasonic radar sensing data, camera sensing data, and lidar sensing data, can be converted into a uniform data structure. This simplifies the association between data from different data sources, and between data from the same data source but different acquisition times. In a 2D scenario, a vector of 2D points can be used as the uniform structure. For a 3D scenario, a vector of 3D points can be used. At point localization 460, each point in the vector can be traversed, and a corresponding grid cell (or voxel in the 3D case) can be computed from the coordinates of each point. At ray casting 470, a virtual ray can be cast from the vehicle itself to the center point of the target grid cell, or even further. It can be reasonably assumed that all grid cells along this path are affected by the measurement point currently being processed.

[0058] According to embodiments of the present disclosure, a ray can be virtually cast from the coordinate origin to the target sensing point, and a predetermined number of neighbor grid cells upstream, downstream, or both of the target grid cell can be identified in the direction of the ray. Furthermore, the cell identities of the target grid cell and the identified neighbor grid cells can be output. As shown in the identification output 480 in Figure 4 As shown in the identification output 480 in

[0059] Figure 5 The above-described data association process is exemplarily and non- limitingly illustrated in a diagram of an example implementation 500. It should be understood that, Figure 5 The example steps in are not limiting, and more or fewer steps, or modifications to these steps, can be selected depending on the specific situation. In Figure 5In some embodiments, at 510, an index of the grid cell where the target sensing point is located (point cell idx) and an index of the grid cell where the sensor is located (sensor cell idx) are determined, which can be determined by dividing the coordinates of the point and the coordinates of the sensor by the size of the grid cell, respectively. At 520, a direction of the ray (ray direction) and a length of the ray (ray length) are determined, for example, the difference between the index of the grid cell where the target sensing point is located and the index of the grid cell where the sensor is located can be normalized and its norm can be taken.

[0060] At 530, a step can be initialized, which is equal to the size of the grid cell (cell size), and all the grid cells that the ray passes through can be tracked in a loop from the index of the grid cell where the sensor is located (sensor cell idx) with the step until the index of the grid cell where the target sensing point is located (point cell idx) is reached (or approached or exceeded). At 535, it is determined whether the index of the grid cell where the target sensing point is located (point cell idx) is reached. If yes, go to 540, then the index of the current grid cell (cur cell idx) is added to the list or set of associated cells. If no, go back to 530 and continue the loop tracking. Below, the ray casting process according to embodiments of the present disclosure will be described in further detail.

[0061] According to embodiments of the present disclosure, in response to the target sensing point being from camera sensing data or lidar sensing data for the target object, a ray can be virtually cast from the optical center of the camera or the mounting position of the lidar on the vehicle to the center of the target grid cell. In addition, in response to the target sensing point being from ultrasonic radar sensing data for the target object, it can be determined whether the ultrasonic radar sensing data is sensed by an ortho-sonar or a side-sonar on the vehicle. In the case of determining that the ultrasonic radar sensing data is sensed by the ortho-sonar, a ray can be virtually cast from the self-vehicle contour perpendicularly to the center of the target grid cell along a straight segment corresponding to the ortho-sonar. In the case of determining that the ultrasonic radar sensing data is sensed by the side-sonar, a ray can be virtually cast from the self-vehicle contour to the center of the target grid cell along an arc segment corresponding to the side-sonar. Below, the virtual casting process of the ray will be described in further detail with the aid of Figure 6 An exemplary illustration is given.

[0062] Figure 6 A diagram of a virtual casting process 600 of a ray according to embodiments of the present disclosure is illustrated. As shown in FIG. 6, a ray is virtually cast from the optical center of the camera or the mounting position of the lidar on the vehicle to the center of the target grid cell. In addition, in response to the target sensing point being from ultrasonic radar sensing data for the target object, it can be determined whether the ultrasonic radar sensing data is sensed by an ortho-sonar or a side-sonar on the vehicle. In the case of determining that the ultrasonic radar sensing data is sensed by the ortho-sonar, a ray can be virtually cast from the self-vehicle contour perpendicularly to the center of the target grid cell along a straight segment corresponding to the ortho-sonar. In the case of determining that the ultrasonic radar sensing data is sensed by the side-sonar, a ray can be virtually cast from the self-vehicle contour to the center of the target grid cell along an arc segment corresponding to the side-sonar. Below, the virtual casting process of the ray will be described in further detail with the aid of Figure 6As shown in the figure, the ego-vehicle geometry can be simplified as a rounded rectangle, which is divided into multiple regions. Among them, regions 610, 630, 650 and 670 are front regions, and regions 620, 640, 660 and 680 are side regions. In addition, point 615 indicates the center of the vehicle, and point 625 indicates the center of the rear axle of the vehicle.

[0063] When the processing object is a measurement point generated by a camera or a laser radar, it is convenient and intuitive to take the camera optical center or the laser radar installation position as the starting point and the position where the measurement point is located as the ending point to obtain a ray. This conforms to the observation principle of the camera and the laser radar, that is, the measurement medium is light, and the light propagates along a straight line as an electromagnetic wave. However, when the processing object is a measurement point obtained by an ultrasonic radar, the measurement medium is no longer an electromagnetic wave, but an ultrasonic wave. The ultrasonic wave is essentially a mechanical wave and does not propagate along a straight line, but spreads uniformly in all directions from the sound source.

[0064] Therefore, when performing virtual ray projection on a certain ultrasonic observation point P, it can be judged which region of the above-mentioned regions the P point is located in. If the region where the P point is located is a front region (that is, the regions 610, 630, 650 and 670 shown in the figure), then at this time the ego-vehicle contour corresponding to the P point is a line segment, and the perpendicular from the P point to the line segment is the starting point of the ray. If the region where the P point is located is a side region (that is, the regions 620, 640, 660 and 680 shown in the figure), then at this time the ego-vehicle contour corresponding to the P point is a circular arc, and the intersection of the line segment PO and the circular arc is the starting point of the ray. That is, it can be considered that the measurement point obtained by the ultrasonic wave is equivalent to the point observed by the light uniformly radiated from the ego-vehicle contour.

[0065] Figure 7A The figure illustrates an example effect of virtual projection of a ray according to an embodiment of the present disclosure, and Figure 7B is Figure 7A a partial enlarged view. Among them, the ego-vehicle geometry can be smaller than the real size of the vehicle, for example, the ego-vehicle geometry can correspond to the shape formed by multiple wheels. In this way, the sensors arranged along the ego-vehicle geometry can sense the underbody and even the obstacles close to it.

[0066] Figure 8 The figure illustrates an example flow of a data fusion process 800 according to an embodiment of the present disclosure. As Figure 8As shown in FIG. 8, the sensing stage 810 can include USS sensing 811, video sensing 812, and lidar sensing 813. The real-time sensing information of the sensing stage 810 can be transmitted to the fusion stage 820. The fusion stage 820 can include time-space synchronization 821, data association 822, and data fusion 823. The environment information of the fusion stage 820 can be transmitted to the regulation stage 830. The regulation stage 830 can include planning prediction 831, planning 832, and control 833.

[0067] The association of sensing information received from different sensors is to identify which sensing data comes from the same obstacle, so that the sensing data related to the same obstacle is fused when fusing. This is the key to ensuring that the system accurately understands the environment, by combining data from various sensors to form a coherent and comprehensive representation. For example, the output is the association between historical information and current information, which can then be used to update the historical state of these grid cells with the new state generated by the new measurement points, thereby facilitating fusion.

[0068] According to embodiments of the present disclosure, the pair-wise distance calculation between sensing points is effectively avoided. Instead, the process only involves using the coordinates of the sensing points in combination with the grid cell size to determine the cell identification of the grid cell to which each observation point belongs. Furthermore, with the construction of the ray direction, and the imposition of a limit on the number of grid cells affected on the ray, the overall complexity should be reduced to O(n). This optimization significantly improves the efficiency of the data association process, making it more suitable for larger data sets.

[0069] Figure 9 A schematic diagram of an apparatus 900 for fusing sensing data according to embodiments of the present disclosure is shown. The apparatus 900 can include a plurality of units or modules for performing the corresponding steps in the method 200 as discussed in Figure 2 FIG. 8. As shown in Figure 9 FIG. 9, the apparatus 900 includes a coordinate formation module 910 configured to form, based on sensing data from sensors on a vehicle, a global grid map including a plurality of grid cells; a grid identification module 920 configured to identify, in the global grid map, a target grid cell in which a target sensing point of a target object between the vehicle and the target object, and a predetermined number of neighbor grid cells adjacent to the target grid cell; and a sensing point association module 930 configured to associate, in the target grid cell and the neighbor grid cells, the target sensing point of the target object and respective sensing points satisfying a predetermined condition, wherein the target sensing point and the respective sensing points are from at least one of: different sensors, or different time instances.

[0070] In some embodiments, wherein the predetermined condition can comprise: in the target grid cell or the neighbor grid cell, the respective sensing point corresponds to a same ground truth point of the target object as the target sensing point, and a distance between the respective sensing point and the target sensing point is not greater than a predetermined distance threshold.

[0071] In some embodiments, wherein the sensor can comprise a plurality of sensors, the plurality of sensors can comprise an ultrasonic radar, a camera, and a lidar, and wherein the respective sensing points to be associated can further comprise at least one of: a sensing point sensed by a same sensor as the target sensing point at a different time instance; or a sensing point sensed by a different sensor from the target sensing point at a same time instance.

[0072] In some embodiments, wherein the sensing data can comprise ultrasonic radar sensing data, camera sensing data, lidar sensing data, the apparatus (900) can further comprise: a projection module configured to project the sensing data for the target object to the global grid map corresponding to a plane of the vehicle to generate a plurality of sensing points of the target object.

[0073] In some embodiments, further comprising: the apparatus (900) can further comprise an estimation module configured to: obtain boundary information of the target object in an image by performing spatial estimation based on the camera sensing data for the target object; and obtain depth information of the target object in the image by performing depth estimation based on the camera sensing data for the target object. The boundary information and the depth information can be projected to the global grid map.

[0074] In some embodiments, wherein each of the plurality of grid cells included in the global grid map can have a same predetermined size, the global grid map can further comprise a coordinate origin, a horizontal axis, and a vertical axis perpendicular to the horizontal axis corresponding to the vehicle, and identifying the target grid cell can comprise: determining a coordinate of the target sensing point in the global grid map based on the sensing data corresponding to the target sensing point; and determining a first number of grid cells of the target sensing point from the vertical axis and a second number of grid cells of the target sensing point from the horizontal axis based on the predetermined size of a grid cell, a horizontal coordinate value and a vertical coordinate value of the coordinate of the target sensing point.

[0075] In some embodiments, further comprising: the apparatus (900) can further comprise a projecting module configured to: virtually project a ray from the coordinate origin to the target sensing point; identify the predetermined number of the neighbor grid cells upstream, downstream, or both upstream and downstream of the target grid cell in the direction of the ray; and output the cell identifications of the target grid cell and the identified neighbor grid cells.

[0076] In some embodiments, wherein virtually projecting the ray can comprise: in response to the target sensing point being from camera sensing data or lidar sensing data for the target object, virtually projecting a ray from a principal point of a camera or a mounting position of a lidar on the vehicle to a center of the target grid cell.

[0077] In some embodiments, wherein virtually projecting the ray can comprise: in response to the target sensing point being from ultrasonic radar sensing data for the target object, determining whether the ultrasonic radar sensing data is sensed by a front-facing ultrasonic radar or a side-facing ultrasonic radar on the vehicle; in a case that the ultrasonic radar sensing data is determined to be sensed by the front-facing ultrasonic radar, virtually projecting a ray from a center of a straight line segment corresponding to the front-facing ultrasonic radar perpendicularly to a self-vehicle contour to a center of the target grid cell; and in a case that the ultrasonic radar sensing data is determined to be sensed by the side-facing ultrasonic radar, virtually projecting a ray from a center of a fillet of an arc line segment corresponding to the side-facing ultrasonic radar to a center of the target grid cell.

[0078] In some embodiments, can further comprise a fusing module configured to: facilitate fusion of the sensing data based on the associated target sensing points and the corresponding sensing points; and control the vehicle to perform driving functionality based on a fusion result of the sensing data.

[0079] Figure 10 A schematic block diagram of an example device 1000 suitable for implementing embodiments of the present disclosure is shown. The controller in the foregoing can be implemented with the device 1000. As shown, the device 1000 includes a processor 1001, which can execute various appropriate actions and processes according to computer program instructions loaded into a random access memory (RAM) 1003 from a read only memory (ROM) 1002. Various programs and data required for operation of the device 1000 can also be stored in the RAM 1003. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0080] The various processes and processes described above, such as the method 200 and the processes described above, can be performed by the processor 1001. For example, in some embodiments, the method 200 and the processes described above can be implemented as a computer software program tangibly embodied in a machine-readable medium. In some embodiments, some or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM 1002. When the computer program is loaded onto the RAM 1003 and executed by the processor 1001, one or more acts of the method 200 and the processes described above can be performed. In accordance with embodiments of the present disclosure, a vehicle can include the device 1000 as described above for performing various aspects of the present disclosure.

[0081] The present disclosure can be a method, apparatus, electronic device, vehicle, computer readable storage medium, and / or computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for performing various aspects of the present disclosure.

[0082] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a magnetically encoded device such as tape, an optically encoded device such as a compact disc (CD) or DVD, or any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0083] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0084] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0085] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0086] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or nonvolatile memory, or a suitable combination of the different types of computer readable storage media. The computer readable program instructions can include program elements, such as an operating system, a database management system, and / or one or more applications that, in combination with the computer readable storage medium, provide one or more functions / acts for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0087] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0088] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0089] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of terms is intended to best describe the principles of the embodiments, practical application, or technical improvements over the technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method (200) for fusing sensed data, comprising: Based on sensing data from sensors on the vehicle, a global grid map (210) is formed, the global grid map comprising multiple grid cells; In the global grid diagram, the target grid cell where the target sensing point of the target object is located and a predetermined number of neighboring grid cells adjacent to the target grid cell are identified between the vehicle and the target object (220). as well as In the target grid cell and the neighboring grid cell, the target sensing point of the target object is associated (230) with a corresponding sensing point that meets a predetermined condition, wherein the target sensing point and the corresponding sensing point come from at least one of the following: different sensors or different time instances.

2. The method (200) according to claim 1, wherein the predetermined conditions include: In the target grid cell or the neighboring grid cell, The corresponding sensing point and the target sensing point correspond to the same truth point of the target object, and The distance between the corresponding sensing point and the target sensing point is not greater than a predetermined distance threshold.

3. The method (200) according to claim 1, wherein the sensor comprises a plurality of sensors, the plurality of sensors including ultrasonic radar, a camera, and lidar, and The corresponding sensing point to be associated further includes at least one of the following: Sensing points sensed at different times by the same sensor that sensed the target sensing point; or Sensing points are instances sensed at the same time by a different sensor than the one that senses the target sensing point.

4. The method (200) according to claim 3, wherein the sensing data includes ultrasonic radar sensing data, camera sensing data, and lidar sensing data, and the method further includes: The sensing data for the target object is projected onto a global grid map corresponding to the plane of the vehicle to generate multiple sensing points for the target object.

5. The method (200) according to claim 4, further comprising: The boundary information of the target object in the image is obtained by performing spatial estimation based on the camera sensing data for the target object; Depth information of the target object in the image is obtained by performing depth estimation based on the camera sensing data for the target object; as well as The boundary information and the depth information are projected onto the global raster map.

6. The method (200) according to claim 1, wherein each of the plurality of grid cells included in the global grid map has the same predetermined size, the global grid map further includes a coordinate origin corresponding to the vehicle, a horizontal axis, and a vertical axis perpendicular to the horizontal axis, and identifying the target grid cell includes: Based on the sensing data corresponding to the target sensing point, the coordinates of the target sensing point in the global grid are determined; as well as Based on the predetermined size of the grid cells, the horizontal and vertical coordinates of the target sensing point, a first number of grid cells from the vertical axis and a second number of grid cells from the horizontal axis are determined for the target sensing point.

7. The method (200) according to claim 6, further comprising: A ray is virtually projected from the origin of the coordinate system toward the target sensing point; In the direction of the ray, the predetermined number of neighboring grid cells are identified upstream, downstream, or both upstream and downstream of the target grid cell; and Output the cell identifiers of the target raster cell and the identified neighboring raster cells.

8. The method (200) of claim 7, wherein virtually projecting the ray comprises: In response to the target sensing point receiving camera sensing data or lidar sensing data for the target object, a ray is virtually projected from the optical center of the camera or the mounting location of the lidar on the vehicle toward the center of the target grid cell.

9. The method (200) of claim 7, wherein virtually projecting the ray comprises: In response to the target sensing point being derived from ultrasonic radar sensing data for the target object, it is determined whether the ultrasonic radar sensing data was sensed by a frontal ultrasonic radar or a lateral ultrasonic radar on the vehicle. When it is determined that the ultrasonic radar sensing data is sensed by the azimuth ultrasonic radar, a ray is virtually projected perpendicularly from the vehicle profile of the straight line segment corresponding to the azimuth ultrasonic radar toward the center of the target grid cell; and When it is determined that the ultrasonic radar sensing data is sensed by the lateral ultrasonic radar, a ray is virtually projected from the center of the rounded corner of the vehicle profile of the arc segment corresponding to the lateral ultrasonic radar to the center of the target grid cell.

10. The method (200) according to claim 1, further comprising: Based on the associated target sensing point and the corresponding sensing point, the fusion of the sensing data is facilitated; as well as Based on the fusion result of the sensed data, the vehicle is controlled to perform driving functions.

11. An apparatus for fusing sensed data, comprising: A coordinate forming module is configured to form a global grid map based on sensing data from sensors on the vehicle, the global grid map comprising multiple grid cells; The grid identification module is configured to identify, in the global grid map, the target grid cell where the target sensing point of the target object is located, and a predetermined number of neighboring grid cells adjacent to the target grid cell; as well as A sensing point association module is configured to associate the target sensing point of the target object with a corresponding sensing point that meets predetermined conditions in the target grid cell and the neighboring grid cells, wherein the target sensing point and the corresponding sensing point come from at least one of the following: different sensors or different time instances.

12. A controller, comprising: At least one processor; as well as A memory, coupled to the at least one processor and storing instructions thereon, which, when executed by the at least one processor, cause the controller to perform the method according to any one of claims 1-10.

13. A vehicle comprising the controller according to claim 12.

14. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a processor of a computer, cause the computer to perform the method according to any one of claims 1 to 10.

Images