Detecting three-dimensional structural models at runtime in vehicles

Abstract

Device with which a vehicle (101) is equipped, the device comprising: a receiver (155) of a navigation system; a camera (108); a memory (145); and a processor (140) coupled to the receiver (155) of the navigation system, the memory (145) and the camera (108), wherein the processor (140) is configured to execute a method for detecting one or more three-dimensional structures in an environment of the vehicle (101) at runtime, the method comprising: Generating, by the processor (140), using the camera (108), a bird's-eye view (BEV) camera image (204) of the environment of the vehicle (101), wherein the BEV camera image (204) includes two-dimensional coordinates of one or more structures in the environment; Generating, by the processor (140), a BEV height image of the environment of the vehicle (101), wherein the BEV height image provides a height of one or more structures in the environment; Detect, by the processor (140), one or more edges (402) of one or more three-dimensional structures in the environment based on the BEV camera image (204) and the BEV height image; Generating, by the processor (140), models of the one or more three-dimensional structures in the environment of the vehicle (101) by plane fitting based on the boundaries (402) of the one or more three-dimensional structures; and Reconfiguring, by the processor (140), of the receiver (155) of the navigation system based on the models of one or more three-dimensional structures in the environment of the vehicle (101); wherein the BEV height image is generated based on radar data acquired by a radar (110) of the vehicle (101); wherein the BEV elevation image is a BEV radar image (206) generated using radar data; including the detection of the edges (402) of one or more three-dimensional structures: Generating, by the processor (140), a merged image by combining the BEV camera image (204) and the BEV radar image (206) to add the elevation information as a channel for each pixel in the BEV camera image (204); and Detect, by the processor (140), the edges (402) by inputting the merged image into an artificial neural network.

Description

[0001] This disclosure relates to improving the performance of a global navigation satellite system (GNSS) with which vehicles may be equipped to provide autonomous geospatial positioning, and more particularly to improving performance by determining three-dimensional (3D) models of buildings and other structures from a vehicle using a combination of sensor devices, such as a camera, a radio detection and ranging (radar) device, and / or a light detection and ranging (lidar) device.

[0002] Vehicles such as cars, trucks, boats, etc., equipped with GNSS receivers can provide navigation information to a (human, autonomous, or semi-autonomous) operator. Common GNSS systems include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), Galileo, Beidou, and other regional systems. GNSS receivers can provide navigation information up to a specific level of accuracy; however, this level of accuracy is typically limited to open-air environments. For example, GNSS receivers can achieve measurements within an accuracy of at least 1.5 meters (m) in open-air environments. In urban areas, such as cities, GNSS receivers are generally less accurate, especially in the presence of buildings and other structures.By blocking the receivers' view of the open sky, the buildings and other structures block the path of the GNSS receivers to receive signals from the GNSS systems and instead create an environment of high multipath radio frequency (RF) signals.

[0003] US 2017 / 0 109 940 A1 discloses systems and methods for generating an image visualization, in which a combined image is received consisting of a combination of images captured by one or more image sensors. A visualization of the combined image is adapted based on acquired depth information based on distance measurements.

[0004] Document DE 10 2011 052 950 A1 discloses a satellite-based positioning system that enables improved performance for different environments and types of use. SUMMARY

[0005] A computer-implemented method for detecting one or more three-dimensional structures in an environment of a vehicle at runtime includes generating, by a processor, a bird's-eye view (BEV) camera image in the environment of the vehicle, wherein the BEV camera image includes two-dimensional coordinates of one or more structures in the environment. The method further includes generating, by the processor, a BEV elevation image of the environment of the vehicle, wherein the BEV elevation image provides the elevation of the one or more structures in the environment. The method further includes detecting, by the processor, one or more edges of the one or more three-dimensional structures in the environment based on the BEV camera image and the BEV elevation image.The method further includes generating, by the processor, models of the one or more three-dimensional structures in the environment of the vehicle through plane matching based on the edges of the one or more three-dimensional structures. The method further includes reconfiguring, by the processor, a receiver of a navigation system based on the models of the one or more three-dimensional structures in the environment of the vehicle.

[0006] According to one or more embodiments, the BEV elevation image is generated based on radar data acquired by a radar of the vehicle.

[0007] According to one or more embodiments, the BEV elevation image is a BEV radar image generated using the radar data.

[0008] According to one or more embodiments, detecting the edges of the one or more three-dimensional structures includes generating a merged image by merging, by the processor, the BEV camera image and the BEV radar image to add the elevation information as a channel for each pixel in the BEV camera image, and detecting the edges by inputting, by the processor, the merged image to an artificial neural network.

[0009] According to one or more embodiments, the BEV elevation image is generated based on the lidar data acquired by a lidar of the vehicle.

[0010] According to one or more embodiments, the camera includes a plurality of cameras mounted on different sides of the vehicle.

[0011] According to one or more embodiments, reconfiguring the receiver of the navigation system includes changing a frequency used by the receiver of the navigation system.

[0012] According to the invention, a device with which a vehicle is equipped includes a receiver of a navigation system, a camera, a memory, and a processor coupled to the receiver of the navigation system, the memory, and the camera. The processor is configured to execute a method for detecting three-dimensional structures in an environment of the vehicle at runtime. The method includes generating a bird's-eye view (BEV) camera image of the environment of the vehicle, wherein the BEV camera image includes two-dimensional coordinates of one or more structures in the environment. The method further includes generating a BEV elevation image of the environment of the vehicle, wherein the BEV elevation image provides the elevation of the one or more structures in the environment.The method further includes detecting one or more edges of the one or more three-dimensional structures in the environment based on the BEV camera image and the BEV elevation image. The method further includes generating models of the one or more three-dimensional structures in the environment of the vehicle through plane fitting based on the edges of the one or more three-dimensional structures. The method further includes reconfiguring a receiver of a navigation system based on the models of the one or more three-dimensional structures in the environment of the vehicle.

[0013] According to the invention, the BEV elevation image is generated based on the radar data acquired by a radar of the vehicle.

[0014] According to the invention, the BEV elevation image is a BEV radar image generated using the radar data.

[0015] According to the invention, detecting the edges of the one or more three-dimensional structures includes generating a merged image by merging the BEV camera image and the BEV radar image by the processor to add the height information as a channel for each pixel in the BEV camera image, and detecting the edges by inputting the merged image into an artificial neural network by the processor.

[0016] According to one or more embodiments, the BEV elevation image is generated based on the lidar data acquired by a lidar of the vehicle.

[0017] According to one or more embodiments, the camera includes a plurality of cameras mounted on different sides of the vehicle.

[0018] According to one or more embodiments, reconfiguring the receiver of the navigation system includes changing a frequency used by the receiver of the navigation system.

[0019] A computer program product includes a computer storage device containing computer-executable instructions that, when executed by a processing unit, cause the processing unit to perform a method for detecting three-dimensional structures in an environment of the vehicle at runtime. The method includes generating a bird's-eye view (BEV) camera image of the environment of the vehicle, wherein the BEV camera image includes two-dimensional coordinates of one or more structures in the environment. The method further includes generating a BEV elevation image of the environment of the vehicle, wherein the BEV elevation image provides the elevation of the one or more structures in the environment. The method further includes detecting one or more edges of the one or more three-dimensional structures in the environment based on the BEV camera image and the BEV elevation image.The method further includes generating models of the one or more three-dimensional structures in the environment of the vehicle by plane fitting based on the edges of the one or more three-dimensional structures. The method further includes reconfiguring a receiver of a navigation system based on the models of the one or more three-dimensional structures in the environment of the vehicle.

[0020] According to the invention, the BEV elevation image is generated based on the radar data acquired by a radar of the vehicle.

[0021] According to the invention, the BEV elevation image is a BEV radar image generated using the radar data.

[0022] According to the invention, detecting the edges of the one or more three-dimensional structures includes generating a merged image by merging the BEV camera image and the BEV radar image by the processor to add the height information as a channel for each pixel in the BEV camera image, and detecting the edges by inputting the merged image into an artificial neural network by the processor.

[0023] According to one or more embodiments, the BEV elevation image is generated based on the lidar data acquired by a lidar of the vehicle.

[0024] According to one or more embodiments, the camera includes a plurality of cameras mounted on different sides of the vehicle.

[0025] According to one or more embodiments, reconfiguring the receiver of the navigation system includes changing a frequency used by the receiver of the navigation system.

[0026] The above features and advantages and other features and advantages of the disclosure are readily apparent from the following detailed description when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Other features, advantages and details appear only by way of example in the following detailed description, which refers to the drawings; in which: Fig. 1 is a block diagram of an exemplary system for sensing the environment near a vehicle and dynamically detecting 3D structures according to one or more embodiments; Fig. 2 Examples of various types of data collected and obtained by a vehicle, according to one or more embodiments; Fig. 3 is a flowchart of a method for estimating 3D models of static and dynamic structures in real time of a vehicle according to one or more embodiments; Fig. 4 Examples of images used for edge detection according to one or more embodiments; and Fig. 5 illustrates a computing system used to implement one or more embodiments described herein. DETAILED DESCRIPTION

[0028] The following description is merely exemplary and is not intended to limit the present disclosure, its application, or uses. Throughout the drawings, corresponding reference characters indicate like or corresponding parts and features. Control module, module, controller, control unit, electronic control unit, processor, and similar terms as used herein mean any one or various combinations of one or more application-specific integrated circuits (ASICs), electronic circuits, central processing units (preferably microprocessors), and associated data storage and memory (read-only memory (ROM), random access memory (RAM), electrically programmable read-only memory (EPROM), hard disk, etc.).), graphics processing units or microcontrollers executing one or more software or firmware programs or one or more software or firmware routines, combinational logic circuitry, input / output (I / O) circuitry and devices, and suitable signal conditioning and buffering circuitry, a high-speed clock, analog-to-digital (A / D) and digital-to-analog (D / A) circuitry, and other components to provide the described functionality. A control module may include various communication interfaces, including, for example, point-to-point or discrete lines and wired or wireless interfaces to networks including wide area networks and local area networks, in-vehicle controller area networks, and plant and service networks.The control module functions set forth in this disclosure may be implemented in a distributed control architecture among multiple networked control modules. Software, firmware, programs, instructions, routines, code, algorithms, and similar terms refer to any controller-executable sets of instructions, including calibrations, data structures, and lookup tables. A control module includes a set of control routines that execute to provide the described functions. The routines are executed, such as by a central processing unit, and are operable to monitor inputs from sensor devices and other networked control modules and to execute control and diagnostic routines to control the operation of actuators. The routines may be executed at regular intervals during ongoing operation of the engine and vehicle.Alternatively, the routines can be executed in response to the occurrence of an event, to software calls, or on demand via inputs or requests from the user interface.

[0029] The technical solutions described here address technical challenges related to the accuracy of Global Navigation Satellite System (GNSS) receivers. As previously stated, vehicles such as passenger cars, trucks, boats, etc., equipped with GNSS receivers can provide navigation information to a human, autonomous, or semi-autonomous operator. However, the level of accuracy is typically limited to open-air environments, with GNSS receivers being less accurate in urban areas such as cities, particularly in the presence of buildings and other structures. By blocking the receivers' view of the open sky, the buildings and other structures block the path for GNSS receivers to receive signals from the GNSS systems, creating an environment with high multipath radio frequency (RF) signals.

[0030] The technical solutions described here address such technical challenges with accuracy by dynamically detecting three-dimensional (3D) structures, such as buildings, which can prevent the GNSS receiver from receiving navigation information. The detection of the 3D structures is performed entirely at runtime without any a priori information about the models of such structures, such as static maps containing location and other information of the 3D structures. In other words, the technical solutions described here for dynamically detecting 3D structures do not require any a priori information or continuous map updates. The technical solutions accordingly provide cost savings compared to existing solutions that use static maps or continuously update maps to detect such 3D structures.In one or more embodiments, structures such as buildings that are detected with height measurements below (or above) a predetermined threshold are reported, for example, to reduce false alarms caused by structures such as curbs.

[0031] In Fig. 1 illustrates a block diagram of an exemplary system 100 for scanning the environment near a vehicle and dynamically detecting 3D structures. The system 100 includes a vehicle 101 equipped with a processor 140, a memory 145, a GNSS receiver 155, a camera 108, and a radar 110. In one or more embodiments, the vehicle 101 is also equipped with a lidar 112. It is recognized that the vehicle 101 may be equipped with additional devices and / or components not listed here.

[0032] Processor 140 may be a microprocessor, a graphics processing unit, a digital signal processor, or any other processing unit. Processor 140 may execute one or more computer-executable instructions to perform one or more methods described herein. Such computer-executable instructions may be stored in memory 145. Memory 145 may store additional information used by processor 140 during execution of the computer-executable instructions. For example, memory 145 may store temporary values, media (e.g., images, audio, video), and other information used during execution.

[0033] The system 100 may use the camera 108 among other sensors to create an image 202 ( Fig. 2) the surroundings of the vehicle 101. Here, the "surroundings" of the vehicle 101 can be the environment of the vehicle within a predetermined distance (e.g., 50 meters, 100 meters, etc.) from the center (or any other reference point) of the vehicle 101. The camera 108 is operable to capture the image 202 within a field of view (FOV), which can include static and dynamic objects within that FOV. Image processing techniques can be used to locate and classify objects within the FOV. The FOV is generally correlated with the driving scene or a portion of the driving scene bounded by the FOV.

[0034] The camera 108 may include multiple cameras located inside and / or outside the vehicle 101. A subset of the cameras 108 may, for example, be located on the sides, top, front, and / or rear of the vehicle 101 and operable to capture the image 202 or a series of images of a field of view immediately adjacent to the vehicle 101. In an exemplary embodiment, the camera 108 may include a wide-angle camera. Sensor fusion may be performed to provide accurate detection, classification, tracking, etc., of external objects, as well as the calculation of appropriate attributes, such as relative velocities, accelerations, and the like. The camera 108, in one or more embodiments, enables the generation of a bird's-eye view (BEV) image 204 of the surroundings of the vehicle 101.

[0035] The BEV camera image 204 is generated using one or more known computer vision techniques. For example, a predetermined number of cameras 108 are mounted around the vehicle 101 so that their fields of view cover the surroundings. The parameters of the cameras 108 are calibrated in advance so that the captured images 202 from the cameras 108 can be rectified for integration into perspective views. Overlapping areas of adjacent views in the images 202 are merged by aligning along a seam using a dynamic programming method, followed by spreading the deformation field of the alignment using Wendland functions. In this way, the images 202 can be integrated into a single, panoramic, and seamless BEV camera image 204.It is recognized that other techniques may be used to generate the BEV camera image 204 and that aspects of the technical solutions described herein are not limited by which technique is used to generate the BEV camera image 204.

[0036] The radar 110 is used to detect objects in the environment of the vehicle 101. In one or more embodiments, the radar data acquired by the radar 110 provides distance information to the 3D structures in the environment of the vehicle 101. In one or more embodiments, the vehicle 101 is equipped with multiple radars 110. The radar data acquired by the multiple radars 110 is used to create a BEV radar image 206 ( Fig. 2) the surroundings of the vehicle 101. Multiple scans from the radar(s) may be acquired and spatially aligned using information from an inertial measurement unit (IMU) to generate the BEV radar image 206. The Doppler information from the radar 110 may additionally be used to identify and remove moving objects from the multiple scans prior to generating the BEV radar image 206. The BEV radar image 206 is an image of the surroundings of the vehicle 101, with each pixel of the BEV radar image 206 representing a detected object, confidence in a detection, or radar cross-section.

[0037] The BEV camera image 204 and the BEV radar image 206 are used together to generate a model containing the 3D structures in the environment of the vehicle 101. Generating the model includes calculating the height of the structures in the environment. The height of a target can be calculated using a distance R of the target from the vehicle 101, which is provided by the range information from the radar 110. Furthermore, calculating the height of a target is based on an elevation angle E of the target, which can be determined from the image 202. The height for each of the pixels in the camera image(s) 202 representing the structures in the environment of the vehicle 101 can be determined using the corresponding distance R from the range data and the elevation angle E from the camera image 202.

[0038] Based on the model of the environment, the GNSS receiver 155 can be programmed to improve reception of the navigation signal from the GNSS. The programming may include, for example, changing one or more parameters associated with the receiver 155. Alternatively or additionally, based on the model of the environment, the navigation information for the vehicle 101 is obtained from another source, such as a telephone (not shown) that may be associated with the vehicle 101, e.g., using a wired or wireless connection.

[0039] In embodiments where the vehicle 101 includes a lidar 112, point cloud data 208 of the environment of the vehicle 101 is acquired. The lidar data 208 includes a depth estimate of the 3D structures in the environment of the vehicle 101 from the image plane (i.e., the plane of the image 202). In one or more embodiments, the vehicle 101 is equipped with multiple lidars 112. Further, a BEV lidar image 210 is generated using the lidar data 208 acquired by the lidars 112. The BEV lidar image 210 provides range information of the 3D structures from the ground plane. To generate the BEV lidar image 210, multiple samples may be acquired from the lidar(s) 112 and spatially aligned using information from an inertial measurement unit (IMU).The BEV lidar image 210 is an image of the surroundings of the vehicle 101, where each pixel of the BEV lidar image 210 represents a detected object or the reflectivity of a detection. The range information from the lidar 112 includes the height of each pixel from the image(s) 202.

[0040] In the case where the vehicle 101 is equipped with the lidar(s) 112, the model of the surroundings of the vehicle 101 may be generated using the BEV camera image 204 and the BEV lidar image 210 using the elevation information acquired by the lidar 112. In one or more embodiments, the radar image 206 may not be required to generate the model if the lidar data 208 is available.

[0041] Fig. 3 illustrates a flowchart of a method 300 for estimating 3D models of static and dynamic structures in real time of a vehicle according to one or more embodiments. The method 300 includes, at block 302, generating a BEV camera image 204 of the surroundings of the vehicle 101 using the cameras 108. The BEV camera image 204 may be generated using one or more known techniques using a plurality of images captured by the camera 108.

[0042] The method 300 further includes, at block 304, determining whether the vehicle 101 is equipped with a lidar 112. If the vehicle 101 is not equipped with the lidar 112, the method 300 proceeds to generate the BEV radar image 206 of the surroundings of the vehicle 101 at block 306. The BEV radar image 206 may be generated using one or more known techniques using a plurality of radar data acquired by the radar 110.

[0043] In block 308, a merged image is created by aligning and merging the BEV radar image 206 and the BEV camera image 204. The BEV radar image 206 and the BEV camera image 204 are transformed, e.g., cropped, rotated, and translated, to align and match them. The alignment can be performed using known image registration techniques. Merging the images includes using the radar information for each pixel from the BEV radar image 206 as a first channel in the merged image and the color information from the BEV camera image 204 as the remainder of the channels. For example, if the merged image has four channels: red, green, blue, and depth (RGBD), the RGB channels can be populated with the color data from the BEV camera image 204, while the D channel can be populated with the range data from the BEV radar image 206.It is recognized that other combinations of channels may be used, and that in other examples, the merged image may contain different channels than the above example. The merged image is accordingly a tensor that provides the distance and color information of the surroundings of the vehicle 101, as represented by the pixels of the merged image.

[0044] In block 310, a neural network analyzes the merged image to detect 3D structures, such as buildings, trees, towers, and other such objects, in the environment of the vehicle 101. The neural network is an artificial neural network, such as a convolutional neural network (CNN), a feedforward neural network, a multi-layer perceptron, or any other such neural network. The neural network is pre-trained to detect the 3D structures, and in particular the edges of buildings, in the merged image, which includes the range data and the camera data in the data channels of the merged image. In this case, the range data is the data from the BEV radar image 206. Fig. 4 illustrates example edges 402 of buildings detected in the merged image 401 from an example scenario. It is recognized that in other embodiments, the merged image 401 and / or the detected edges 402 may be different from those illustrated.

[0045] In block 312, the coordinates for the detected edges 402 are determined in two dimensions, e.g., XY coordinates, relative to the vehicle 101. For example, the coordinates of the vehicle 101 (e.g., the center of the vehicle 101) are configured to be (0, 0) (i.e., the origin, with the coordinates of the edges 402 being determined with this reference). In one example, the coordinates are determined based on the number of pixels from the origin or in some other relationship to the number of pixels.

[0046] Further, in block 314, the coordinates in a third dimension, e.g., the Z coordinates, for the detected edges 402 are determined. The Z coordinates may be determined based on the range data in the merged image and / or in the radar image 206. As previously stated, the height of the edges 402 is calculated using the range data and the image 202 from the camera 108. In one example, the height information is stored as the value of the Z coordinates of the pixels represented by the XY coordinates in the merged images. Alternatively, the pixels in the BEV radar image 206 that correspond to the XY coordinates of the edges 402 are identified. The depth information from the BEV radar image 206 is used as the Z coordinates.

[0047] In block 316, models of the 3D structures 104 are generated by performing a plane fitting using the XYZ coordinates of the edges 402. The plane fitting may be performed using one or more known techniques, such as using the random sample matching (RANSAC) algorithm or other known plane fitting algorithms.

[0048] Alternatively, if the lidar 112 is available for the vehicle 101 at block 304, the method 300 continues to generate the BEV lidar image 210 using the lidar data of the surroundings of the vehicle 101 at block 318. The BEV lidar image 210 may be generated using one or more known techniques using multiple lidar data acquired by the lidar 112. The BEV lidar image 210 is generated, for example, by acquiring the point cloud (i.e., the lidar data 208 using the lidar 112). Further, the lidar data 208 is converted into a range image, where each pixel in the range image represents one (or no) detection from the lidar data 208. Such conversion includes comparing the elevation (Z) at each point in the lidar data 208 with respect to a ground plane. The plane of the vehicle 101 is used as the base plane for such calculations.The elevation relative to the ground plane for each point in the lidar data 208 can be calculated using trigonometric functions and the range data for that point.

[0049] It should be appreciated that in other embodiments, other techniques may be used to determine the elevation of each point in the lidar data 208. The elevation image is then projected onto the ground plane, e.g., using homography, to obtain the BEV lidar image 210. Each pixel in the BEV lidar image 210 represents the X, Y, Z coordinates of the 3D structures in the vicinity of the vehicle 101, where the X and Y coordinates are the 2D coordinates in the ground plane relative to the vehicle 101, e.g., with the vehicle 101 as the origin. The Z coordinates (i.e., elevation) may be represented by a grayscale value (or any other color) at the pixel.

[0050] Further, in block 320, the BEV lidar image 210 is analyzed to detect the edges 402 of the structures in the environment of the vehicle 101. The analysis may be performed by a neural network pre-trained to detect the edges based on the XYZ values ​​stored in the BEV lidar image 210, where the Z value represents the height stored as a color / intensity at each pixel coordinate (XY). Alternatively, the Z value represents the distance data for each pixel. Fig. 4 illustrates the detected edges 402 in a denoised BEV lidar image 403. The denoised BEV lidar image 403 is obtained by processing the BEV lidar image 210 using filters, such as spatial domain filters, transform domain filters, etc. Such filtering may be linear and / or nonlinear. Typical examples of such filters may include a mean filter, a Weiner filter, a median filter, a nonlinear threshold filter, etc.

[0051] Once the edges 402 are detected, the method 300 further includes detecting the XY coordinates of the edges at block 312. Furthermore, the Z coordinates of the edges are detected at block 314 using the range data. The range data may be obtained from sensors such as radar 110, lidar 112, etc. At block 316, plane fitting techniques are used to determine the models of the 3D structures 104.

[0052] The method 300 further includes, at block 322, reconfiguring the GNSS receiver 155 based on the 3D models of the structures in the environment of the vehicle 101. The reconfiguration may include changing one or more radio channels (i.e., the frequency) used by the GNSS receiver 155. The 3D models are used to model the errors in the radio signals that would otherwise not be detected by conventional GNSS receivers.

[0053] The embodiments described here enable the real-time estimation of 3D models of static and dynamic structures from a vehicle. The estimated 3D models can be applied to GNSS environmental modeling and mapping. Existing "camera-only" approaches can have errors in the estimated dimensions due to the lack of depth information. The technical solutions described here use radar and / or lidar to accurately detect the footprint of the structures and combine this information with the camera images to derive 3D models of the structures in the vehicle's environment.

[0054] The 3D models of structures, such as buildings, in the vicinity of the vehicle 101 are used to improve the performance of the GNSS receiver. Improving the operation of the GNSS receiver can be performed entirely online using the technical solutions described here, eliminating the need for a priori models, static maps of various areas, or continuously updating a map of the area in which the vehicle is traveling. The technical solutions described here therefore provide a practical application for improving GNSS operation by saving communication-intensive resources and operations.

[0055] In Fig. 5 generally shows a computer system 500 according to one embodiment. The computer system 500 may be used as any of the devices and / or appliances described herein, such as those with which the vehicle 101 is equipped. In one or more embodiments, the computer system 500 implements one or more methods described herein. The computer system 500 may be an electronic computing device that includes and / or employs any number and any combination of computing devices and networks using various communication technologies, as described herein. The computer system 500 may be easily scalable, expandable, and modular, with the ability to switch to different services or reconfigure some features independently of others. For example, the computer system 500 maya server, a desktop computer, a laptop computer, a tablet computer, or a smartphone. In some examples, the computer system 500 may be a cloud computing node. The computer system 500 may be described in the general context of executable computer system instructions, such as program modules, that are executed by a computer system. In general, the program modules may include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. The computer system 500 may be practiced in distributed cloud computing environments where tasks are executed by remote processing devices connected by a communications network. In a distributed cloud computing environment, the program modules may be located on both local and remote storage media of the computer system, including storage devices.

[0056] As in Fig. 5, computer system 500 includes one or more central processing units (CPU(s)) 501a, 501b, 501c, etc. (collectively or generically referred to as processor(s) 501). Processors 501 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Processors 501, also referred to as processing circuitry, are coupled to system memory 503 and various other components via a system bus 502. System memory 503 may include read-only memory (ROM) 504 and random access memory (RAM) 505. ROM 504 is coupled to system bus 502 and may include a basic input / output system (BIOS) that controls certain basic functions of computer system 500. The RAM is a random access memory coupled to the system bus 502 for use by the processors 501.System memory 503 provides temporary storage for the operations of the instructions during operation. System memory 503 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage systems.

[0057] Computer system 500 includes an input / output (I / O) adapter 506 and a communications adapter 507 coupled to system bus 502. I / O adapter 506 may be a small computer system interface (SCSI) adapter that communicates with a hard disk 508 and / or some other similar component. I / O adapter 506 and hard disk 508 are collectively referred to herein as mass storage 510.

[0058] The software 511 for execution on the computer system 500 may be stored in mass storage 510. The mass storage 510 is an example of a tangible storage medium readable by the processors 501, where the software 511 is stored as instructions for execution by the processors 501 to cause the computer system 500 to operate, for example, as described herein with respect to the various figures. The example computer program products and the execution of such instructions are discussed in more detail herein. The communications adapter 507 connects the system bus 502 to a network 512, which may be an external network, enabling the computer system 500 to communicate with other such systems. In one embodiment, a portion of the system memory 503 and the mass storage 510 jointly store an operating system, which may be any suitable operating system, such asthe z / OS or AIX operating system of the IBM Corporation to use the functions of the various systems implemented in . Fig. 5 components shown.

[0059] Additional input / output devices are shown connected to system bus 502 via a display adapter 515 and an interface adapter 516. In one embodiment, adapters 506, 507, 515, and 516 may be connected to one or more I / O buses connected to system bus 502 via an intermediate bus bridge (not shown). A display 519 (e.g., a screen or display monitor) is connected to system bus 502 through a display adapter 515, which may include a graphics controller to enhance the performance of graphics-intensive applications and a video controller. A speaker 523, input devices such as touch screens, buttons, and other such human-interactive devices (not shown), etc., may be connected via interface adapter 516, which may include, for example,may include a super I / O chip that integrates multiple device adapters into a single integrated circuit, interconnected to system bus 502. Suitable I / O buses for connecting peripheral devices, such as hard disk controllers, network adapters, and graphics adapters, typically include common protocols, such as Peripheral Component Interconnect (PCI). Thus, computer system 500, as shown in FIG. Fig. 5, processing capability in the form of processors 501 and storage capability including system memory 503 and mass storage 510, input means such as human-interactive devices (not shown), and output capability including speaker 523 and display 519.

[0060] In some embodiments, communication adapter 507 may transmit data using any suitable interface or protocol, such as, but not limited to, the Internet Small Computer System Interface. Network 512 may be, among others, a cellular network, a wireless network, a wide area network (WAN), a local area network (LAN), or the Internet. An external computing device may be connected to computer system 500 through network 512. In some examples, an external computing device may be an external web server or a cloud computing node.

[0061] It should be recognized that the block diagram according to Fig. 5 is not intended to indicate that the computer system 500 performs all of the Fig. 5. Instead, the computer system 500 may include any suitable fewer or additional components shown in Fig.5 (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Furthermore, the embodiments described herein with respect to computer system 500 may be implemented with any suitable logic, and the logic, as referred to herein, may, in various embodiments, include any suitable hardware (e.g., among others, a processor, an embedded controller, or an application-specific integrated circuit), software (e.g., among others, an application), firmware, or any suitable combination of hardware, software, and firmware.

[0062] When a relationship between a first and a second element is not expressly described as "direct" in the above disclosure, that relationship may be a direct relationship in which no other intervening elements are present between the first and second elements, but may also be an indirect relationship in which one or more intervening elements (either spatial or functional) are present between the first and second elements.

[0063] It should be appreciated that one or more steps within a method or process may be performed in a different order (or simultaneously) without altering the principles of the present disclosure. Furthermore, although each of the embodiments has been described above as having certain features, any one or more of those features described with respect to any embodiment of the disclosure may be implemented in any of the other embodiments and / or combined with the features of any of the other embodiments, even if that combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with one another remain within the scope of this disclosure.

[0064] While the above disclosure has been described with reference to exemplary embodiments, it will be recognized by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope thereof. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the disclosure without departing from its essential scope. Therefore, it is intended that the present disclosure not be limited to the specific embodiments disclosed, but will include all embodiments falling within its scope.

Claims

[1] Device with which a vehicle (101) is equipped, the device comprising: a receiver (155) of a navigation system; a camera (108); a memory (145); and a processor (140) coupled to the receiver (155) of the navigation system, the memory (145) and the camera (108), the processor (140) being configured to execute a method for detecting one or more three-dimensional structures in an environment of the vehicle (101) at runtime, the method comprising: Generating, by the processor (140), using the camera (108), a bird's eye view (BEV) camera image (204) of the surroundings of the vehicle (101), wherein the BEV camera image (204) includes two-dimensional coordinates of one or more structures in the surroundings; generating, by the processor (140), a BEV elevation image of the surroundings of the vehicle (101), the BEV elevation image providing an elevation of the one or more structures in the surroundings; Detecting, by the processor (140), one or more edges (402) of the one or more three-dimensional structures in the environment based on the BEV camera image (204) and the BEV elevation image; Generating, by the processor (140), models of the one or more three-dimensional structures in the environment of the vehicle (101) by plane fitting based on the edges (402) of the one or more three-dimensional structures; and Reconfiguring, by the processor (140), the receiver (155) of the navigation system based on the models of the one or more three-dimensional structures in the environment of the vehicle (101); wherein the BEV elevation image is generated based on the radar data acquired by a radar (110) of the vehicle (101); wherein the BEV elevation image is a BEV radar image (206) generated using the radar data; wherein detecting the edges (402) of the one or more three-dimensional structures comprises: Generating, by the processor (140), a merged image by merging the BEV camera image (204) and the BEV radar image (206) to add the elevation information as a channel for each pixel in the BEV camera image (204); and Detecting, by the processor (140), the edges (402) by inputting the merged image into an artificial neural network. [2] The apparatus of claim 1, wherein the BEV elevation image is generated based on the lidar data acquired by a lidar (112) of the vehicle (101). [3] The apparatus of claim 1, wherein the camera (108) includes a plurality of cameras (108) mounted on different sides of the vehicle (101). [4] The apparatus of claim 1, wherein reconfiguring the navigation system receiver (155) includes changing a frequency used by the navigation system receiver (155).

Images