Detecting 3D structural models while the vehicle is in motion

Abstract

A computer-implemented method for detecting one or more three-dimensional structures near a vehicle at runtime includes generating a bird's-eye view (BEV) camera image of the vicinity of the vehicle by a processor, the BEV camera image including the two-dimensional coordinates of the one or more nearby structures. The method also includes generating a BEV height image of the vicinity of the vehicle by the processor, the BEV height image providing the height of the one or more nearby structures. The method further includes detecting one or more edges of the three-dimensional structures based on the BEV camera image and the BEV height image. The method also includes generating a model of the three-dimensional structure by plane fitting based on the edges of the one or more three-dimensional structures. The method further includes reconfiguring a navigation system receiver based on the model of the three-dimensional structure.

Description

Technical Field

[0001] This disclosure relates to improving the performance of a Global Navigation Satellite System (GNSS) that can be equipped on a vehicle to provide autonomous geospatial positioning, particularly by using a combination of sensor devices such as cameras, radio detection and ranging (radar) devices and / or optical detection and ranging (LiDAR) devices to determine three-dimensional (3D) models of buildings and other structures from the vehicle. Background Technology

[0002] Vehicles equipped with GNSS receivers, such as cars, trucks, and ships, can provide navigation information to operators (human, autonomous, or semi-autonomous). Common GNSS systems include the Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Galileo, BeiDou, and other regional systems. GNSS receivers can provide navigation information with a specific level of accuracy; however, this accuracy is typically limited to open-air environments. For example, in open-air environments, GNSS receivers can achieve measurements with an accuracy of at least 1.5 meters (m). GNSS receivers are generally less accurate in urban areas (such as cities), especially in the presence of buildings and other structures. Buildings and other structures obstruct the path of GNSS receivers receiving signals from the GNSS system by blocking the receiver's line of sight in open-air environments, thereby creating a high multipath radio frequency (RF) signal environment. Summary of the Invention

[0003] A computer-implemented method for detecting one or more three-dimensional structures near a vehicle at runtime includes generating a bird's-eye view (BEV) camera image of the vicinity of the vehicle using a camera, the BEV camera image including the two-dimensional coordinates of the one or more nearby structures. The method also includes generating a BEV height image of the vicinity of the vehicle, the BEV height image providing the height of the one or more nearby structures. Furthermore, the method includes detecting one or more edges of the one or more nearby three-dimensional structures by the processor based on the BEV camera image and the BEV height image. The method further includes generating a model of the one or more three-dimensional structures near the vehicle by the processor through plane fitting based on the edges of the one or more three-dimensional structures. Finally, the method includes reconfiguring a navigation system receiver by the processor based on the model of the one or more three-dimensional structures near the vehicle.

[0004] According to one or more embodiments, a BEV height image is generated based on radar data captured from the vehicle's radar.

[0005] According to one or more embodiments, the BEV altitude image is a BEV radar image generated using radar data.

[0006] According to one or more embodiments, detecting the edges of one or more three-dimensional structures includes: generating a merged image by a processor by merging a BEV camera image and a BEV radar image to add height information as a channel to each pixel in the BEV camera image; and detecting the edges by the processor by inputting the merged image into an artificial neural network.

[0007] According to one or more embodiments, a BEV height image is generated based on lidar data captured from the vehicle's lidar.

[0008] According to one or more embodiments, the camera includes multiple cameras equipped on different sides of the vehicle.

[0009] According to one or more embodiments, reconfiguring a navigation system receiver includes changing the frequency at which the navigation system receiver is being used.

[0010] According to one or more embodiments, a device equipped in a vehicle includes a navigation system receiver, a camera, a memory, and a processor coupled to the navigation system receiver, the memory, and the camera. The processor is configured to perform a method for detecting one or more three-dimensional structures near the vehicle during runtime. The method includes generating a bird's-eye view (BEV) camera image of the vicinity of the vehicle, the BEV camera image including two-dimensional coordinates of the one or more nearby structures. The method also includes generating a BEV height image of the vicinity of the vehicle, the BEV height image providing the height of the one or more nearby structures. The method further includes detecting one or more edges of the one or more nearby three-dimensional structures based on the BEV camera image and the BEV height image. The method also includes generating a model of the one or more three-dimensional structures near the vehicle by plane fitting based on the edges of the one or more three-dimensional structures. The method further includes reconfiguring the navigation system receiver based on the model of the one or more three-dimensional structures near the vehicle.

[0011] According to one or more embodiments, a BEV height image is generated based on radar data captured from the vehicle's radar.

[0012] According to one or more embodiments, the BEV altitude image is a BEV radar image generated using radar data.

[0013] According to one or more embodiments, detecting the edges of one or more three-dimensional structures includes: generating a merged image by a processor by merging a BEV camera image and a BEV radar image to add height information as a channel to each pixel in the BEV camera image; and detecting the edges by the processor by inputting the merged image into an artificial neural network.

[0014] According to one or more embodiments, a BEV height image is generated based on lidar data captured from the vehicle's lidar.

[0015] According to one or more embodiments, the camera includes multiple cameras equipped on different sides of the vehicle.

[0016] According to one or more embodiments, reconfiguring a navigation system receiver includes changing the frequency at which the navigation system receiver is being used.

[0017] A computer program product includes a computer storage device comprising computer-executable instructions that, when executed by a processing unit, cause the processing unit to perform a method for detecting three-dimensional structures near a vehicle at runtime. The method includes generating a bird's-eye view (BEV) camera image of the vicinity of the vehicle, the BEV camera image including two-dimensional coordinates of one or more nearby structures. The method also includes generating a BEV height image of the vicinity of the vehicle, the BEV height image providing the height of the one or more nearby structures. The method further includes detecting one or more edges of the one or more nearby three-dimensional structures based on the BEV camera image and the BEV height image. The method also includes generating a model of the one or more three-dimensional structures near the vehicle by plane fitting based on the edges of the one or more three-dimensional structures. The method further includes reconfiguring a navigation system receiver based on the model of the one or more three-dimensional structures near the vehicle.

[0018] According to one or more embodiments, a BEV height image is generated based on radar data captured from the vehicle's radar.

[0019] According to one or more embodiments, the BEV altitude image is a BEV radar image generated using radar data.

[0020] According to one or more embodiments, detecting the edges of one or more three-dimensional structures includes: generating a merged image by a processor by merging a BEV camera image and a BEV radar image to add height information as a channel to each pixel in the BEV camera image; and detecting the edges by the processor by inputting the merged image into an artificial neural network.

[0021] According to one or more embodiments, a BEV height image is generated based on lidar data captured from the vehicle's lidar.

[0022] According to one or more embodiments, the camera includes multiple cameras equipped on different sides of the vehicle.

[0023] According to one or more embodiments, reconfiguring a navigation system receiver includes changing the frequency at which the navigation system receiver is being used.

[0024] The above features and advantages, as well as other features and advantages, will become apparent when considered in conjunction with the accompanying drawings and the following detailed description. Attached Figure Description

[0025] Other features, advantages, and details appear only by way of example in the detailed description below, which refers to the accompanying drawings, wherein:

[0026] Figure 1 A block diagram of an exemplary system for sensing the environment near a vehicle and dynamically detecting 3D structures according to one or more embodiments is depicted;

[0027] Figure 2 Examples of different types of data captured and acquired by a vehicle according to one or more embodiments are depicted;

[0028] Figure 3 A flowchart is depicted for a method for estimating 3D models of static and dynamic structures from a vehicle in real time, according to one or more embodiments;

[0029] Figure 4 Examples of images for edge detection according to one or more embodiments are depicted; and

[0030] Figure 5 A computing system for implementing one or more embodiments described herein is depicted. Detailed Implementation

[0031] The following description is exemplary in nature only and is not intended to limit this disclosure, its application, or use. In all the accompanying drawings, corresponding reference numerals denote the same or corresponding parts and features. As used herein, control module, module, control, controller, control unit, electronic control unit, processor, and similar terms refer to any one or more of the following: application-specific integrated circuits (ASICs), electronic circuits, central processing units (preferably microprocessors) and associated memory and storage devices (read-only memory (ROM), random access memory (RAM), electrically programmable read-only memory (EPROM), hard disk drives, etc.), graphics processing units or microcontrollers executing one or more software or firmware programs or routines, combinational logic circuits, input / output circuits and devices (I / O) and appropriate signal conditioning and buffering circuits, high-speed clocks, analog-to-digital (A / D) and digital-to-analog (D / A) circuits, and other components that provide the described functionality. Control modules may include various communication interfaces on vehicle controller area networks and on factory and service-related networks, including point-to-point or discrete lines and wired or wireless interfaces to networks including wide area networks and local area networks. The functions of the control module as described in this disclosure can be executed in a distributed control architecture among multiple networked control modules. Software, firmware, program, instruction, routine, code, algorithm, and similar terms refer to any controller-executable instruction set, including calibration, data structures, and lookup tables. The control module has a set of control routines executed to provide the described functions. These routines, for example, are executed by a central processing unit and are operable to monitor inputs from sensing devices and other networked control modules, and to execute control and diagnostic routines to control actuator operation. The routines can be executed at regular intervals during ongoing engine and vehicle operation. Alternatively, the routines can be executed in response to the occurrence of an event, a software call, or as needed via user interface input or request.

[0032] The technical solution described in this article addresses the technical challenges of improving the accuracy of Global Navigation Satellite System (GNSS) receivers. As previously mentioned, vehicles equipped with GNSS receivers, such as cars, trucks, and ships, can provide navigation information to operators (human, autonomous, or semi-autonomous). However, accuracy levels are typically limited in open-air environments, and GNSS receivers generally have lower accuracy in urban areas (e.g., cities), especially in the presence of buildings and other structures. Buildings and other structures obstruct the path of GNSS receivers receiving signals from the GNSS system by blocking the receiver's line of sight in open environments, thereby creating a high-multipath radio frequency (RF) signal environment.

[0033] The technical solution described herein accurately addresses this technical challenge by dynamically detecting three-dimensional (3D) structures (such as buildings) that can suppress navigation information received by GNSS receivers. The detection of 3D structures is performed entirely at runtime without prior information about such structural models, such as static maps containing the location and other information of the 3D structures. In other words, the technical solution described herein for dynamically detecting 3D structures does not require any prior information or any continuous map updates. Therefore, this technical solution is cost-effective compared to existing solutions that use static maps or continuously updated maps to detect such 3D structures. In one or more embodiments, for example, structures such as buildings detected at heights below (or above) a predetermined threshold are reported to reduce false alarms caused by structures such as curbs.

[0034] Now go to Figure 1 The block diagram illustrates an exemplary system 100 for sensing the environment near a vehicle and dynamically detecting 3D structures. System 100 includes a vehicle 101 equipped with a processor 140, a memory 145, a GNSS receiver 155, a camera 108, and a radar 110. In one or more embodiments, vehicle 101 is also equipped with a lidar 112. It should be understood that vehicle 101 may be equipped with additional devices and / or components not listed herein.

[0035] Processor 140 may be a microprocessor, graphics processing unit, digital signal processor, or any other processing unit. Processor 140 may execute one or more computer-executable instructions to perform one or more methods described herein. Such computer-executable instructions may be stored in memory 145. Memory 145 may store additional information used by processor 140 during the execution of the computer-executable instructions. For example, memory 145 may store temporary values, media (e.g., images, audio, video), and other information to be used during execution.

[0036] System 100 can use camera 108 and other sensors to capture images 202 of the vicinity of vehicle 101. Figure 2 Here, the “nearby” of vehicle 101 can be the environment within a predetermined distance (e.g., 50 meters, 100 meters, etc.) from the center of vehicle 101 (or any other reference point). Camera 108 is operable to capture image 202 within a field of view (FOV), which may include static and dynamic objects within the FOV. Image processing techniques can be used to locate and classify objects within the FOV. The FOV is typically associated with a driving scenario or a portion of a driving scenario that is restricted by the FOV.

[0037] Camera 108 may include multiple cameras placed inside and / or outside vehicle 101. For example, a subset of cameras 108 may be placed on the sides, top, front, and / or rear of vehicle 101, operable to capture image 202 or a series of images of a field of view near vehicle 101. In an exemplary embodiment, camera 108 may include a wide-angle camera. Sensor fusion may be performed to provide accurate detection, classification, tracking, etc., of external objects, as well as calculation of appropriate attributes such as relative velocity, acceleration, etc. In one or more embodiments, camera 108 facilitates the generation of a bird's-eye view (BEV) image 204 of the vicinity of vehicle 101.

[0038] BEV camera images 204 are generated using one or more known computer vision techniques. For example, a predetermined number of cameras 108 are mounted around vehicle 101 to cover the vicinity. The parameters of the cameras 108 are pre-calibrated so that images 202 captured from the cameras 108 can be warped into perspective views for integration. Overlapping regions of adjacent views in image 202 can be stitched together by using a dynamic programming method to align along the seams and then propagating an alignment warp field with a Wendland function. In this way, images 202 can be integrated into a single panoramic seamless BEV camera image 204. It is understood that other techniques can be used to generate BEV camera images 204, and the aspects of the technical solutions described herein are not limited to any particular technique used to generate BEV camera images 204.

[0039] Radar 110 is used to detect objects near vehicle 101. In one or more embodiments, radar data captured by radar 110 provides range information of 3D structures near vehicle 101. In one or more embodiments, vehicle 101 is equipped with multiple radars 110. Radar data captured by multiple radars 110 is used to generate a BEV radar image 206 near vehicle 101. Figure 2 Information from the inertial measurement unit (IMU) can be used to capture multiple scans from the radar and spatially aligned to generate the BEV radar image 206. Doppler information from the radar 110 can also be used to identify and remove moving objects from the multiple scans before generating the BEV radar image 206. The BEV radar image 206 is an image of the vicinity of the vehicle 101, where each pixel of the BEV radar image 206 represents a detected object, the confidence level of the detection, or a radar cross-section.

[0040] BEV camera image 204 and BEV radar image 206 are used together to generate a model including the 3D structure near vehicle 101. The generated model includes calculating the height of the nearby structure. The height of the target can be calculated using the distance R between the target and vehicle 101, which is provided by range information from radar 110. Furthermore, the height of the target is calculated based on the elevation angle E of the target, which can be determined from image 202. The height of each pixel representing the structure near vehicle 101 in camera image 202 can be determined using the corresponding distance R from the range data and the elevation angle E from camera image 202.

[0041] Based on a nearby model, the GNSS receiver 155 can be programmed to improve reception of navigation signals from GNSS. For example, programming may include changing one or more parameters associated with the receiver 155. Alternatively or additionally, based on a nearby model, navigation information for the vehicle 101 can be obtained from a different source, such as a telephone (not shown) associated with the vehicle 101 that can be connected via wired or wireless connection.

[0042] In embodiments where vehicle 101 includes LiDAR 112, point cloud data 208 near vehicle 101 is captured. LiDAR data 208 includes depth estimates of 3D structures near vehicle 101 from an image plane (i.e., the plane of image 202). In one or more embodiments, vehicle 101 is equipped with multiple LiDARs 112. Furthermore, the LiDAR data 208 captured by LiDAR 112 is used to generate a BEV LiDAR image 210. BEV LiDAR image 210 provides range information of 3D structures from the ground plane. Multiple scans from LiDAR 112 can be captured using information from an inertial measurement unit (IMU) and spatially aligned to generate BEV LiDAR image 210. BEV LiDAR image 210 is an image of the vicinity of vehicle 101, where each pixel of BEV LiDAR image 210 represents a detected object or detected reflectivity. Range information from LiDAR 112 includes the height of each pixel from image 202.

[0043] When vehicle 101 is equipped with lidar 112, a model of the vicinity of vehicle 101 can be generated using BEV camera image 204 and BEV lidar image 210 using height information captured by lidar 112. If lidar data 208 is available in one or more embodiments, lidar image 206 may not be necessary to generate the model.

[0044] Figure 3A flowchart depicts a method 300 for estimating a 3D model of a vehicle's static and dynamic structures in real time, according to one or more embodiments. Method 300 includes generating a BEV camera image 204 near a vehicle 101 using a camera 108 at block 302. The BEV camera image 204 can be generated using one or more known techniques employing multiple images captured by the camera 108.

[0045] Method 300 also includes determining in block 304 whether vehicle 101 is equipped with lidar 112. If vehicle 101 is not equipped with lidar 112, method 300 continues in block 306 to generate a BEV radar image 206 near vehicle 101. The BEV radar image 206 can be generated using one or more known techniques that utilize multiple radar data captured by radar 110.

[0046] In box 308, a merged image is generated by aligning and merging the BEV radar image 206 and the BEV camera image 204. The BEV radar image 206 and the BEV camera image 204 are transformed (e.g., cropped, rotated, translated) to align and match each other. Alignment can be performed using known image registration techniques. Image merging involves using radar information from each pixel of the BEV radar image 206 as the first channel in the merged image and color information from the BEV camera image 204 as the remaining channels. For example, if the merged image has four channels: red, green, blue, and depth (RGBD), the RGB channels can be filled with color data from the BEV camera image 204, and the D channels can be filled with range data from the BEV radar image 206. It should be understood that other combinations of channels can be used, and in other examples, the merged image may include channels different from those in the example described above. Therefore, the merged image is a tensor that provides range and color information about the vicinity of vehicle 101, represented by the pixels of the merged image.

[0047] In box 310, the neural network analyzes the merged image to detect 3D structures, such as buildings, trees, towers, and other such objects near vehicle 101. The neural network is an artificial neural network, such as a convolutional neural network (CNN), a feedforward neural network, a multilayer perceptron, or any other such neural network. The neural network is pre-trained to detect 3D structures, particularly the edges of buildings in the merged image, which includes range data and camera data from the data channels of the merged image. In this case, the range data is from the BEV radar image 206. Figure 4 Example edges 402 of buildings detected in the merged image 401 from the example scene are depicted. It should be understood that in other embodiments, the merged image 401 and / or the detected edges 402 may differ from those depicted.

[0048] In box 312, the coordinates of the detected edge 402 relative to vehicle 101 are determined in two dimensions (e.g., XY coordinates). For example, the coordinates of vehicle 101 (e.g., the center of vehicle 101) are configured as (0, 0) (i.e., the origin, and the coordinates of edge 402 are determined with reference to this). In the example, the coordinates are determined based on the number of pixels from the origin or in any other relationship with the number of pixels.

[0049] Furthermore, in box 314, the coordinates, such as the Z coordinates, of the third dimension of the detected edge 402 are determined. The Z coordinates can be determined based on range data from the merged image and / or the radar image 206. As previously described, the height of the edge 402 is calculated using the range data and the image 202 from camera 108. In this example, the height information is stored as the Z coordinate value of the pixel represented by the XY coordinates in the merged image. Alternatively, the pixel in the BEV radar image 206 corresponding to the XY coordinates of the edge 402 is identified. Depth information from the BEV radar image 206 is used as the Z coordinate.

[0050] In box 316, a model of 3D structure 104 is generated by performing plane fitting using the XYZ coordinates of edge 402. Plane fitting can be performed using one or more known techniques, such as the Random Sample Consensus (RANSAC) algorithm, or any other known plane fitting algorithm.

[0051] Alternatively, referring to box 304, if LiDAR 112 is available for vehicle 101, then in box 318, method 300 continues to use LiDAR data near vehicle 101 to generate BEV LiDAR image 210. BEV LiDAR image 210 can be generated using one or more known techniques employing multiple LiDAR data captured by LiDAR 112. For example, BEV LiDAR image 210 can be generated by capturing a point cloud (i.e., LiDAR data 208 using LiDAR 112). Furthermore, LiDAR data 208 is converted into a range image, where each pixel in the range image represents a detection (or non-detection) from LiDAR data 208. This conversion involves comparing the height (Z) at each point in LiDAR data 208 to a reference ground plane. The plane of vehicle 101 is used as the ground plane for this calculation. The height of each point in LiDAR data 208 to the reference ground plane can be calculated using trigonometric functions and the range data for that point.

[0052] It should be understood that in other embodiments, different techniques may be used to determine the height of each point in the lidar data 208. The height image is then projected onto a ground plane, for example using homography, to obtain the BEV lidar image 210. Each pixel in the BEV lidar image 210 represents the X, Y, Z coordinates of a 3D structure near the vehicle 101, where the X and Y coordinates are 2D coordinates relative to the ground plane of the vehicle 101, for example, with the vehicle 101 as the origin. The Z coordinate (i.e., height) may be represented by a grayscale (or any other color) value at the pixel.

[0053] Furthermore, in box 320, the BEV LiDAR image 210 is analyzed to detect edges 402 of structures near vehicle 101. The analysis can be performed by a neural network pre-trained to detect edges based on XYZ values ​​stored in the BEV LiDAR image 210, where the Z value, representing height, is stored as color / intensity at each pixel coordinate (XY). Alternatively, the Z value represents range data for each pixel. Figure 4 Edges 402 detected in the denoised BEV LiDAR image 403 are depicted. The denoised BEV LiDAR image 403 is obtained by processing the BEV LiDAR image 210 using filters such as spatial domain filters, transform domain filters, etc. Such filtering can be linear and / or nonlinear. Typical examples of such filters may include mean filters, Weiner filters, median filters, nonlinear threshold filters, etc.

[0054] Once edge 402 is detected, method 300 further includes detecting the XY coordinates of the edge in box 312. Additionally, in box 314, range data is used to detect the Z coordinate of the edge. Range data can be obtained from sensors such as radar 110, lidar 112, etc. In box 316, a plane fitting technique is used to determine the model of the 3D structure 104.

[0055] Method 300 also includes reconfiguring the GNSS receiver 155 based on a 3D model of the structure near vehicle 101 in block 322. Reconfiguration may include changing one or more radio channels (i.e., frequencies) being used by the GNSS receiver 155. The 3D model is used to model errors in the radio signals that would otherwise not be detected by a conventional GNSS receiver.

[0056] The embodiments described herein facilitate the real-time estimation of 3D models of static and dynamic structures from vehicles. The estimated 3D models can be applied to GNSS environment modeling and map building. Existing "camera-only" methods may produce errors in size estimation due to a lack of depth information. The technical solution described herein uses radar and / or lidar to accurately detect the coverage area of ​​structures and combines this information with camera images to derive a 3D model of the structures near the vehicle.

[0057] 3D models of structures near vehicle 101, such as buildings, are used to improve the performance of the GNSS receiver. The technical solution described herein can be used to improve the operation of the GNSS receiver entirely online, thus avoiding the need for continuous updates of prior models, static maps of individual areas, or maps of the areas the vehicle travels in. Therefore, the technical solution described herein improves the practical application of GNSS operation by saving communication-intensive resources and operations.

[0058] Now go to Figure 5 The document generally illustrates a computer system 500 according to an embodiment. The computer system 500 can be used as any of the apparatuses and / or devices described herein, such as those equipped in vehicle 101. In one or more embodiments, the computer system 500 implements one or more methods described herein. The computer system 500 can be an electronic computer framework that includes and / or employs any number of computing devices and networks utilizing various communication technologies and combinations thereof, as described herein. The computer system 500 can be readily extended, extensible, and modularized, with the ability to change for different services or reconfigure certain features independently of other features. The computer system 500 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, the computer system 500 can be a cloud computing node. The computer system 500 can be described in general terms of computer system executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. The computer system 500 can be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0059] like Figure 5As shown, the computer system 500 has one or more central processing units (CPUs) 501a, 501b, 501c, etc. (collectively or collectively referred to as processor 501). Processor 501 can be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Processor 501 (also referred to as processing circuitry) is coupled to system memory 503 and various other components via system bus 502. System memory 503 may include read-only memory (ROM) 504 and random access memory (RAM) 505. ROM 504 is coupled to system bus 502 and may include a basic input / output system (BIOS), which controls certain basic functions of computer system 500. RAM is a read-write memory coupled to system bus 502 for use by processor 501. System memory 503 provides temporary storage space for the operation of the instructions during operation. System memory 503 may include random access memory (RAM), read-only memory, flash memory, or any other suitable memory system.

[0060] Computer system 500 includes an input / output (I / O) adapter 506 and a communication adapter 507 coupled to a system bus 502. I / O adapter 506 may be a Small Computer System Interface (SCSI) adapter that communicates with a hard disk 508 and / or any other similar component. I / O adapter 506 and hard disk 508 are collectively referred to herein as mass storage 510.

[0061] Software 511 for execution on computer system 500 can be stored in mass storage 510. Mass storage 510 is an example of a tangible storage medium readable by processor 501, wherein software 511 is stored as instructions for processor 501 to execute in order to operate computer system 500, as described herein with respect to the various figures. Examples of computer program products and the execution of such instructions are discussed in more detail herein. Communication adapter 507 interconnects system bus 502 with network 512, which may be an external network, thereby enabling computer system 500 to communicate with other such systems. In one embodiment, system memory 503 and a portion of mass storage 510 jointly store an operating system, which may be any suitable operating system, such as IBM's z / OS or AIX operating system, to coordinate... Figure 5 The functions of the various components are shown.

[0062] Additional input / output devices are shown connected to the system bus 502 via display adapter 515 and interface adapter 516. In one embodiment, adapters 506, 507, 515, and 516 may be connected to one or more I / O buses connected to the system bus 502 via an intermediate bus bridge (not shown). A display 519 (e.g., a screen or display monitor) is connected to the system bus 502 via display adapter 515, which may include a graphics controller to improve the performance of graphics-intensive applications and a video controller. Speakers 523, input devices such as touchscreens, buttons, and other such human-computer interaction devices (not shown) may be interconnected to the system bus 502 via interface adapter 516, which may include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit. Suitable I / O buses for connecting peripheral devices (e.g., hard disk controllers, network adapters, and graphics adapters) typically include common protocols such as Peripheral Component Interconnect (PCI). Therefore, as Figure 5 The computer system 500 configured includes processing capabilities in the form of a processor 501, storage capabilities including system memory 503 and mass storage 510, input devices such as human-computer interaction devices (not shown), and output capabilities including a speaker 523 and a display 519.

[0063] In some embodiments, the communication adapter 507 may use any suitable interface or protocol (such as an Internet Small Computer System Interface) to transmit data. The network 512 may be a cellular network, radio network, wide area network (WAN), local area network (LAN), or the Internet. External computing devices may connect to the computer system 500 via the network 512. In some examples, the external computing device may be an external network server or a cloud computing node.

[0064] It should be understood that, Figure 5 The block diagram is not intended to indicate that the computer system 500 will include Figure 5 All the components shown. Conversely, computer system 500 may include... Figure 5 Any suitable fewer or additional components not shown herein (e.g., additional memory components, embedded controllers, modules, other network interfaces, etc.). Furthermore, the embodiments described herein with respect to computer system 500 can be implemented with any suitable logic, wherein, in various embodiments, the logic referred to herein may include any suitable hardware (e.g., processor, embedded controller, or application-specific integrated circuit, etc.), software (e.g., application programs, etc.), firmware, or any suitable combination of hardware, software, and firmware.

[0065] Unless explicitly described as “direct,” the relationship between the first and second elements described in the above disclosure can be a direct relationship in the absence of any other intermediate elements between the first and second elements, or an indirect relationship in the presence of one or more intermediate elements (spatially or functionally) between the first and second elements.

[0066] It should be understood that one or more steps within a method or process may be performed in a different order (or simultaneously) without altering the principles of this disclosure. Furthermore, although each embodiment has been described above as having certain features, any one or more of those features described with respect to any embodiment of this disclosure may be implemented in and / or combined with features of any other embodiment, even if such combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and substitutions of one or more embodiments for each other remain within the scope of this disclosure.

[0067] Although the above disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made and elements can be substituted with equivalents without departing from the scope of the invention. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from its essential scope. Therefore, it is intended that this disclosure be limited to the specific embodiments disclosed, but will include all embodiments falling within its scope.

Claims

1. A device installed in a vehicle, the device comprising: Navigation system receiver; camera; Memory; as well as A processor, coupled to a navigation system receiver, a memory, and a camera, wherein the processor is configured to perform a method for detecting one or more three-dimensional structures near the vehicle during runtime, the method comprising: The processor uses a camera to generate a bird's-eye view (BEV) camera image of the vicinity of the vehicle, which includes the two-dimensional coordinates of one or more nearby structures; The processor generates a BEV height image near the vehicle, which provides the height of one or more nearby structures. The processor detects one or more edges of one or more nearby 3D structures based on BEV camera images and BEV height images; The processor generates a model of one or more three-dimensional structures near the vehicle by plane fitting based on the edges of one or more three-dimensional structures; and The processor reconfigures the navigation system receiver based on a model of one or more three-dimensional structures near the vehicle; Reconfiguring the navigation system receiver includes changing the frequency at which the navigation system receiver is using.

2. The device according to claim 1, wherein, The BEV height image is generated based on radar data captured from the vehicle's radar.

3. The device according to claim 2, wherein, The BEV altitude image is a BEV radar image generated using the radar data.

4. The device according to claim 3, wherein, Detecting the edges of one or more 3D structures includes: The processor generates a merged image by combining BEV camera images and BEV radar images, adding height information as a channel to each pixel in the BEV camera image; and The processor detects edges by feeding the merged images into an artificial neural network.

5. The device according to claim 1, wherein, The BEV height image is generated based on lidar data captured from the vehicle's lidar.

6. The device of claim 1, wherein the camera comprises a plurality of cameras mounted on different sides of the vehicle.

7. A computer program product comprising a computer storage device, the computer storage device including computer-executable instructions, which, when executed by a processor, cause the processor to perform a method for detecting one or more three-dimensional structures near a vehicle during runtime, the method comprising: The processor uses a camera to generate a bird's-eye view (BEV) camera image of the vicinity of the vehicle, which includes the two-dimensional coordinates of one or more nearby structures; The processor generates a BEV height image near the vehicle, which provides the height of one or more nearby structures. The processor detects one or more edges of one or more nearby 3D structures based on BEV camera images and BEV height images; The processor generates a model of one or more three-dimensional structures near the vehicle by plane fitting based on the edges of one or more three-dimensional structures; as well as The processor reconfigures the navigation system receiver based on a model of the structure near the vehicle; Reconfiguring the navigation system receiver includes changing the frequency at which the navigation system receiver is using.

8. The computer program product according to claim 7, wherein, The BEV height image is generated based on radar data captured from the vehicle's radar.

9. The computer program product according to claim 8, wherein, The BEV altitude image is a BEV radar image generated using the radar data.

Images