Generating vehicle top view with object detection-based image reconciliation

Abstract

A composite top view of the area around the vehicle is generated using the image reconciliation and proximity sensor data. Images from various vehicle cameras are received, where at least two of the cameras may see the same portion of the environment in the overlap region. The image is segmented into segments. The overlap region includes some of the segments of the one image and some of the segments of the other region to define overlapping segments. Sensor data is received from a plurality of proximity sensors, where the sensor data is indicative of a location of an object external to the vehicle. The color and / or brightness of the image is reconciled based at least in part on the sensor data. In a particular embodiment, the weights are placed on the color and / or brightness of the overlapping segments associated with the location of the object. Alternatively, the overlapping segments may be removed from the harmony.

Description

Technical Field

[0001] This disclosure relates to a method and system for generating synthetic images of the environment outside a motor vehicle using techniques based on the detection of objects on the exterior of the vehicle. Background Technology

[0002] To assist with parking or maneuvering at low speeds, vehicles can be equipped with the ability to generate a top-down view on the vehicle's displays (e.g., infotainment screens). This top-down view, also known as a top-down view, overhead view, or bird's-eye view, is generated by stitching together images from various cameras located around the vehicle. The images are processed, analyzed, and stitched together to provide a composite but positionally accurate top-down view of the vehicle and its 360-degree surroundings; to the driver, it appears as a single image taken directly above the vehicle and its environment. Summary of the Invention

[0003] According to one embodiment, a method for generating a top-down view of a region surrounding a vehicle is provided. The method includes: receiving images from a plurality of vehicle cameras, wherein each vehicle camera is configured to observe a corresponding area outside the vehicle, and portions of some of these areas overlap portions of other areas to define an overlapping region; segmenting each image into a corresponding plurality of segments, wherein the overlapping region includes segments from one image and segments from another image to define overlapping segments; reconciling the images based on at least one of brightness values ​​and color values ​​within the overlapping segments to generate the top-down view of the region surrounding the vehicle; receiving sensor data generated from a plurality of proximity sensors, wherein the sensor data indicates the location of an object outside the vehicle; associating a set of overlapping segments with the location of the object; adjusting weights associated with the set of overlapping segments; and reconciling the images with the adjusted weights associated with the set of overlapping segments.

[0004] In another embodiment, a system for generating a top-down view of a region surrounding a vehicle is provided. The system includes a plurality of vehicle cameras, each configured to observe a corresponding area outside the vehicle, portions of some of these areas overlapping portions of other areas to define an overlapping region. The system also includes a plurality of proximity sensors disposed around the vehicle. The system further includes at least one processor coupled to the vehicle cameras and proximity sensors, wherein the at least one processor is programmed to: receive images generated by the vehicle cameras; segment each image into a corresponding plurality of segments, wherein the overlapping region includes some of the segments from one image and some of the segments from another image to define overlapping segments; process sensor data generated by the proximity sensors to determine the location of an object outside the vehicle; associate a set of overlapping segments with the determined location of the object; adjust weights associated with the set of overlapping segments, the set of overlapping segments being associated with the determined location of the object; and harmonize the images based on at least one of brightness and color values ​​within the overlapping segments to generate a top-down view of the region surrounding the vehicle, wherein the harmonization of the images is further based on the adjusted weights.

[0005] In another embodiment, a method for reconciling images based on proximity sensor data includes: receiving images from a plurality of vehicle cameras, wherein each vehicle camera is configured to observe a corresponding area outside the vehicle, and wherein a portion of each area overlaps with another portion of an adjacent area to define an overlapping area; segmenting the images into a plurality of associated segments, wherein the overlapping area includes some segments of one image and some segments of another image to define overlapping segments; receiving sensor data generated from a plurality of proximity sensors, wherein the sensor data indicates the location of objects outside the vehicle; and reconciling the images based on at least one of brightness values ​​and color values ​​within the overlapping segments to generate a composite view of an area around the vehicle, wherein the reconciliation is further based on the sensor data. Attached Figure Description

[0006] Figure 1 A schematic diagram of a vehicle having various cameras and proximity sensors according to one embodiment is shown.

[0007] Figure 2 A schematic representation of an image region captured by various vehicle cameras is shown according to one embodiment.

[0008] Figure 3 Another schematic representation of an image area captured by various vehicle cameras according to one embodiment is shown.

[0009] Figure 4 The diagram shows micro-blocks with markings. Figure 3 A schematic diagram.

[0010] Figure 5 A two-dimensional (2D) obstacle map generated based on data from a proximity sensor is shown according to one embodiment.

[0011] Figure 6 A schematic diagram of an image region having micro-patterns associated with the detected locations of objects based on a 2D obstacle map is shown according to one embodiment.

[0012] Figure 7 According to the embodiments Figure 4 The diagram illustrates how a sign or marker (indicated by an X) is associated with some of the microblocks, which are linked to the location of a detected object.

[0013] Figure 8 An example flowchart of a method for harmonizing an image based on the detected object location according to one embodiment is shown.

[0014] Figure 9A A top view of a vehicle that does not achieve the reconciliation disclosed herein is shown. Figure 9B The same view showing the implementation of reconciliation is shown.

[0015] Figure 10A Another top view of the vehicle that does not achieve the reconciliation disclosed herein is shown. Figure 10B The same view showing the implementation of reconciliation is shown. Detailed Implementation

[0016] This document describes embodiments of the present disclosure. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily drawn to scale; some features may be exaggerated or minimized to show detail of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but rather serve only as a representative basis for teaching those skilled in the art to employ the embodiments in various ways. As will be understood by those skilled in the art, various features shown and described with reference to any of the drawings may be combined with features shown in one or more other drawings to produce embodiments not explicitly shown or described. The combinations of features shown provide representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desired for a particular application or implementation.

[0017] As explained in the background section, motor vehicles (e.g., cars, trucks, vans, SUVs, etc.) may be equipped with camera systems that generate a top-down view on the vehicle's displays (e.g., infotainment screens, dashboard units, tablets, mobile devices, phones, etc.). This top-down view, also known as a top-down view, overhead view, or bird's-eye view, is generated by stitching together images from various cameras located around the vehicle. The images are processed, analyzed, and stitched together to provide a composite but positionally accurate top-down view of the vehicle and its 360-degree surroundings; to the driver, it appears as a single image taken directly above the vehicle and its surroundings. This view can assist the driver in parking in tight spaces or maneuvering at low speeds.

[0018] These camera systems typically consist of four or more cameras surrounding the vehicle, mounted at or near the front, rear, and either side. Each camera can have its own independent image signal processing before the images are stitched together. Therefore, cameras can have different exposures, contrasts, brightness, etc. Because each camera faces a different direction and sees different areas around the vehicle, the brightness and color values ​​can vary slightly for each camera. These differences in color (chroma) and brightness (luminance) can negatively impact the top-down view. For example, if one camera has a significantly different brightness or color exposure compared to another camera, the resulting top-down view may not be aesthetically pleasing due to the corresponding noticeable changes in brightness or color at certain points.

[0019] Therefore, a harmonic technique is applied when generating a top view to harmonize and merge camera images regarding brightness and color. Harmonization algorithms are commonly used in image processing to harmonize and merge the brightness or luminance and chromaticity of camera images. Typically, harmonicization works to perform color and brightness transformations between images from different cameras. When forming a composite top view, the color and brightness statistics of the images can be averaged and matched. For example, the region of the histogram that can best match between two images is predicted or determined, and then the colors are adjusted for both images so that the colors are consistent in the composite view. German Patent Publication No. DE102016121755 (“'755 Publication”), entitled “Method For Determining A Composite Image Of ASurrounding Area Of A Motor Vehicle With Adaptation Of Brightness And / Or Color, Camera System And Power Vehicle”, is incorporated herein by reference in its entirety. '755 Publication describes a harmonic technique in which a calculated histogram is used as input to generate correction values ​​for each camera and each corresponding Y, U, and V channel. Harmonization can be particularly helpful in areas seen by two or more cameras (in other words, areas where views from vehicle cameras overlap). German Patent Publication No. DE102018110597 (“'597 Publication”), entitled “Method for Image Harmonization, Image Processing Device, Camera System and Motor Vehicle,” is also incorporated herein by reference in its entirety. '597 Publication describes a harmonization technique in which color analysis of regions of interest from multiple images is performed. Color correction values ​​are determined based on the color analysis and applied to a portion of the image, thereby adjusting the colors of that image.

[0020] However, existing technology systems do not adequately account for objects that may appear in the field of view and cause color or brightness distortion. For example, a vehicle may perform many different maneuvers, parking sessions, driving around parking spaces, entering and exiting tunnels, etc., during which the vehicle's camera system is exposed to a wide range of lighting environments and a high dynamic range of scene content. This can produce significant differences in brightness and color among objects seen by one or more cameras. Key object scene content that the camera system needs to accurately reproduce in variable lighting environments includes grassy areas, lighting conditions in garages or tunnels, traffic cones, other vehicles, etc. If techniques that distort the appearance of these images are used, it may result in a top-down view that displays objects with brightness or color that is very different from what the human eye actually sees.

[0021] Therefore, according to the various embodiments described herein, the vehicle camera system includes a harmonic processing technique (also known as a harmonic algorithm or harmonic correction algorithm) capable of detecting that the vehicle is surrounded by colored textures or strong colors from objects. The primary goal of the harmonic processing technique is to accurately reproduce the vehicle's surrounding environment, where harmonic correction originates from the brightness and color cast (in an absolute and relative camera sense) of camera limitations (e.g., lenses, image sensors, etc.), rather than from the brightness and color cast of objects within the camera's field of view.

[0022] Figure 1 A schematic diagram of a vehicle 10 according to one embodiment is shown, illustrated herein in top view. Vehicle 10 is a passenger car, but may be other types of vehicles, such as trucks, vans, or sports utility vehicles (SUVs). Vehicle 10 includes a camera system 12, which includes an electronic control unit (ECU) 14 connected to a plurality of cameras 16a, 16b, 16c, and 16d. Typically, ECU 14 includes one or more processors programmed to process image data associated with cameras 16a-d and generate a composite top view on a vehicle display 18. Furthermore, as will be further described below, vehicle 10 includes a plurality of proximity sensors (e.g., ultrasonic sensors, radar, sonar, lidar, etc.) 19. Proximity sensors 19 may be connected to their own designated ECU, which forms a sensor map of objects outside the vehicle. Alternatively, proximity sensors may be connected to ECU 14.

[0023] The ECU disclosed herein may be more generally referred to as a controller. For example, in the case of the ECU in camera system 12, the ECU may be able to receive image data from each camera (or their respective processors), process information, and output instructions to combine the image data to generate a composite top-down view. In the case of the ECU associated with proximity sensor 19, the ECU may be able to receive sensor data from each proximity sensor (or their respective processors), process information, and output a sensor map of objects around the vehicle; the ECU may also be able to send an alert to the driver during parking maneuvers, which may warn the driver of the approach of a detected object. In this disclosure, the terms "controller" and "system" may refer to a portion of, or include, processor hardware (shared, dedicated, or grouped) that executes code and memory hardware (shared, dedicated, or grouped) that stores the code executed by the processor hardware. The code is configured to provide the characteristics of the controller and system described herein. In one example, the controller may include a processor, memory, and non-volatile storage. The processor may include one or more devices selected from microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in memory. The memory may include a single memory device or multiple memory devices, including, but not limited to, random access memory (“RAM”), volatile memory, non-volatile memory, static random access memory (“SRAM”), dynamic random access memory (“DRAM”), flash memory, cache memory, or any other device capable of storing information. Non-volatile memory may include one or more persistent data storage devices, such as hard disk drives, optical disk drives, magnetic tape drives, non-volatile solid-state devices, or any other device capable of persistently storing information. The processor may be configured to read from memory and execute computer-executable instructions embodying one or more software programs residing in the non-volatile memory. Programs residing in non-volatile memory may include, or be part of, an operating system or application, and may be compiled or interpreted from computer programs created using various programming languages ​​and / or technologies, including, but not limited to, Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL, individually or in combination. The computer-executable instructions of the program may be configured to cause the harmonic techniques and algorithms described herein when executed by a processor.

[0024] exist Figure 1In the illustrated embodiment, cameras 16-d are located around different quadrants of the vehicle, but more than four cameras can be provided in the camera system 12. Each camera 16a to 16d may have a fisheye lens to obtain an image with a magnified field of view, indicated by boundary lines 20a to 20d. In the example, the first camera 16a faces the area in front of the vehicle and captures an image with a field of view indicated by boundary line 20a. Therefore, the first camera 16a may be referred to as the front camera. The second camera 16b faces the area behind the vehicle and captures an image with a field of view indicated by boundary line 20b. Therefore, the second camera 16b may be referred to as the rear camera. The third camera 16c faces the area to the left of the vehicle and captures an image with a field of view indicated by boundary line 20c. Therefore, the third camera 16c may be referred to as the left camera or left-side camera. The third camera 16c may also be mounted on or near the left-side rearview mirror of the vehicle and may therefore be referred to as the left-side (ML) rearview mirror camera. The fourth camera 16d faces the area to the right of the vehicle and captures an image with a field of view indicated by boundary line 20d. Therefore, the fourth camera 16d can be referred to as the right-side camera or the right-side camera. The fourth camera 16d can also be mounted on or near the right-side rearview mirror of the vehicle, and thus can be referred to as a rearview mirror right (MR) camera. Images (or associated image data) derived from cameras 16a-d can be processed by ECU 14 (e.g., stitched together, distorted, combined, and harmonized) to generate a composite top view on the vehicle display 18.

[0025] Figure 2 A schematic representation of the image regions captured by each of the cameras 16a to 16d is shown. Here, the image regions associated with each of the cameras 16a to 16d are as follows: the front camera 16a produces an image of the front view (FV) region; the rear camera 16b produces an image of the rear view (RV) region; the left camera 16c or the left rearview mirror camera produces an image of the left side view region (ML); and the right camera 16d or the right rearview mirror camera produces an image of the right side view region (MR).

[0026] Furthermore, the image regions FV, ​​RV, ML, and MR are divided into several corresponding regions of interest (ROIs). These ROIs can be fixed and configured according to each camera. Images from each of the front and rear cameras 16a-b can be divided into two ROIs, such as FV-A, FV-B, RV-A, and RV-B. Images from each of the side cameras 16c-d can be divided into four ROIs, such as ML-A, ML-B, ML-C, ML-D, MR-A, MR-B, MR-C, and MR-D. The number of ROIs can be more or less. Figure 2The numbers shown are arbitrary and depend on, for example, the texture size and its relative proportion to the image sizes of other cameras. For instance, to create a spherical view, the number of ROIs can differ from those shown here.

[0027] Some overlap exists within these regions of interest. Note the following: Figure 1 In the diagram, the boundary lines 20a-d intersect in various parts. This indicates that a portion of each camera's field of view overlaps with a portion of another camera's field of view. This is also... Figure 2 As shown in the diagram. For example, regions of interest FV-A and ML-A overlap. In other words, the area of ​​the external environment of the vehicle seen by the left-side camera 16c, as shown in region ML-A, is also seen by the front camera 16a, as shown in region FV-A. This can be seen in... Figure 2 As can be seen, the scene of the environment that can be seen in region ML-A is the same as the scene of the environment that can be seen in region FV-A. The same overlap occurs between regions FV-B and MR-A, between regions RV-B and MR-D, and between regions RV-A and ML-D.

[0028] In contrast, some ROIs may not overlap. For example, the scenes depicted in regions ML-B, ML-C, MR-B, and MR-C do not overlap with any views from any other camera.

[0029] Image harmonization compares the luminance and color components of identical scene content (i.e., within overlapping regions of interest). In these overlapping ROIs, the scene content is identical, but the luminance and color of that content can appear different between different cameras. For example, the scene content shown in region ML-A may be brighter than the same scene content shown in region FV-A. This is because each camera adjusts its parameters (e.g., via its own designated processor) to attempt and accurately reproduce the entire scene it sees. For instance, the front camera 12a may be positioned more towards the sky and see a bright sun in its field of view; therefore, the front camera 12a correspondingly reduces its exposure time to avoid an oversaturated image. On the other hand, the left camera 16c may be positioned so that it primarily sees the dark ground and correspondingly increases its exposure to receive enough light to reproduce the scene. The luminance and color components of each overlapping ROI are analyzed and compared by two or more cameras in the harmonization algorithm. The harmonization algorithm attempts to match the luminance and color components of overlapping regions of interest (e.g., FV-A and ML-A) to harmonize the image. For example, the image harmonization described herein can be performed by ECU 14.

[0030] To improve the robustness of image harmonicization, each region of interest (e.g., FV-A, FV-B, ML-A, ML-B, ML-C, etc.) can be further divided into sub-regions, also known as micro-blocks or segmentations. This is in Figure 2 The diagram illustrates that each region of interest (ROI) is divided into four sub-regions or micro-blocks. Of course, the number of micro-blocks per ROI does not need to be four, and the number, shape, or size of ROIs do not need to be the same; shapes or numbers can be tailored to specific applications. However, for simplicity, each micro-block is represented as a square here. Image harmonics can be configured to calculate or estimate the luminance (luminosity) and color (chromaticity) of each micro-block in each overlapping ROI. This can produce Y, U, and V estimates for each micro-block. Image harmonics can then adjust the image generated by the camera so that the luminance and color in the micro-blocks of each overlapping ROI are matched or more aligned.

[0031] As mentioned earlier, this matching is expected to be relatively reliable when there are no objects in the field of view. However, sometimes objects in the surrounding environment lie within these overlapping regions of interest. When camera images of such objects are acquired from different viewpoints, the overlapping regions of interest will no longer necessarily have the same scene content due to the different camera perspectives of the objects. For example, this is in Figure 3 Example in. Figure 3 Another schematic representation of the image regions captured by each camera 16a-d is illustrated, which produces FV, ​​RV, ML, and MR images. Figure 3 As can be seen, in the upper left corner, the object (in this case, the red vehicle) is located in the overlapping region of interest (ROI) between the front-view (FV) and left-view (ML) images. The scene content is not identical in this overlapping ROI, labeled "A". When the scene content is not identical, blurring can be introduced into the reconciliation image matching method. Micro-patches help solve this problem by comparing micro-patches of each ROI with corresponding micro-patches of the overlapping ROI. However, this method may be unreliable when the object covers a large area in the overlapping region. Therefore, according to the various embodiments disclosed herein, the camera system is able to adjust its reconciliation technique based on the presence of detected objects. As will be explained, when an object is detected to be present in one of these overlapping ROIs, the camera system can ignore, exclude, or reduce the weight of the influence of these ROIs (or sub-regions) in the reconciliation process.

[0032] According to an embodiment, the region of interest corresponding to the micro-block (e.g., in) is utilized. Figure 3In some cases, the confidence scores or compatibility levels of the regions of interest (ML-A and FV-A, where objects are detected) are used to weight micro-patches. The confidence score is based on the degree to which object content is present in overlapping ROI micro-patches. If no object is present, the corresponding overlapping micro-patches will be weighted equally (e.g., because they have the same or very similar scene content). However, in some cases, objects are detected in those ROIs, and due to the perspective projection of each camera, the overlapping ROIs will not have the same scene content. And in other cases, objects may only exist in a small portion of a micro-patch; based on this knowledge, the system can assign a confidence score (e.g., between 0 and 1) to each micro-patch within that ROI.

[0033] Figure 4 This is illustrated with an example. Here, numerical symbols are assigned to each microblock (e.g., 0, 1, 2, or 3). Of course, if more microblocks exist in each ROI, more numerical symbols can be provided. In this scene with an object (e.g., a red vehicle), the microblocks labeled 0 and 2 in zone ML-A are almost entirely filled with the body of the red vehicle. Moreover, the corresponding microblocks 0 and 2 in zone FV-A have different scene content due to the presence of the object.

[0034] In such a scenario, the system can rely on objects detected by one or more of the proximity sensors 19. Typically, the ECU 14 can modify its harmonic algorithm based on the presence of objects detected in regions corresponding to overlapping ROIs. For example, the ECU 14 can receive sensor data indicating the presence of objects outside the vehicle generated from one or more proximity sensors, determine that the object is located in a region corresponding to an overlapping ROI (e.g., FV-A and ML-A) in the camera system, and adjust the harmonic weights associated with the micro-blocks that detected the object.

[0035] In one embodiment, to determine whether a detected object is located at a position corresponding to an overlapping ROI, ECU14 (or other vehicle ECU) may rely on an obstacle map generated from sensor data derived from a proximity sensor. An example of the generated obstacle map is... Figure 5 As shown in the diagram. Typically, obstacle mapping converts proximity sensor data into a two-dimensional (2D) map indicating the location of detected objects. Figure 5In the example shown, 28 proximity sensors are provided around the vehicle, labeled R1-R8, S1-S6, T1-T6, and F1-F8. Of course, the number of proximity sensors can be more or less than 28. If an object has been detected by at least some of the proximity sensors (e.g., R7-R8, T3-T6 in this case), the 2D obstacle map generated by the ECU visualizes the object's presence as "OBJ_1". If another object has been detected by at least some of the proximity sensors (e.g., R1-R2, S3-S6 in this case), the 2D obstacle map generated by the ECU visualizes the object's presence as "OBJ_2".

[0036] This information can be overlaid or otherwise correlated with image data of the vehicle's surrounding environment. Using image data from cameras 16a-d, the system can calculate the positions of regions of interest and micro-patterns in the image data, corresponding to the coordinates of a 2D obstacle map generated by the proximity sensors relative to the vehicle. This can be done once within a calibration procedure, thereby reconciling the 2D obstacle map and the surround-view multi-camera texture using the camera positions and viewport parameters. For example, the 2D obstacle map can provide the system with the two-dimensional coordinates (e.g., X and Y) of the area surrounding the vehicle and any detected objects within it. These coordinates can correspond to... Figure 4 Interpreting micro-blocks. When an object appears in a 2D obstacle map, the ECU can accurately know which micro-blocks are associated with the object's location, thus enabling the ECU to determine which micro-blocks contain the object.

[0037] According to an embodiment, the result of this association is Figure 6 As shown in the image. Figure 6 Showing something similar Figure 4 The schematic diagram shows views seen by vehicle cameras 16a to 16d arranged in a composite manner, with micro-blocks serving as part of various regions of interest. In this embodiment, there is a greater... Figure 4 More micro-blocks are shown, but again, this is just an example. As you can see, now in Figure 6 The synthetic micro-particle map shows the composition of Figure 5 The object detected by the proximity sensor (“OBJ_1”) is shown. Some micro-blocks overlapping both the FV and ML image regions contain the object. Another object (“OBJ_2”) also detected by the proximity sensor is also present. Figure 5 The 2D obstacle map is shown and converted to Figure 6 Multiple micro-blocks within.

[0038] Using this information, certain micro-blocks can then be weighted or removed from the harmonic algorithm. This can be done by applying weighting factors to reduce the influence of micro-blocks based on the detected object's location / position and associated confidence score. Figure 7 An "X" is shown placed within a set of micro-blocks where objects are detected in the overlapping ROI. This set of micro-blocks can be referred to as labeled or marked micro-blocks. For example, micro-blocks 0 and 2 from FV-A and ML-A are labeled. These labeled micro-blocks can be less weighted in the reconciliation algorithm or removed from the reconciliation algorithm. Micro-blocks in the MR-B region of interest are also marked or marked, namely boxes 1 and 3 from MR-B. The corresponding overlapping boxes 1 and 3 from FV-B are also marked because they are sampled in areas where obstacles have already been detected via a 2D obstacle map.

[0039] Once micro-patterns containing detected objects are identified or marked, several methods can be used in the reconciliation algorithm. In one embodiment, brightness and color matching in the reconciliation algorithm can ignore these marked micro-patterns. This way, the brightness and color appearing in these micro-patterns will not affect the reconciliation algorithm. This results in a smoother, more natural-looking composite top-down view, especially in areas where the view transitions from one camera's view to another's view.

[0040] The exclusion of these labeled micro-patches can be done in a binary manner; labeled micro-patches can be excluded, while unlabeled micro-patches are not excluded from the reconciliation. In another embodiment, labeled micro-patches are not completely excluded, but rather weighted. Confidence scores can be used to define the degree to which the system will use each particular micro-patch for the final estimate of luminance and color correction in the reconciliation. Weights can also be between 0 and 1, depending on how many micro-patches the object occupies. For example, if the object occupies 75% of the micro-patch, the weight could be 0.75. The assigned weight can also be 0 (making it completely removed from the reconciliation algorithm) or 1 (making it fully participate in the reconciliation algorithm without damping).

[0041] As described above, the harmonic reconciliation algorithm implemented by the ECU can be designed to average color and brightness across parts of an image to achieve consistency and smoothness in the synthesized image. The harmonic reconciliation algorithm can provide estimates of each pixel component (Y, U, and V) for each micro-block within each region of interest (ROI) for each camera. Using weighting factors now determined from the 2D obstacle map, the system can multiply the estimated Y, U, and V values ​​for each micro-block within each overlapping ROI by the corresponding weights assigned to that micro-block.

[0042] To summarize the embodiments disclosed herein, the system disclosed herein can improve the harmonic analysis algorithm for vehicle top-view. First, the system obtains the coordinates (X and Y) of one or more micro-blocks in a specific region of interest in the real world relative to the vehicle. The system obtains a 2D obstacle map from the vehicle's ultrasonic sensor data. The 2D obstacle data is segmented, and each micro-block is associated with a corresponding segment of the obstacle data. A weighting factor can be assigned to each micro-block based on the presence or extent of objects. Then, each estimate of a micro-block is multiplied by its corresponding weighting factor for harmonic analysis.

[0043] Figure 8 An example flowchart 800 of a method or process that may be implemented by one or more ECUs or processors described herein is shown. For example, these steps may be stored in memory such that when acted upon by the processor, the processor executes these steps. In some alternative embodiments, the functions indicated in the flowchart blocks may occur in a non-conforming order. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagram and / or flowchart illustrations, and combinations of blocks in the block diagram and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0044] At 802, the processor receives image data generated from multiple vehicle cameras. For example, the processor in ECU 14 can receive image data from vehicle cameras 16a-d. Using this image data, at 804, ECU 14 generates a top-down composite view using the aforementioned harmonic algorithm. The harmonic algorithm harmonizes the color and brightness between the image data from each vehicle camera. This helps improve the appearance of the composite image, especially in areas of the composite image generated from image data in the overlapping areas of two cameras. At 806, the top-down composite view is segmented into multiple segments or micro-blocks. Various overlapping regions of interest (ROIs) exist such that a micro-block of the first camera image corresponds to a micro-block of the second camera image that overlaps with the first camera image.

[0045] At 808, the processor receives sensor data generated from multiple proximity sensors. For example, the processor in ECU 14 can receive sensor data from each proximity sensor 19. At 810, a two-dimensional (2D) obstacle map is generated based on the sensor data. This 2D obstacle map includes the position of one or more detected objects relative to the proximity sensors. In one embodiment, the 2D obstacle map is segmented, and each segment of the map is labeled or marked as an area where the proximity sensors have detected objects.

[0046] At 812, the processor associates the position of one or more obstacles with one or more segmented blocks. In doing so, each segment of the 2D obstacle map can correspond to the physical location of a segmented block visible to the camera. As an example, the various regions of the 2D obstacle map that detect objects are associated with... Figure 6 Multiple micro-blocks are associated, representing portions of the surrounding environment visible to the vehicle's cameras. Using this association, at 814, the processor performs a reconciliation process without image data associated with one or more of the associated segmented blocks. This removes the image data of those micro-blocks from the reconciliation process. In another embodiment, those micro-blocks are weighted rather than completely removed from the reconciliation process.

[0047] Figure 9A The image shows a top view of a vehicle without the tuning and techniques disclosed herein. To the left of the vehicle is a large expanse of green grass. This causes the exposure adjustment of the left-hand camera (e.g., camera 16c) to make the left-hand area of ​​the top view brighter compared to the rest of the image. The same occurs on the right-hand side of the vehicle. Therefore, the areas in front of and behind the vehicle appear darker than the areas to the sides. This could be unpleasant for the driver and is not a true representation of the vehicle's surroundings. Figure 9B A top view of a vehicle implementing the tuning and techniques disclosed herein is shown. There is a much smoother transition of shadows and colors between the individual vehicle cameras stitched together to form the top view.

[0048] Figure 10A Another top view of the vehicle is shown without the tuning and techniques disclosed herein implemented. The red vehicle is located on the left side of the vehicle, and the dark black vehicle is located on the right side. This allows for exposure adjustments to the side cameras (e.g., cameras 16c, 16d), making the left and right sides of the top view appear brighter compared to the areas in front of and behind the vehicle. Figure 10B A top view of a vehicle implementing the tuning and techniques disclosed herein is shown. Again, there is a much smoother transition of shadows and colors between the individual vehicle cameras stitched together to form the top view.

[0049] The embodiments of the subject matter and operations described in this specification can be implemented in digital electronic circuits or in computer software embodied on tangible media, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs embodied on tangible media, such as one or more modules of computer program instructions encoded on one or more computer storage media for execution by or control of the operation of a data processing device. The computer storage medium can be or is included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or combinations thereof. The computer storage medium can also be one or more separate components or media (e.g., multiple CDs, disks, or other storage devices), or be included in said one or more separate components or media. The computer storage medium can be tangible and non-transitory.

[0050] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled languages, interpreted languages, declarative languages, and procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, libraries, subroutines, or multiple code sections). Computer programs can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected through a communication network.

[0051] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform actions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry, and the devices can be implemented as special-purpose logic circuitry, such as field-programmable gate arrays (“FPGAs”) or application-specific integrated circuits (“ASICs”). Such special-purpose circuitry can be referred to as a computer processor, even if it is not a general-purpose processor.

[0052] While exemplary embodiments have been described above, these embodiments are not intended to describe all possible forms covered by the claims. The language used in this specification is descriptive rather than restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of various embodiments may be combined to form other embodiments of the invention that may not be explicitly described or shown. While various embodiments may have been described as providing advantages or superiority over other embodiments or prior art implementations with respect to one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, etc. Therefore, any embodiment described as less desirable than other embodiments or prior art implementations in relation to one or more characteristics is not outside the scope of this disclosure and may be ideal for a particular application.

Claims

1. A method for generating a top-view of an area surrounding a vehicle, the method comprising: Images are received from multiple vehicle cameras, each of which is configured to observe a corresponding area outside the vehicle, and wherein portions of some areas overlap with portions of other areas to define an overlapping area. Each image is divided into multiple corresponding segments, wherein the overlapping area includes some segments of one image and some segments of another image to define the overlap; The image is harmonized based on at least one of the brightness and color values ​​within the overlapping segments to generate the top view of the area surrounding the vehicle; Receive sensor data generated from multiple proximity sensors, wherein the sensor data indicates the position of an object outside the vehicle; Associate a set of overlapping segments with the location of the object; Adjust the weights associated with the overlapping segments of the group; and The image is harmonized using adjusted weights associated with the group of overlapping segments.

2. The method according to claim 1, wherein, Harmonizing the image with the adjusted weights includes removing brightness and color values ​​associated with the overlapping segments of the group.

3. The method according to claim 1, wherein, The weights are adjusted based on the degree to which the object exists in the group of overlapping segments.

4. The method according to claim 1, wherein, The proximity sensor is an ultrasonic sensor installed at various locations around the vehicle.

5. The method according to claim 1, further comprising: A 2D obstacle map is generated based on the sensor data, wherein the 2D obstacle map provides the location of the object, and wherein the association includes associating the group of overlapping segments with segments of the 2D obstacle map associated with the location of the object.

6. The method of claim 1, further comprising displaying the top view on a vehicle display, wherein, The top view includes a harmonic image with adjusted weights associated with the group of overlapping segments.

7. A system for generating a top-view of an area surrounding a vehicle, the system comprising: Multiple vehicle cameras, each configured to observe a corresponding area outside the vehicle, wherein portions of some areas overlap with portions of other areas to define an overlapping area; Multiple proximity sensors are arranged around the vehicle; and At least one processor, coupled to the vehicle camera and the proximity sensor, is programmed to: Receive images generated by the vehicle's camera. Each image is segmented into a plurality of corresponding segments, wherein the overlapping region comprises some of the segments from one image and some of the segments from another image to define the overlapping segments. The sensor data generated by the proximity sensor is processed to determine the location of objects outside the vehicle. Associating a set of overlapping segments with the determined location of the object. Adjust the weights associated with the group of overlapping segments, which are associated with the determined position of the object, and The image is harmonized based on at least one of the brightness and color values ​​within the overlapping segments to generate the top view of the area surrounding the vehicle, wherein the harmonization of the image is further based on adjusted weights.

8. The system according to claim 7, wherein, The at least one processor is also programmed to: The image is re-harmonized without taking into account the brightness or color values ​​associated with the overlapping segments.

9. The system according to claim 7, wherein, The weights are adjusted based on the degree to which the object exists in the group of overlapping segments.

10. The system according to claim 7, wherein, The proximity sensor is an ultrasonic sensor.

11. The system according to claim 7, wherein, The at least one processor is also programmed to: A 2D obstacle map is generated based on the sensor data, wherein the 2D obstacle map provides the location of the object, and The overlapping segments are associated with the segments of the 2D obstacle map that are associated with the location of the object.

12. The system according to claim 7, wherein, The at least one processor is also programmed to: The top view is displayed on a vehicle display, wherein the top view includes a harmonic image with adjusted weights associated with the group of overlapping segments.

13. A method for harmonizing an image based on proximity sensor data, the method comprising: Images are received from multiple vehicle cameras, each of which is configured to observe a corresponding area outside the vehicle, and a portion of each area overlaps with another portion of an adjacent area to define an overlapping area. The image is segmented into multiple associated segments, wherein the overlapping region includes some segments of one image and some segments of another image to define overlapping segments; Receive sensor data generated from multiple proximity sensors, wherein the sensor data indicates the position of an object outside the vehicle; and The image is harmonized based on at least one of the brightness and color values ​​within the overlapping segment to generate a composite view of the area surrounding the vehicle, wherein the harmonization is further based on the sensor data.

14. The method of claim 13, further comprising: Associate a set of overlapping segments with the location of the object.

15. The method according to claim 14, wherein, The harmonization includes adjusting the weight associated with at least one of the luminance and color values ​​of the overlapping segments.

16. The method according to claim 15, wherein, The weights are adjusted based on the degree to which the object exists within the group of overlapping segments.

17. The method of claim 14, wherein, The harmonization is based on at least one of the luminance and color values ​​associated with the overlapping segments that exclude the group of overlapping segments.

18. The method of claim 14, further comprising: A 2D obstacle map is generated based on the sensor data, wherein the 2D obstacle map includes segments corresponding to the location of the object, and wherein the association includes associating the group of overlapping segments with the segments of the 2D obstacle map corresponding to the location of the object.

19. The method of claim 13, further comprising: The composite view is displayed on the vehicle's display screen.

20. The method according to claim 19, wherein, The composite view is a top view of the area surrounding the vehicle.

Images