Providing environmental data for motor vehicle

By scanning and processing the texture information of the surrounding environment of the motor vehicle, detailed textured illustrations are generated, which solves the problem of incomplete landmark information in the existing map data and improves the navigation accuracy and safety of motor vehicle driving.

CN119025602APending Publication Date: 2024-11-26BAYERISCHE MOTOREN WERKE AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410285555.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-03-13
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Incomplete information on landmarks in the existing motor vehicle map data makes it difficult for the driver to perform maneuvering actions correctly in an abstract or symbolic environment, especially in the absence of detailed landmarks.

Method used

By scanning the surrounding environment of the motor vehicle, identifying and creating textured illustrations of objects, including surface information and geometry, using machine learning and artificial intelligence technologies such as image repair and extension, supplementing and merging scan data, to generate detailed textured illustrations To improve the practicality of map data.

Benefits of technology

It provides more realistic map data to help drivers navigate and identify their surroundings more accurately, improving the safety and efficiency of motor vehicle driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025602A_ABST
    Figure CN119025602A_ABST
Patent Text Reader

Abstract

The invention relates to a method (200), comprising the steps of: scanning (205) an environment (125) of a motor vehicle (105); identifying (215) a scanned object (140) in the environment; determining (230) a texture of the scanned object (140); and creating (245) a textured representation of the object (140).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the provision of environmental data for a motor vehicle. In particular, the present invention relates to the provision of map data. Background Art

[0002] The motor vehicle comprises a local map memory in which map information about the surroundings is stored. The map information may include, in particular, a road network on which the motor vehicle can travel. A route can be determined between a current position and a predetermined target position based on the travel information. Prompts can be given to the driver of the motor vehicle, which enable the driver to follow the route. Such prompts may relate, for example, to maneuvers such as turning at an intersection or entering or leaving a highway.

[0003] The maneuver can be graphically represented by showing a view of the road being traveled and superimposing instructions for the maneuver, such as a directional arrow, on the representation. The more realistic the view of the surroundings, the easier it is for the driver to carry out the displayed maneuver and steer the motor vehicle in the correct direction.

[0004] The map data may include information about landmarks, which are located in the area of ​​the road network. The landmarks may be shown in more or less detail, so that the driver can better recognize the roadside again. However, the landmarks available in the existing map data are often incomplete, and symbolic or abstract surrounding information is shown between the landmarks. For example, in an inner city, a uniform area may be shown between two landmarks, which may represent a building. In particular, if a landmark that is not shown in good detail is in view, the driver may find it difficult to correctly carry out the proposed maneuver based on the abstract or symbolic environment. Summary of the invention

[0005] The object of the present invention is to provide improved environmental data of a motor vehicle. This object is achieved by means of the technical solutions of the independent claims. The dependent claims in turn specify preferred embodiments.

[0006] According to a first aspect of the invention, a method comprises the steps of: scanning the surroundings of a motor vehicle; identifying a scanned object in the environment; determining a texture of the scanned object; and creating a textured representation of the object.

[0007] The method can be used to improve the representation of the identified object in a realistic manner by showing the object with a previously observed texture. The texture can include surface information, in particular color, pattern, structure or regular or irregular interruptions. If the object is, for example, a house, different textures can be shown for different sections of the house. The texture can be interrupted, for example, by windows or doors, or windows or doors can be considered as part of the texture or as independent textures.

[0008] The identification preferably includes determining which of the scan data relate to the object and which do not. In addition, the geometric shape of the object can be determined. Approximations to the shape are acceptable, for example a house can be represented by a cuboid. Other abstract or complex shapes are also possible. The geometric shape is preferably three-dimensional, even if the scan contains only incomplete information about the shape.

[0009] The object may include a landmark, in particular a building. The building may include, for example, a residence, a church, a public building, an office building or a hall. Other exemplary landmarks include, for example, a plateau, vegetation such as one or more trees, a driveway (e.g. in the form of an entrance, a ramp or an exit), a road sign or a body of water.

[0010] Preferably, a plurality of scans of the object are collected, wherein the textured representation of the object is determined based on the scans. In a first variant, the texture can be determined separately for different scans and then the determined textures can be combined for the textured representation of the object. In another variant, the scans can first be aggregated before the texture of the object is determined. The aggregation preferably involves determining which section of the scan is arranged at which position of the geometry of the object. Meta information can be taken into account when determining the texture, such as the lighting present at the time of the scan (if an optical scan is involved), the applicable season or the age of the scan.

[0011] It is further preferred that the scans come from different motor vehicles. Preferably, a fleet of motor vehicles can be used to create scans in a predetermined geographical area, wherein a textured representation of one or more objects in the area can be determined based on the individual scans. The scans can be sorted, for example, with regard to resolution, viewing angle or driving speed during the scan. Information that is consistent can be emphasized and information that deviates from one another can be weakened. In this way, it can be determined for different sections of an object how reliable the existing scan data are. The reliability can be taken into account when determining the textured representation.

[0012] It is particularly preferred that the scans show different sections of the object. The scans can overlap each other. The more frequently a section is detected by the scans, the more reliable the information about this section can be. On the other hand, the more the scans deviate from each other, the larger the portion of the surface of the object can be scanned.

[0013] It is particularly preferred that the textured representation is determined taking into account distortions resulting from the viewing angle of the motor vehicle relative to the object. Scans determined from different viewing angles can be processed or linked to one another in an improved manner in this way.

[0014] For the purpose of identifying an object, the geographical location of the motor vehicle can be determined. The object can be identified based on map data in the area of ​​the determined location. The map data can include, for example, the geometric shape of the object. The geometric shape can be enriched with the aid of texture information in order to provide a textured representation. Based on the geographical location of the motor vehicle and the viewing angle at which the scan was produced, the object can be easily found in the map data. If the map data should already carry texture information, the texture information included by the scan can be merged with the existing texture information.

[0015] It is possible that a section of the object is not detected by any of the scans. In this case, the texture on this section can be determined based on the texture on another section of the object. In a simple embodiment, the texture information can be taken from an adjacent section for which scan data exists. However, it is preferred that further away sections can also be taken into account for determining the texture on the unscanned section.

[0016] The texture on the section can be determined by means of machine learning methods. Such methods can also include artificial intelligence. For this purpose, for example, a correspondingly trained artificial neural network KNN can be used.

[0017] In a particularly preferred embodiment, the texture on the segment is determined by means of image restoration (Inpainting). Image restoration techniques are often used to restore damaged or worn-out photographs. Such techniques can in particular include providing texture information on a predetermined segment based on image information of the surrounding area. A graphical form of interpolation in a surface can be used here. Another technique that can be used to determine the texture of an object is image extension (Outpainting). Here, an artificial intelligence algorithm is used to extend an existing image by adding content outside the original image area. Unlike image restoration, which reconstructs missing parts of an image, image extension aims to extend the image content by adding more environment or context (Kontext). Image restoration or image extension corresponds in a certain way to the interpolation or extrapolation of measured values ​​using an artificial intelligence method. In another embodiment, the texture can be determined based on stable diffusion (Stable Diffusion).

[0018] In a further embodiment of the invention, map data comprising the textured object may be provided. The map data may include different planes representing different contents. One of the planes may be arranged to show a landmark. Preferably, the representation of the textured object relates to this plane.

[0019] It should be noted that the processing of the scan can alternatively take place on board the motor vehicle or at a central location. Alternatively, part of the processing can also be carried out on the motor vehicle side and another part of the processing can be carried out on the central location side. Preferably, the map information is created on the central location side. In this case, the map information can be tested and supplemented with further information. The map information can then be provided to virtually any number of motor vehicles, for example in the form of a map update. Alternatively, the map information can also be prepared on the central location side so that it is available when a motor vehicle queries it.

[0020] According to another aspect of the invention, the system comprises a scanning device on board a motor vehicle, wherein the scanning device is configured to scan the surroundings of the motor vehicle, and the system comprises a central location configured to receive the scan, identify the scanned object in the environment, determine the texture of the scanned object and determine a textured representation of the object.

[0021] The scanning device preferably operates optically and can in particular comprise an external camera of the motor vehicle. A plurality of scanning devices can also be installed on the motor vehicle, which can be distinguished in particular with regard to their viewing angles. The scanning devices can also operate differently, for example based on lidar, radar or ultrasound. In one embodiment, one or more scanning devices on the motor vehicle are provided for collecting both geometric information about the object and optical information about the object.

[0022] The central location is preferably located outside the motor vehicle and can receive information from the motor vehicle by means of a wireless communication connection. In the same way, the motor vehicle can also receive a representation of the textured object from the central location.

[0023] The system may include a plurality of motor vehicles, wherein the central location is preferably configured to collect scans from the motor vehicles and to determine a textured representation based on the scans.

[0024] The motor vehicle preferably comprises a car or a motorcycle. In other embodiments, the motor vehicle, for example, may also comprise a truck or a bus.

[0025] The system can be configured to partially or completely execute the method described here. For this purpose, a preferably electronic processing device can be provided on the vehicle side and / or on the central location side, which processing device can in particular include a programmable microcomputer or microcontroller. The method can be in the form of a computer program product with program code segments. The computer program product can also be stored on a computer-readable data carrier. Features or advantages of the method can be transferred to the system or vice versa.

[0026] According to yet another aspect of the invention, it is proposed to use image restoration to determine the texture on a section of an object based on texture information on another section of the object. Preferably, the texture information is determined based on one or more scans of the object from one or more motor vehicles. The scans can be combined with geometric information about the shape of the object in order to create an at least partially textured representation of the object. In another variant, image expansion is used to determine the texture on sections for which no direct information is available. By using artificial intelligence methods, convincing textures can be determined that are well integrated into the overall image consisting of the available direct information.

[0027] It is generally preferred that the generation of texture information is used at most for a predetermined portion of the visible surface of the object. For example, the generation of texture information on more than approximately 10% or approximately 20% of the visible surface of the object can be rejected. The propagation of seriously erroneous or inconsistent information can be prevented by the restriction. On the other hand, possible errors or deviations from the actual situation on a surface below a defined threshold value can be tolerated in order to ensure that the object is re-identified on the spot by the observer. The threshold value can be adapted even more accurately based on experience when creating the texture data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention will now be described more precisely with reference to the accompanying drawings, in which:

[0029] Figure 1 Showing the system;

[0030] Figure 2 a flow chart showing the method; and

[0031] Figure 3 An exemplary scan of an object is shown. DETAILED DESCRIPTION

[0032] Figure 1 A system 100 is shown, which has a motor vehicle 105 with a device 110 installed thereon, and a central location 115 . Motor vehicle 105 is shown as an example as a passenger car. Central location 115 is located outside motor vehicle 105 .

[0033] Device 110 has a scanning device 120 for scanning surroundings 125 of motor vehicle 105 and preferably a processing device 130 . Wireless communication device 135 may be provided for communicating with central location 115 .

[0034] Scanning device 120 preferably operates optically and may include, for example, a camera, a color camera, a stereo camera, a depth camera or a lidar sensor. A plurality of scanning devices 120 may also be used. The scan of surroundings 125 provided by means of scanning device 120 may include image data of objects 140 located in surroundings 125. Figure 1 In the illustration of , a building is assumed as an object 140 by way of example and is schematically illustrated.

[0035] The viewing angle, focal length, resolution and orientation of the scanning device 120 on the motor vehicle 105 can be fixedly predetermined in each case. However, some of these parameters can also be controlled, in particular by means of the processing device 130. The object 140 is usually three-dimensional and cannot be completely optically detected from any viewing angle from the motor vehicle 105. However, the following part of the object 140 can be imaged in one or more scans, the surface of which is facing the motor vehicle 105. The scan can include all areas of the object 140 that can be optically detected from the current viewing angle or sections of these areas. The scanning device 120 can be controlled to provide a scan of the surroundings 125 in a periodic or event-controlled manner. The slower the motor vehicle 105 travels or the farther the object 140 is from the motor vehicle 105, the less frequently the scanning device 120 can scan the surroundings 125.

[0036] In one embodiment, the processing device 130 is configured to forward the scans of the scanning device 120 to the central location 115 without modification. Optionally, multiple scans can be combined or compressed. In another embodiment, the processing device 130 can also process one or more scans, and then transmit the processing results to the central location 115.

[0037] The central location 115 preferably has a processing device 145, a communication device 150 and preferably a data memory 155. With the aid of the communication device 150, information can be exchanged unilaterally or bilaterally with the motor vehicles 105 or the device 110. Preferably, the communication device 150 is provided for communication with a plurality of motor vehicles 105. The processing device 145 is provided for receiving and further processing a scan of at least one motor vehicle 105. The received information or the processing results can be stored in the data memory 155.

[0038] Track, using the scanning of the object 140 from one or more motor vehicles 105, in order to determine the texture of the object 140 and provide a textured representation of the object 140. The texture includes in particular visible surface information, such as color, texture or division. The collected information about one or more objects 140 can be used as a basis for providing map information. The map information can be distributed to one or more motor vehicles 105. On board the motor vehicle 105, the representation of the surrounding environment 125 generated based on the map information can use the textured representation of the object 140 to provide an improved realistic view of the surrounding environment 125. Such a view can be used, for example, to provide the driver of the motor vehicle 105 with instructions for following a predetermined route.

[0039] Figure 2A flow chart of a method 200 for providing surroundings data of a motor vehicle 105 is shown. The method 200 can be carried out in particular by means of the system 100. It should be noted that the method 200 is preferably carried out with a plurality of motor vehicles 105. To improve clarity, reference is made to a single motor vehicle 105 for the description of the method.

[0040] In step 205, surroundings 125 of motor vehicle 105 may be scanned. The scanning may be performed by means of one or more scanning devices 120. In a step 210, the geographical location of motor vehicle 105 may be determined. For example, the location may be determined by means of a receiver for a satellite-supported global navigation system (GNSS). Alternatively, the location may also be determined based on map data and landmarks identified in surroundings 125, for example.

[0041] In step 215, objects 140 in the surroundings 125 can be identified. Information from the scan can be used for this purpose. For example, automatic image recognition can be used, which is provided for identifying typical objects 140, which can be in the area of ​​the road for the motor vehicle 105. Such objects include, for example, buildings, public facilities or vegetation. The objects 140 can also be determined based on map data in the area of ​​a specific geographical location. The map data can include identification, position and / or geometric information about the objects 140. In a further preferred embodiment, the two variants can be combined with each other.

[0042] In step 220, the texture of object 140 can be determined on the previously completed scan. In parallel, the position of the scan relative to object 140 can be determined in step 225. The position can indicate which section of object 140 is recognizable on the scan and, if necessary, how the section or the scan must be geometrically distorted in order to provide a view of object 140 from another perspective. This correction can be carried out before evaluating the texture in step 220.

[0043] In step 230, the textures of different sections of object 140 can be combined. For this purpose, overlapping sections of the scans can be determined and merged with each other. If there are enough scans, the texture can be determined for a large part of the surface of object 140. In this way, it is usually not possible to detect surfaces of object 140 that are away from the location where motor vehicle 105 can travel. In particular, if the textured representation of object 140 that is provided later is only to be used on board motor vehicle 105, the determination of the texture in these areas can be omitted.

[0044] In step 235 , sections of object 140 for which no texture information or no scan is available can be determined. Typically, this can be the case for a higher section of object 140 .

[0045] In step 240, the texture on the determined section can be determined based on texture information on another area of ​​the object 140. Machine learning methods can be used for this purpose. Preferably, the texture on the missing section is determined by means of image restoration and / or image extension methods. In image extension, artificial intelligence algorithms can be used to extend an existing image of the surface of the object 140 by adding content outside the initial image area. In this case, the image content can be extended by adding more environment or context.

[0046] In step 245, a textured representation of the object 140 may be provided. The textured representation preferably includes geometric information and texture information. In this way, a view of the object 140 from any viewing angle may be generated.

[0047] The determined representation may be integrated into already existing map data in step 250 . The map data relate to an area around the geographical location of motor vehicle 105 .

[0048] In step 255, the determined map data or the textured representation of object 140 may be provided to motor vehicle 105. This may be another motor vehicle 105 than the motor vehicle that carried out the scan of object 140. Preferably, the map data are provided to a plurality of motor vehicles 105.

[0049] Figure 3 1 shows a scan of an exemplary object 140 of a road 305 on which motor vehicle 105 is traveling. Object 140 is again assumed to be a building by way of example and is shown separated from other objects. Figure 3 , two side faces 310 of the object 140 are shown. On these side faces 310, exemplary sections 315 are plotted, for which a scan is present. Sections 315 at different heights can be scanned, for example, by motor vehicles 105 at different heights. The section 315 shown in the right region, which is located further up the building 140, can also be scanned by a tilted motor vehicle 105, for example, while it is using an entrance or exit (e.g. to a garage).

[0050] The part of the side surface 310 that is not covered by the at least one section 315 may be referred to as a missing section 320. For the missing section 320, there is no scanning information and accordingly no direct texture information.

[0051] Under predetermined conditions, for example, when a surface of object 140 for which there is a directly scanned section 315 has reached a predetermined proportion of the total surface of object 140, a missing section 320 can be created by means of artificial intelligence methods from the field of image processing and, in particular, image generation. Texture information for such a section 320 can be determined, preferably based on interpolation or extrapolation, in particular by means of image restoration or image expansion, on the basis of the information of the scanned section 315.

[0052] Reference numerals list

[0053] 100 Systems

[0054] 105 Motor Vehicles

[0055] 110 Equipment

[0056] 115 Central Location

[0057] 120 Scanning device

[0058] 125 Surroundings

[0059] 130 Processing device

[0060] 135 Communication devices

[0061] 140 Objects

[0062] 145 Processing Device

[0063] 150 Communication devices

[0064] 155 Data storage

[0065] 200 Methods

[0066] 205 Scan the surroundings

[0067] 210 Determine location

[0068] 215 Object recognition

[0069] 220 Determine Texture

[0070] 225 Determine the scan location

[0071] 230 Merge Textures

[0072] 235 Determine missing segments

[0073] 240 Determine the texture on the segment

[0074] 245 Provides textured illustrations

[0075] 250 Integrate graphics into map data

[0076] 255 Provide map data

[0077] 305 Driving Road

[0078] 310 Side

[0079] 315 Segment

[0080] 320 Missing Segment

Claims

1. A method (200), comprising the following steps: - scanning (205) the surroundings (125) of the motor vehicle (105); - identifying (215) a scanned object (140) in the environment; - determining (230) the texture of the scanned object (140); and - Creating (245) a textured representation of said object (140).

2. The method (200) according to claim 1, wherein: A plurality of scans of the object (140) are collected; wherein a textured representation of the object (140) is determined (245) based on the scans.

3. The method (200) according to claim 2, wherein: Each of the scans is from a different motor vehicle (105).

4. The method (200) according to claim 2 or 3, wherein: Each of the scans shows a different section (315) of the object (140).

5. The method (200) according to any one of the preceding claims, wherein: The textured representation is determined taking into account distortions of the scan that are caused by a viewing angle of the motor vehicle (105) relative to the object (140).

6. The method (200) according to any of the preceding claims, wherein: A geographic location of the motor vehicle (105) is determined; and the object (140) is identified (215) based on map data in the area of ​​the determined location.

7. The method (200) according to any of the preceding claims, wherein: The texture of a section (320) of the object (140) that was not scanned is determined based on the texture of another section (315) of the object (140).

8. The method (200) according to claim 7, wherein: The texture on the section (320) is determined using a machine learning method.

9. The method (200) according to claim 7 or 8, wherein: The texture on the section (320) is determined by means of image inpainting.

10. The method (200) according to one of claims 3 to 9, further comprising: Map data including a textured object (140) is provided (255).

11. The method (200) according to claim 10, wherein: The map data is transmitted to a plurality of motor vehicles (105).

12. A system (100), comprising the following elements: - a scanning device (120) on board a motor vehicle (105); wherein The scanning device (120) is configured to scan the surroundings (125) of the motor vehicle (105); - a central location (115) configured to: receive a scan; identify a scanned object (140) in an environment; determine a texture of the scanned object (140); and determine a textured representation of the object (140).

13. The system (100) of claim 12, comprising a plurality of motor vehicles (105); wherein: The central location (115) is configured to collect a scan of the motor vehicle (105) and to determine the textured representation based on the scan.

14. Application of image inpainting so that texture on a section (320) of an object (140) is determined based on texture information on another section (315) of the object (140).