Electronic device for vehicle, and operation method therefor

Abstract

The present disclosure provides an electronic device for a vehicle, which controls automatic parking of the vehicle. The electronic device for a vehicle may comprise: a plurality of cameras; a memory for storing one or more instructions; and at least one processor for executing the one or more instructions stored in the memory. The at least one processor may be configured to, by executing the one or more instructions: obtain a plurality of images obtained by photographing the surroundings of the vehicle by using the plurality of cameras; detect features from the plurality of images; generate a spatial map of the surroundings of the vehicle on the basis of the features; determine a driving route for parking the vehicle on the basis of the spatial map; control driving of the vehicle on the basis of the driving route; and control the vehicle to perform a parking operation on the basis of the vehicle arriving at a destination of the driving route.

Description

Automotive electronic device and its operating method

[0001] The present disclosure relates to a vehicle electronic device capable of controlling automatic parking of a vehicle using sensors in the vehicle and an operating method thereof.

[0002] As autonomous driving technology advances, various methods are being applied to automatically park vehicles. For example, a parking lot is equipped with a system that supports automatic parking by installing sensors such as cameras and LiDAR to detect vehicle movements within the parking lot. However, this requires separate infrastructure in the parking lot, and automatic parking is not supported in environments without such infrastructure. Another example is a method in which a vehicle drives within a parking lot and uses in-vehicle sensors to detect parking spaces and automatically parks in the detected space. However, due to the nature of wide-angle cameras mounted on vehicles, they only monitor the area immediately adjacent to the vehicle, detecting empty spaces and / or parking lines adjacent to the vehicle. This makes it difficult to estimate the vehicle's path if a parking space exists outside the coverage area of ​​the in-vehicle sensors.

[0003] In order to support automatic parking using in-vehicle sensors, it is necessary to identify parking spaces in a wider range and efficiently determine a driving path for parking, rather than just identifying parking spaces adjacent to the vehicle at every moment.

[0004] According to one aspect of the present disclosure, a vehicle electronic device may be provided. The vehicle electronic device may include: a plurality of cameras; a memory storing one or more instructions; and at least one processor executing the one or more instructions stored in the memory.

[0005] The at least one processor can acquire a plurality of images of the surroundings of the vehicle by using the plurality of cameras by executing the one or more instructions.

[0006] The at least one processor can detect features from the plurality of images by executing the one or more instructions.

[0007] The at least one processor may generate a spatial map around the vehicle based on the features by executing the one or more instructions.

[0008] The at least one processor can determine a driving path for parking the vehicle based on the spatial map by executing the one or more instructions.

[0009] The at least one processor can control driving of the vehicle based on the driving path by executing the one or more instructions.

[0010] The at least one processor may control the vehicle to perform a parking operation based on the vehicle arriving at a destination of the driving route by executing the one or more instructions.

[0011] According to one aspect of the present disclosure, a method of operating an electronic device for a vehicle may be provided.

[0012] The above method may include a step of acquiring a plurality of images taken of the surroundings of the vehicle using a plurality of cameras.

[0013] The method may include a step of detecting features from the plurality of images.

[0014] The method may include a step of generating a spatial map around the vehicle based on the features.

[0015] The method may include a step of determining a driving path for parking the vehicle based on the spatial map.

[0016] The method may include a step of controlling driving of the vehicle based on the driving path.

[0017] The method may include a step of controlling the vehicle to perform a parking operation based on the vehicle arriving at a destination of the driving route.

[0018] According to one aspect of the present disclosure, a computer-readable recording medium having recorded thereon a program for causing a vehicle electronic device to control the operation of a vehicle, the program causing the vehicle electronic device to execute any one of the methods described above and below.

[0019] FIG. 1 is a drawing schematically illustrating an operation of a vehicle electronic device controlling a vehicle according to one embodiment of the present disclosure.

[0020] FIG. 2 is a flowchart for explaining an operation of a vehicle electronic device controlling a vehicle according to one embodiment of the present disclosure.

[0021] FIG. 3 is a drawing for explaining an operation of a vehicle electronic device according to one embodiment of the present disclosure to determine a driving path for automatic parking.

[0022] FIG. 4 is a drawing for explaining a plurality of cameras of a vehicle electronic device according to one embodiment of the present disclosure.

[0023] Figure 5a is a drawing for explaining detectable features in the environment surrounding a vehicle.

[0024] FIG. 5b is a diagram for explaining an operation of a vehicle electronic device detecting a feature from an image according to one embodiment of the present disclosure.

[0025] FIG. 5c is a diagram for explaining an operation of a vehicle electronic device detecting a feature from an image according to one embodiment of the present disclosure.

[0026] FIG. 6A is a diagram for explaining an operation of a vehicle electronic device generating a spatial map according to one embodiment of the present disclosure.

[0027] FIG. 6b is a drawing to further explain a spatial map generated by a vehicle electronic device according to one embodiment of the present disclosure.

[0028] FIG. 6c is a diagram illustrating features within layers of a spatial map generated by an in-vehicle electronic device according to one embodiment of the present disclosure.

[0029] FIG. 7 is a drawing for explaining an operation of a vehicle electronic device according to one embodiment of the present disclosure to determine a driving path for parking.

[0030] FIG. 8 is a diagram for explaining an operation of a vehicle electronic device updating a spatial map according to one embodiment of the present disclosure.

[0031] FIG. 9 is a drawing for explaining an operation of a vehicle electronic device controlling a vehicle according to one embodiment of the present disclosure.

[0032] Figure 10 is a drawing for explaining an operation performed using artificial intelligence technology in the present disclosure.

[0033] FIG. 11 is a block diagram illustrating a configuration of a vehicle electronic device according to one embodiment of the present disclosure.

[0034] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0035] The terms used in this disclosure are selected from widely used, common terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0036] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein. Furthermore, terms containing ordinal numbers, such as "first" or "second," used herein may be used to describe various components, but such components should not be limited by such terms. Such terms are used solely to distinguish one component from another.

[0037] When a part of the specification is said to "include" a component, unless otherwise specifically stated, this does not exclude other components but rather implies the inclusion of other components. Furthermore, terms such as "part" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0038] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts that are not related to the description are omitted in order to clearly describe the present disclosure, and similar parts are designated with similar reference numerals throughout the specification. In addition, the reference numerals used in each drawing are only for the purpose of describing each drawing, and different reference numerals used in different drawings do not indicate different elements. The present disclosure will be described in detail below with reference to the attached drawings.

[0039] FIG. 1 is a drawing schematically illustrating an operation of a vehicle electronic device controlling a vehicle according to one embodiment of the present disclosure.

[0040] Referring to FIG. 1, a vehicle electronic device (2000) according to one embodiment of the present disclosure may be mounted on a vehicle. The location of the vehicle electronic device (2000) illustrated in FIG. 1 is merely an example.

[0041] In one embodiment, the vehicle electronics device (2000) can control the vehicle to automatically find a parking slot and park while driving in a space without direct driver control. Specifically, the vehicle electronics device (2000) can sense the surrounding environment and search for a parking path using multiple cameras.

[0042] In one embodiment, the vehicle electronics device (2000) can generate a spatial map for a space (e.g., a parking lot) in which a vehicle is located. The vehicle electronics device (2000) can identify various semantic elements present within the parking lot for automatic parking. For example, the vehicle electronics device (2000) can identify driving direction signs displayed on the parking lot road surface, signs indicating empty parking slots (e.g., available / unavailable signs for parking slots installed on the ceiling, etc.), and signs indicating the number of available parking spaces, and map features to the spatial map generated by the vehicle electronics device (2000).

[0043] The vehicle electronic device (2000) can determine a driving route based on a spatial map on which various features are mapped, and update the optimal driving route in real time while driving along the driving route. For example, as illustrated in FIG. 1, the vehicle electronic device (2000) can sense the surrounding environment at the current location of the vehicle (point A) and generate a spatial map including spatial information around the vehicle based on the sensed range. Thereafter, the vehicle electronic device (2000) can expand and update the spatial map while driving in the space. For example, the vehicle electronic device (2000) can identify features within the space (driving direction for parking, exit direction, empty parking slots), etc. while driving in the space, and control the vehicle to move to a destination (point B) for automatic parking. When the vehicle reaches a destination with an empty parking slot, the vehicle electronic device (2000) can perform a parking operation.

[0044] An electronic device (2000) for a vehicle according to one embodiment analyzes the surrounding environment and generates and searches for a path for automatic parking using only sensors mounted in the vehicle without additional sensors.

[0045] According to one embodiment, a vehicle electronic device (2000) uses only sensors included in the vehicle. For example, the vehicle electronic device (2000) may control the vehicle to perform automatic parking using only a plurality of cameras. Alternatively, the vehicle electronic device (2000) may further utilize sensors that are basically installed in the vehicle. The vehicle electronic device (2000) may obtain sensor data from sensors that are basically installed in the vehicle and utilize the data for operations such as feature detection, depth estimation, vehicle driving, and vehicle parking. Since the automatic parking method of the vehicle electronic device (2000) according to the present disclosure can be implemented using a plurality of cameras, the present disclosure will describe an embodiment utilizing a plurality of cameras. However, it will be apparent to those skilled in the art that additional basic sensors installed in the vehicle can be utilized in the present disclosure.

[0046] According to one embodiment, a vehicle electronic device (2000) can control a vehicle to automatically find an empty parking slot and park itself, even without various facilities (sensors / cameras installed in the parking lot, communication systems within the parking lot, etc.) utilized in a smart parking lot system for automatic parking. The specific operations for controlling automatic parking by the vehicle electronic device (2000) will be described in more detail in the drawings and descriptions thereof described below.

[0047] FIG. 2 is a flowchart for explaining an operation of a vehicle electronic device controlling a vehicle according to one embodiment of the present disclosure.

[0048] In operation S210, a vehicle electronic device (2000) according to one embodiment acquires multiple images of the surroundings of the vehicle using multiple cameras.

[0049] In one embodiment, the vehicle electronic device (2000) may include a plurality of cameras. For example, the number of cameras included in the vehicle electronic device (2000) may be six. Specifically, the vehicle electronic device (2000) may include one front camera, one rear camera, and two side cameras (a first side camera and a second side camera) for each of the left and right sides. The first side camera may be a camera that photographs the front for each of the left and right sides, and the second side camera may be a camera that photographs the rear for each of the left and right sides.

[0050] The vehicle electronic device (2000) can acquire multiple images of the surroundings of the vehicle using multiple cameras. Specifically, if the vehicle electronic device (2000) includes six cameras, six images captured in six directions from the vehicle can be acquired. The acquired multiple images can be used to analyze the surrounding space at the current location of the vehicle. In one embodiment, the cameras included in the vehicle electronic device (2000) can be configured as cameras having a wider vertical angle of view than a typical camera. The vehicle electronic device (2000) can capture images including walls, pillars, ceilings, etc. using cameras having a wider vertical angle of view. When analyzing the surrounding space, the vehicle electronic device (2000) can detect features existing on the road surface and features existing on walls, pillars, ceilings, etc.

[0051] The vehicle electronic device (2000) can control the driving and parking operations of the vehicle so that the vehicle can automatically move from the current location of the vehicle to an empty parking slot based on the analysis results of the images.

[0052] In operation S220, an in-vehicle electronic device (2000) according to one embodiment detects features from a plurality of images.

[0053] In one embodiment, the vehicle electronic device (2000) can detect first features related to a space in which a vehicle is located from a plurality of images. The first features can be used to identify the structure of the space in which the vehicle is located, and can correspond to three-dimensional space data and data related to objects within the space. The first features can include, but are not limited to, for example, a drivable space, a parking slot, a layout of the space (e.g., walls, pillars, etc.), a size of the space (e.g., width, length, height, etc.), and a position and size of one or more static objects (e.g., cones, curbs, etc.) within the space.

[0054] In one embodiment, the vehicle electronics device (2000) may detect second features related to semantic elements related to driving and parking of a vehicle from a plurality of images. The second features may correspond to driving-related data having a meaning related to driving and / or parking, which are features that the vehicle electronics device (2000) can use to determine the driving path and parking location of the vehicle. The second features may include, but are not limited to, driving direction indicators located within a space such as walls, pillars, road surfaces, and ceilings, zone identification indicators indicating a current location, available / unavailable parking slot indicators, and the like.

[0055] In operation S230, an electronic device for a vehicle (2000) according to one embodiment generates a spatial map around the vehicle based on features.

[0056] In one embodiment, the spatial map may be map data for an indoor space (e.g., an indoor parking lot). In this case, the vehicle electronic device (2000) may identify the location of the indoor space, determine the direction, and search for a route based on the spatial map. The spatial map for the indoor space may be structured as a layered structure of data for representing indoor spatial information. For example, the spatial map may include, but is not limited to, a first layer containing spatial information, a second layer containing driving and parking-related information, and a third layer containing information on dynamic objects within the space.

[0057] In one embodiment, the first layer of the spatial map may be an occupancy grid map layer. The vehicle electronic device (2000) may generate a grid map composed of cells having a certain size and location, and update the occupancy status of each cell. The vehicle electronic device (2000) may preprocess images acquired through multiple cameras and map features detected from the images to update the occupancy grid map. For example, the vehicle electronic device (2000) may mark cells in which objects (e.g., parked vehicles, pillars, walls, etc.) exist as occupied in the occupancy grid map, and mark cells corresponding to empty spaces as empty.

[0058] In one embodiment, the second layer of the spatial map may be a semantic feature layer generated based on semantic features related to driving and parking. For example, the second layer may be mapped based on the first layer representing spatial information, including driving direction indicators, identification indicators indicating the current location, available / unavailable parking slots, and the number of available parking spaces.

[0059] In one embodiment, the third layer of the spatial map may be a dynamic object layer containing information on moving objects within the space. For example, the third layer may be mapped after identifying moving objects within the space, such as pedestrians and vehicles, that may become obstacles when a vehicle drives within the space, based on the first layer representing spatial information.

[0060] In some embodiments, the spatial map may be map data for an outdoor space (e.g., an outdoor parking lot). If the space where the vehicle is located is an outdoor space, the spatial map may be generated by omitting features that exist only in indoor spaces (e.g., pillars).

[0061] In one embodiment, the vehicle electronics device (2000) may generate a spatial map of a parking building. A parking building refers to a building that includes one or more parking lot levels. A parking building may be, for example, a building that includes one or more parking lot levels above ground and / or underground. If a building, such as a parking building, includes multiple parking lot levels, the vehicle electronics device (2000) may generate a spatial map for each of the multiple parking lot levels. The spatial map corresponding to each parking lot level may include identification information for identifying each floor.

[0062] In operation S240, an electronic device for a vehicle (2000) according to one embodiment determines a driving path for parking a vehicle based on a spatial map.

[0063] In one embodiment, the vehicle electronic device (2000) can identify an empty parking slot where the vehicle can be parked based on a space map. The space map generated in operation S230 is a local space map, not a global space map representing the entire space, as it is generated by photographing the surroundings of the vehicle from the vehicle's initial location. If an empty parking slot where parking is possible is identified in the space map generated in operation S230, the vehicle electronic device (2000) can determine a driving route with the identified empty parking slot as the destination. In this case, there may be one or more driving routes.

[0064] In one embodiment, if no empty parking slot is identified in the space map, the vehicle electronics device (2000) may determine a destination candidate to identify an empty parking slot and determine a driving route toward the destination candidate.

[0065] In operation S250, a vehicle electronic device (2000) according to one embodiment controls driving of the vehicle based on a driving path.

[0066] In one embodiment, the vehicle electronics device (2000) may control control devices embedded in the vehicle (e.g., powertrain and steering device, etc.) to cause the vehicle to drive along a determined driving path.

[0067] The vehicle electronic device (2000) can continuously capture images of the surroundings of the vehicle using multiple cameras while controlling the vehicle to drive, thereby acquiring new images. The vehicle electronic device (2000) can expand and update a spatial map based on the new images acquired while the vehicle is driving. The vehicle electronic device (2000) can identify an empty parking slot within the updated spatial map and update a driving route for parking the vehicle. For example, if a destination candidate is set as described in operation S250, the vehicle electronic device (2000) can re-set an empty parking slot identified within the updated spatial map as the final destination.

[0068] In operation S260, the vehicle electronic device (2000) according to one embodiment controls the vehicle to perform a parking operation based on arrival at a destination of a driving route.

[0069] As the vehicle electronic device (2000) performs operations S210 to S250, the destination of the driving route is determined to be an empty parking slot. When the vehicle reaches a predetermined distance from the empty parking slot, the vehicle electronic device (2000) can control a parking operation so that the vehicle can be parked in the empty parking slot. For example, the vehicle electronic device (2000) can execute an automatic parking mode and control the power unit and steering unit built into the vehicle so that the vehicle can be parked in the empty parking slot.

[0070] In one embodiment, the vehicle electronic device (2000) may provide parking information for the vehicle to the user based on the completion of parking of the vehicle. The parking information may include, but is not limited to, information such as the vehicle's parked location within the parking lot, the parking start time, and the parking fee.

[0071] FIG. 3 is a drawing for explaining an operation of a vehicle electronic device according to one embodiment of the present disclosure to determine a driving path for automatic parking.

[0072] Referring to FIG. 3, a vehicle electronic device (2000) according to one embodiment can acquire multiple images captured around the vehicle using multiple cameras (310). The multiple images captured using the multiple cameras (310) can be stored in the form of an IPM (Inverse Perspective Mapping) image (320), but is not limited thereto. The IPM image (320) acquired by the vehicle electronic device (2000) can be used by the vehicle electronic device (2000) to estimate the size, location, distance, etc. of an object, and to recognize a road area and lanes, etc., to plan a driving route.

[0073] In one embodiment, the vehicle electronic device (2000) can perform a feature detection (330) operation to detect various features from an image. For example, the vehicle electronic device (2000) can perform a feature detection (330) operation on an IPM image (320). Alternatively, the vehicle electronic device (2000) can process the IPM image (320) to convert it into an original image and perform a feature detection (330) operation on the converted image. Alternatively, the vehicle electronic device (2000) can perform a feature detection (330) operation on an image acquired through an individual camera. For a specific example, the vehicle electronic device (2000) can perform a feature detection (330) operation on a front image (322) acquired through a front camera (312) and detect semantic elements (340).

[0074] As a result of the feature detection (330) operation performed by the vehicle electronic device (2000) based on images acquired from the plurality of cameras (310), first features related to a space in which a vehicle is located and second features related to semantic elements for driving and parking of the vehicle may be detected. The first features may include, but are not limited to, for example, a drivable space, a parking slot, a layout of the space (e.g., a wall, a pillar, etc.), a size of the space (e.g., width, length, height, etc.), and a position and size of one or more static objects (e.g., a cone, a curb, etc.) within the space. The second features may include, but are not limited to, for example, a driving direction indicator within the space (e.g., a wall, a pillar, a road surface, a ceiling, etc.), an identification indicator indicating a current location, an available / unavailable indication of a parking slot, an indication of the number of available parking spaces, etc.

[0075] A vehicle electronic device (2000) can perform a spatial map generation operation (350) based on first features and second features. The spatial map can be configured as a layer structure of data for representing spatial information of a space in which a vehicle is located.

[0076] In one embodiment, the spatial map may include an occupancy grid map layer. The vehicle electronics device (2000) may map features detected from images to the occupancy grid map layer. For example, the vehicle electronics device (2000) may mark cells in the occupancy grid map where objects (e.g., parked vehicles, pillars, walls, etc.) exist as occupied, and mark cells corresponding to empty spaces as empty.

[0077] In one embodiment, the spatial map may include a semantic feature layer containing semantic features related to driving and parking. The vehicle electronics device (2000) may map features detected from images to the semantic feature layer. For example, the vehicle electronics device (2000) may map semantic feature elements (340) such as driving direction indicators, identification indicators indicating current location, available / unavailable parking slot indicators, and available parking space indicators, which are present in the space, such as walls, pillars, road surfaces, and ceilings.

[0078] In one embodiment, the vehicle electronics device (2000) can generate a temporary spatial map at the current location of the vehicle and update the spatial map based on additional images acquired as the vehicle drives.

[0079] A vehicle electronic device (2000) according to one embodiment can perform a localization (380) operation for estimating the current location of a vehicle based on odometry (370) data that estimates the location and attitude of the vehicle based on data obtained from an IMU sensor (360) and a wheel encoder (362) built into the vehicle, a result of feature detection (330), and a result of spatial map generation (350).

[0080] Odometry (370) data refers to data estimating the speed and travel distance of a vehicle. For example, the vehicle electronic device (2000) can measure the acceleration and rotation speed of the vehicle using an IMU sensor (360) built into the vehicle, and can detect the rotation of the wheels using a wheel encoder (362) and convert it into travel distance.

[0081] The localization (380) operation refers to the process of estimating the current location of the vehicle. The vehicle electronic device (2000) can accurately determine the current location of the vehicle by using at least some of the sensor data, odometry (370) data, spatial features, and spatial maps acquired from sensors built into the vehicle (e.g., IMU sensors, cameras, LiDAR, cameras, GPS sensors, etc.). In this case, an algorithm and information for localization (380) can be used. For example, the Extended Kalman Filter (EKF) algorithm can be applied to accurately estimate the location of the vehicle, and a data format such as 6-DoF (Degree of Freedom) pose data, which is information representing the location and attitude in 3D space, can be applied. The 6-DoF pose data can include three degrees of freedom (x, y, z) representing the location of the vehicle and three degrees of freedom (roll, pitch, yaw) representing the rotation of the vehicle. However, the manner in which the vehicle electronic device (2000) performs localization (380) operation is not limited to the examples described above.

[0082] A vehicle electronic device (2000) can acquire images using multiple cameras (310) and perform a path planning (390) operation to determine a path of a vehicle for parking based on the results of feature detection (330), spatial map generation (350), and localization (380).

[0083] When a driving path is determined, the vehicle electronic device (2000) can control the driving of the vehicle based on the driving path, and control the vehicle to perform a parking operation based on the vehicle arriving at the destination of the driving path.

[0084] FIG. 4 is a drawing for explaining a plurality of cameras of a vehicle electronic device according to one embodiment of the present disclosure.

[0085] Referring to FIG. 4, the vehicle electronic device (2000) may include multiple cameras. The vehicle electronic device (2000) may acquire multiple images through the multiple cameras.

[0086] The first image (410) may be an image acquired through a front camera installed at the front of the vehicle. The vehicle electronic device (2000) may detect features located in front of the vehicle based on the first image (410) acquired through the front camera.

[0087] The second image (420) may be an image acquired through a rear camera installed at the rear of the vehicle. The vehicle electronic device (2000) may detect features at the rear of the vehicle based on the second image (420) acquired through the rear camera.

[0088] The third image (430) and the fourth image (440) may be images acquired through side cameras (a first left-side camera and a second left-side camera) installed on the left side of the vehicle. The first left-side camera may capture the left front side of the vehicle, and the second left-side camera may capture the left rear side of the vehicle. The vehicle electronic device (2000) may detect features on the left side of the vehicle based on the third image (430) and the fourth image (440) acquired through the left-side cameras.

[0089] The fifth image (450) and the sixth image (460) may be images acquired through side cameras (a first right-side camera and a second right-side camera) installed on the right side of the vehicle. The first right-side camera may capture a photo of the right front side of the vehicle, and the second right-side camera may capture a photo of the right rear side of the vehicle. The vehicle electronic device (2000) may detect features on the left side of the vehicle based on the third image (430) and the fourth image (440) acquired through the right-side cameras.

[0090] In one embodiment, the cameras (front camera, rear camera, first left-side camera, second left-side camera, first right-side camera, and second right-side camera) included in the vehicle electronic device (2000) may be configured as cameras having a wider angle of view in the vertical direction compared to general cameras. For example, a camera having an aspect ratio of 5:4 for an image acquired by the camera may be used, but the aspect ratio of the image is not limited thereto. Generally, surround cameras installed in vehicles are generally installed to look at the road surface around the vehicle in order to recognize parking lines, lanes, etc., and thus, images having a low vertical field of view are acquired. In contrast, the vehicle electronic device (2000) of the present disclosure is configured to include a camera having a wide vertical angle of view. That is, the cameras of the present disclosure are configured as cameras having a wide angle of view in the vertical direction, rather than wide-angle cameras having a wide angle of view in the horizontal direction. Therefore, it has a high vertical field of view (e.g., human-like field of view), and thus can detect features not only on the road surface, but also on walls, pillars, ceilings, etc.

[0091] When the cameras included in the vehicle electronic device (2000) have a wide angle of view in the vertical direction, they have a relatively narrow angle of view in the horizontal direction, so the horizontal field of view of each camera may be relatively limited. Accordingly, the vehicle electronic device (2000) uses a total of six cameras, including a front camera, a rear camera, a first left-side camera, a second left-side camera, a first right-side camera, and a second right-side camera, so that the vehicle electronic device (2000) can secure a horizontal field of view in all directions of the vehicle.

[0092] In one embodiment, the vehicle electronic device (2000) may include two or more cameras facing a specific direction. For example, the vehicle electronic device (2000) may include two or more cameras facing one direction, such as the front, the side, or the rear, and having different angles of view. Specifically, taking the front direction as an example, the vehicle electronic device (2000) may obtain a first image (410) through a first front camera and obtain a telephoto image (412) through a second front camera having a larger focal length than the first front camera. By using the telephoto image (412) when detecting features around the vehicle, the vehicle electronic device (2000) may more accurately detect features at a distance.

[0093] Figure 5a is a drawing for explaining detectable features in the environment surrounding a vehicle.

[0094] In one embodiment, the vehicle electronic device (2000) can detect features from a plurality of images acquired from a plurality of cameras.

[0095] In one embodiment, features that can be detected within a space may include, but are not limited to, driving direction signs (510) on the floor of the space, signs for parking guidance (520), area identification signs (530), parking slots (540), pedestrians (550), etc. For example, although not illustrated in FIG. 5A, the layout of the space (e.g., walls, pillars, etc.) may also be detected as features.

[0096] Some of the features detected by the vehicle electronic device (2000) may be features that include semantic elements related to driving and / or parking of a vehicle. For example, a driving direction sign (510) on the floor of a space may have a meaning of guiding a vehicle to make a left turn at a fork in the road within the current space, which is a parking lot. Alternatively, a sign (520) for parking guidance may have a meaning of indicating that the current location is the third basement level (B3), that one should go left when attempting to park, that there is one empty parking slot, and that one should go right when attempting to exit the parking lot. Alternatively, a parking occupancy indicator that may be placed at each parking slot may have a no-parking sign (542) (e.g., a red light) indicating that a vehicle is parked and the parking slot is occupied, or a parking available sign (544) (e.g., a green light) indicating that a parking slot is empty and a vehicle can be parked.

[0097] According to one embodiment, a vehicle electronic device (2000) can map features detected from an image to a spatial map. In this case, features containing semantic elements among the detected features can be mapped to a semantic feature layer of the spatial map. The semantic feature layer of the spatial map can be used by the vehicle electronic device (2000) to determine a driving path for automatic parking.

[0098] FIG. 5b is a diagram for explaining an operation of a vehicle electronic device detecting a feature from an image according to one embodiment of the present disclosure.

[0099] In one embodiment, the vehicle electronic device (2000) can detect one or more objects from an image acquired using a camera, and analyze the detected one or more objects to extract features that can be mapped to a spatial map.

[0100] For example, a vehicle electronic device (2000) can detect a sign (562) for parking guidance, an exit direction sign (564), etc., within a first image (560) acquired through a front camera, and extract features such as a driving direction, a parking slot direction, the number of parking slots, the current floor, and an exit direction.

[0101] For example, the vehicle electronic device (2000) can detect a parking slot occupancy sign (572) (e.g., a parking available sign or a parking not allowed sign) hanging from the ceiling in a second image (570) acquired through a second left-side camera (left rear camera), and extract features such as the occupancy status and empty status of the parking slot.

[0102] For example, the vehicle electronic device (2000) can detect another sign (582) for parking guidance in a third image (580) acquired through a rear camera, and extract features such as a driving direction, a parking slot direction, the number of parking slots, and the current floor.

[0103] In one embodiment, the vehicle electronic device (2000) may detect features using a feature detection model, which is an artificial intelligence model. The feature detection model may be an artificial intelligence model that receives an image as input and outputs data about features within the image. For example, the feature detection model may be an artificial intelligence model that outputs the locations and classes of features within the image, but is not limited thereto. The feature detection model may be trained using a training dataset in which feature classes are labeled. The feature detection model may be implemented using a known deep learning architecture and algorithm, or may be implemented through a modification of a known deep learning architecture and algorithm. For example, the feature detection model may use a Fully Convolutional Network (FCN) based on a Convolutional Neural Network (CNN), Mask R-CNN, etc., but is not limited thereto.

[0104] In one embodiment, the vehicle electronic device (2000) may estimate depth within an image using a depth estimation model, which is an artificial intelligence model. The depth estimation model may be an artificial intelligence model that receives an image as input and outputs depth information. For example, the depth estimation model may be an artificial intelligence model that outputs depth information of objects (e.g., walls, pillars, obstacles, etc.) within the image. The depth estimation model may be trained using a training dataset configured so that depth information corresponds to the image. The depth detection model may be implemented using a known deep learning architecture and algorithm, or may be implemented through a modification of a known deep learning architecture and algorithm. For example, the depth estimation model may use, but is not limited to, DepthNet, which is based on a convolutional neural network (CNN).

[0105] In one embodiment, the vehicle electronic device (2000) can obtain depth information using a sensor. The vehicle electronic device (2000) can obtain depth information of objects on the driving path of the vehicle using a sensor capable of obtaining depth information, for example, an RGB-depth (RGB-D) sensor. The RGB-D camera may include a depth sensor that obtains depth information of an object using a time-of-flight (ToF) method. However, the method by which the vehicle electronic device (2000) obtains depth information is not limited thereto. For example, the vehicle electronic device (2000) may also obtain depth information by estimating depth based on images obtained from each camera using a stereo camera.

[0106] In one embodiment, the vehicle electronic device (2000) may detect features using a plurality of feature detection models. For example, the vehicle electronic device (2000) may detect first features related to a space in which a vehicle is located using a first feature detection model. The first features may be features that can be used to identify the structure of the space in which the vehicle is located, and may include, but are not limited to, a drivable space, a parking slot, a layout of the space (e.g., walls, pillars, etc.), a size of the space (e.g., width, length, height, etc.), and a position and size of one or more static objects (e.g., cones, curbs, etc.) within the space. Alternatively, the vehicle electronic device (2000) may detect second features related to semantic elements related to driving and parking of the vehicle using a second feature detection model. The second features may include, but are not limited to, driving direction indicators within a space such as walls, pillars, road surfaces, and ceilings, zone identification indicators indicating the current location, and parking slot availability / unavailability indicators. The detected features may be mapped to a spatial map. The vehicle electronic device (2000) according to one embodiment may use multiple feature detection models to segment and detect features to be mapped to each layer of the spatial map within an image.

[0107] An electronic device (2000) for a vehicle according to one embodiment can map a plurality of features to a spatial map having a multi-layer structure. A description of the spatial map having a multi-layer structure generated by the electronic device for a vehicle (2000) is further described in the description of FIGS. 6A to 6C .

[0108] FIG. 5c is a diagram for explaining an operation of a vehicle electronic device detecting a feature from an image according to one embodiment of the present disclosure.

[0109] In one embodiment, the vehicle electronic device (2000) can use a camera to photograph a parking lot information sign (590) and extract features mappable to a spatial map from the parking lot information sign (590).

[0110] For example, the vehicle electronic device (2000) may photograph a parking lot information sign (590) at the entrance to a parking building when entering the parking building. Alternatively, the vehicle electronic device (2000) may photograph a parking lot information sign (590) at the entrance to each floor of the parking building.

[0111] The vehicle electronic device (2000) can extract the number of available parking spaces on each floor of a parking building as features from parking lot information signs (590) in the captured image. For example, the vehicle electronic device (2000) can identify that 70 parking spaces are available on the B2 floor, 102 parking spaces are available on the B3 floor, 222 parking spaces are available on the B4 floor, and 238 parking spaces are available on the B5 floor, and that there are no parking spaces available on the B6 floor.

[0112] The vehicle electronic device (2000) can determine a driving route with a specific parking floor as a temporary destination based on the number of available parking spaces on each floor. For example, the vehicle electronic device (2000) can determine the B2 floor, which is closest to the vehicle's current location, as the temporary destination. Alternatively, the vehicle electronic device (2000) can determine the B5 floor, which has the largest number of available parking spaces, as the temporary destination.

[0113] The vehicle electronic device (2000) controls the vehicle to drive, and can control the control devices (e.g., power unit and steering device) built into the vehicle to drive the vehicle using a plurality of cameras. In this case, the vehicle electronic device (2000) can move to a specific floor within a parking building, photograph the road surface, ramps, pillars, walls, ceilings, etc., and detect features from the photographed images to create a movement passageway space map for moving between floors within the parking building.

[0114] In one embodiment, when a vehicle arrives at a parking lot, the vehicle electronic device (2000) may use multiple cameras to capture images of the vehicle's surroundings and generate a spatial map of the parking lot. Thereafter, the vehicle electronic device (2000) may identify an empty parking slot for parking the vehicle based on the spatial map of the parking lot, and determine a driving path for parking the vehicle in the empty parking slot.

[0115] Meanwhile, according to the method described above, the vehicle electronic device (2000) can generate a space map of a passageway for movement between floors within a parking building and a space map of a parking lot floor. That is, the vehicle electronic device (2000) can generate a space map for the entire parking building by generating space maps corresponding to each parking lot floor and a space map of a passageway for movement between parking lot floors.

[0116] FIG. 6A is a diagram for explaining an operation of a vehicle electronic device generating a spatial map according to one embodiment of the present disclosure.

[0117] In one embodiment, the vehicle electronics device (2000) can generate a spatial map (600). The vehicle electronics device (2000) can generate the spatial map (600) using various spatial map generation algorithms. For example, the vehicle electronics device (2000) can generate a 3D map while estimating the location of the vehicle using a simultaneous localization and mapping (SLAM) algorithm, but the algorithm for generating the spatial map is not limited thereto.

[0118] The vehicle electronic device (2000) can acquire a first image (602) captured from the right front through the first right-side camera, and can acquire a second image (604) captured from the right rear through the second right-side camera. The vehicle electronic device (2000) can correct internal and external parameters of the cameras to correct camera distortion of the first right-side camera and the second right-side camera, detect feature points within the images, and map coordinates within the images to actual 3D space coordinates. As a more specific example, the vehicle electronic device (2000) can generate a 3D point cloud representing points within the 3D space, and analyze the 3D point cloud to extract spatial structures such as walls, floors, and ceilings. The vehicle electronic device (2000) can model the spatial structure based on the point cloud and generate a 3D map. Thereafter, the vehicle electronic device (2000) can generate an occupancy grid map based on the 3D map. However, the method by which the vehicle electronic device (2000) generates an occupancy grid map is not limited to the above-described example.

[0119] An in-vehicle electronic device (2000) according to one embodiment can map features to a spatial map. For example, as illustrated in FIG. 6A, features can be detected based on a first image (602) and a second image (604) acquired using a first right-side camera and a second right-side camera, and the features can be mapped to a spatial map (600) on the right side of the vehicle based on the vehicle's location. In the same manner, the in-vehicle electronic device (2000) can generate a spatial map and map features for all surrounding directions based on the vehicle's current location.

[0120] FIG. 6b is a drawing to further explain a spatial map generated by a vehicle electronic device according to one embodiment of the present disclosure.

[0121] In one embodiment, the spatial map (600) may be configured as a layered structure of data for representing indoor spatial information. For example, the spatial map may include, but is not limited to, a first layer (610) including spatial information, a second layer (620) including driving and parking-related information, and a third layer (630) including information on dynamic objects within the space. Features detected from images captured by the vehicle electronic device (2000) through multiple cameras may be mapped to each layer of the spatial map (600). For example, the vehicle electronic device (2000) may map first features to the first layer (610) and second features to the second layer (620). In addition, cells with the same coordinates within each layer of the spatial map (600) indicate the same location within the space. Cells included in the spatial map (600) may include location information (e.g., coordinates) and index information.

[0122] The first layer (610) of the spatial map (600) may be an occupancy grid map layer. The vehicle electronic device (2000) may generate a grid map composed of cells having a constant size and location. The occupancy grid map may include, but is not limited to, the index, location, and values ​​within each cell (e.g., information about an object occupying the cell).

[0123] The vehicle electronic device (2000) can update the occupancy status of each cell of the first layer (610). The vehicle electronic device (2000) can preprocess images acquired through multiple cameras and map features detected from the images to update the first layer (610). For example, the vehicle electronic device (2000) can mark cells in which objects (e.g., parked vehicles, pillars, walls, etc.) exist as occupied in the occupancy grid map, and mark cells corresponding to empty spaces as empty.

[0124] The second layer (610) of the spatial map (600) may be a semantic feature layer generated based on semantic features related to driving and parking. The vehicle electronic device (2000) may preprocess images acquired through multiple cameras and map features detected from the images to update the second layer (610). For example, the vehicle electronic device (2000) may map features such as driving direction indicators, identification indicators indicating current locations, available / unavailable parking slot indicators, and available parking space indicators, located within a space such as walls, pillars, road surfaces, and ceilings, based on the first layer (610) representing spatial information.

[0125] The third layer (630) of the spatial map (600) may be a dynamic object layer containing information on moving objects within the space. For example, the vehicle electronic device (2000) may map features of moving objects, such as pedestrians and vehicles, within the space that may become obstacles when the vehicle drives within the space, based on the first layer (610) representing spatial information.

[0126] FIG. 6c is a diagram illustrating features within layers of a spatial map generated by an in-vehicle electronic device according to one embodiment of the present disclosure.

[0127] Referring to FIG. 6c, features mapped to each layer are illustrated when a vehicle electronic device (2000) according to one embodiment maps features to a spatial map (600).

[0128] As a result of the vehicle electronic device (2000) performing feature mapping on the spatial map (600), features representing spatial information, such as drivable spaces, parking slots, lines (lanes, parking lines), cones, curbs, and pillars, may be mapped to the occupancy grid map layer, which is the first layer (610).

[0129] The second layer (620), which is a semantic feature layer, may be mapped with features representing driving-related semantic elements, such as wall-mounted driving direction signs, pillar-mounted driving direction signs, road-surface driving direction signs, driving direction signs hung from the ceiling, and exit signs on pillars or ceilings. In addition, features representing parking-related semantic elements, such as parking area identification signs (for example, signs for identifying areas that distinguish spaces so that a vehicle can be identified, such as C234 blocks, C233 blocks, etc.), ceiling-mounted parking slot occupancy signs, parking slot occupancy information, and the number of available parking slots on signs for ceiling-mounted parking guides, may be mapped.

[0130] The dynamic object layer, which is the third layer (630), may have features mapped to it, such as pedestrians and vehicles within the space, which are dynamic objects that become obstacles when a vehicle drives through the space.

[0131] FIG. 7 is a drawing for explaining an operation of a vehicle electronic device according to one embodiment of the present disclosure to determine a driving path for parking.

[0132] In one embodiment, the vehicle electronic device (2000) can generate a spatial map using the current location of the vehicle as a starting point and determine a driving route for parking based on the spatial map. In describing FIG. 7, the starting point, which is the current location of the vehicle electronic device, is A, and the vehicle electronic device (2000) has completed generating the spatial map, as an example.

[0133] Referring to Fig. 7, there is an empty parking slot (700) at the right rear based on the current location A of the vehicle. The vehicle electronic device (2000) can determine the driving path of the vehicle so that the vehicle can be parked in the empty parking slot (700).

[0134] The vehicle electronic device (2000) can acquire a front image (710) from a starting point A through a front camera. Since the front image (710) includes an exit direction sign and a sign for a parking guide hanging from the ceiling, the vehicle electronic device (2000) can extract features from the front image (710) and map them to a spatial map. For example, information indicating that the left direction in front is the direction toward the exit and the right direction is the direction for parking can be updated in the spatial map. Accordingly, the vehicle electronic device (2000) can determine that if the vehicle reaches point B and turns left, it is heading toward the exit, and if the vehicle reaches point B and turns right, it is heading toward the direction for searching for parking slots.

[0135] Additionally, the vehicle electronic device (2000) can acquire a rear image (720) from a starting point A through a rear camera. Since the rear image (720) includes a sign (722) for parking guidance hanging from the ceiling, the vehicle electronic device (2000) can extract features from the rear image (720) and map them to a spatial map. For example, information indicating that there are no empty parking slots in the left rear direction and that there is one empty parking slot in the right rear direction can be updated in the spatial map. Accordingly, the vehicle electronic device (2000) can determine an empty parking slot in the right rear direction as a destination candidate.

[0136] In addition, the vehicle electronic device (2000) can acquire a right rear image (730) obtained through the second right side camera at the starting point A. Since the right rear image (730) includes a parking slot occupancy sign (732) hanging from the ceiling, the vehicle electronic device (2000) can extract features from the right rear image (730) and map them to a spatial map. For example, since the parking slot occupancy sign (732) hanging from the ceiling at the right rear indicates a parking availability sign (e.g., a green light), information indicating that there is one empty parking slot at the right rear can be updated in the spatial map. Accordingly, the vehicle electronic device (2000) can determine an empty parking slot located in the right rear direction as a destination candidate.

[0137] According to one embodiment, a vehicle electronic device (2000) may determine a driving path for parking a vehicle based on features updated in a spatial map. For example, the vehicle electronic device (2000) may determine a destination candidate based on a first layer of the spatial map. Specifically, the vehicle electronic device (2000) may determine a destination candidate (e.g., a parking slot) based on drivable spaces, parking slots, lines, etc. included in the first layer of the spatial map. In addition, the vehicle electronic device (2000) may generate a temporary driving path and a driving path based on a second layer of the spatial map. Specifically, the vehicle electronic device (2000) may generate a temporary driving path / driving path corresponding to a destination candidate based on wall driving direction marks, road driving direction marks, etc. included in the second layer of the spatial map. The vehicle electronic device (2000) may update the spatial map by sensing the surroundings as the vehicle drives, and may determine a final destination among the destination candidates as the spatial map is updated. Therefore, a temporary driving route may refer to a driving route before the final destination is determined.

[0138] In one embodiment, there may be one or more destination candidates. For example, there may be one or more empty parking slots. Accordingly, there may also be one or more temporary driving routes corresponding to the destination candidates. If there is one or more temporary driving routes, the vehicle electronic device (2000) may determine the priority of one or more temporary driving routes based on preset conditions. The preset conditions may include, but are not limited to, various conditions such as the distance from the vehicle to the destination candidate (empty parking slot), available parking space, and direction to the parking slot, but are not limited to the examples described above. The specific operation of the vehicle electronic device (2000) determining the temporary driving route will be further described with reference to FIG. 9.

[0139] As a result of determining a temporary driving route by a vehicle electronic device (2000) according to one embodiment, the temporary driving route may be determined to be a route that sequentially passes through points A, B, and C. In the example of Fig. 7, a case is illustrated where the temporary driving route to an empty parking slot (700), which is the destination, and the final driving route are the same.

[0140] FIG. 8 is a diagram for explaining an operation of a vehicle electronic device updating a spatial map according to one embodiment of the present disclosure.

[0141] In one embodiment, the vehicle electronic device (2000) may determine an empty parking slot (810) as a destination candidate and determine a temporary driving route that sequentially passes through points A, B, and C. However, the spatial map used when the vehicle electronic device (2000) determines the temporary driving route is not a spatial map generated for the entire space, but a spatial map generated based on the results of the vehicle electronic device (2000) sensing the surroundings based on the vehicle's position. Accordingly, the vehicle electronic device (2000) may update the spatial map, change the destination candidate, or change the driving route while driving along the temporary route.

[0142] The vehicle electronic device (2000) can acquire images of the surroundings of the vehicle using multiple cameras while the vehicle is driving. The vehicle electronic device (2000) can update a spatial map based on the acquired images. For example, if the vehicle reaches point B and changes direction to the right, new spatial images can be acquired for the left and front sides of the vehicle. In this case, the vehicle electronic device (2000) can update the spatial map based on the images acquired through the multiple cameras. In addition, the vehicle electronic device (2000) can detect features from the multiple images and map the features to a spatial map. For example, the vehicle electronic device (2000) can map first features related to space to a first layer (occupancy grid map layer) and second features related to semantic elements related to driving and parking of the vehicle to a second layer (semantic feature layer). Since the operation of the vehicle electronic device (2000) to generate a spatial map and map features has been described above, a repeated description of the specific operation is omitted.

[0143] In the same way, when the vehicle reaches point C and changes direction to the right, the vehicle electronic device (2000) can update the spatial map based on images acquired using multiple cameras. In this case, the vehicle electronic device (2000) can change the destination candidate. For example, when a new empty parking slot (820) is identified as a result of the vehicle electronic device (2000) updating the spatial map, the vehicle electronic device (2000) can determine the new empty parking slot (820) as the final destination and determine a route corresponding to the final destination. Alternatively, unlike the illustration, when the empty parking slot (820) does not exist in the space, the vehicle electronic device (2000) can determine the empty parking slot (810), which was the existing destination candidate, as the final destination and determine a route corresponding to the final destination. Once the route corresponding to the final destination is determined, the vehicle electronic device (2000) can control the driving of the vehicle so that the vehicle reaches the final destination.

[0144] In one embodiment, the vehicle electronic device (2000) may change the frame rate at which the plurality of cameras capture images of the surroundings of the vehicle based on the driving path and the destination direction while driving along the driving path. For example, the vehicle electronic device (2000) may maintain the capture frame rate of the cameras facing the driving path and the destination direction for which a new spatial map must be generated while driving, and reduce the capture frame rate of the cameras facing other directions. For example, the vehicle electronic device (2000) may reduce the capture frame rate of the cameras facing an area for which a spatial map has already been generated. The vehicle electronic device (2000) according to one embodiment may optimize the computing resources required for generating and updating the spatial map by adjusting the capture frame rate of the cameras.

[0145] Meanwhile, when the vehicle electronic device (2000) controls the driving and parking of the vehicle, the vehicle electronic device (2000) can obtain depth information of objects on the driving path of the vehicle using images acquired while the vehicle is driving, and control the driving and parking of the vehicle based on the depth information.

[0146] The operation of the vehicle electronic device (2000) to acquire depth information has been described above, so a repeated description is omitted.

[0147] FIG. 9 is a drawing for explaining an operation of a vehicle electronic device controlling a vehicle according to one embodiment of the present disclosure.

[0148] In describing FIG. 9, operations S910 to S950 may be performed after operations S210 to S250 of FIG. 2 are performed. In this case, the vehicle electronic device (2000) may control the vehicle to drive along a temporary driving path corresponding to a destination candidate.

[0149] In operation S910, the vehicle electronic device (2000) acquires a plurality of images while the vehicle is driving. The vehicle electronic device (2000) can acquire a plurality of images captured around the vehicle while the vehicle is driving by using a plurality of cameras mounted on the vehicle. The vehicle electronic device (2000) can detect features from the plurality of images acquired while the vehicle is driving. For example, the vehicle electronic device (2000) can detect first features such as a drivable space, a parking slot, a layout of the space (e.g., a wall, a pillar, etc.), a size of the space (e.g., width, length, height, etc.), a position and size of one or more static objects (e.g., a cone, a curb, etc.) within the space, and second features such as a driving direction indicator within the space such as a wall, pillar, road surface, or ceiling, an area identification indicator indicating a current location, an available / unavailable sign of a parking slot, an indication of the number of available parking spaces, etc.

[0150] In operation S920, the vehicle electronic device (2000) updates a spatial map. The vehicle electronic device (2000) can update the spatial map based on a plurality of images acquired while the vehicle is driving. The vehicle electronic device (2000) can expand the spatial map by generating a spatial map corresponding to the location where the vehicle is driving, and map features detected from surrounding images to the expanded spatial map. Since the operation of the vehicle electronic device (2000) updating the spatial map has been described above, a repeated description will be omitted.

[0151] In operation S930, the vehicle electronic device (2000) updates the driving route based on preset conditions.

[0152] The vehicle electronic device (2000) can update the spatial map while the vehicle is driving to a destination candidate and search for a new optimal driving path. In this case, multiple destination candidates can be searched while the vehicle is driving, and multiple provisional driving paths corresponding to each destination candidate can be calculated. Alternatively, multiple provisional driving paths can be calculated for a single destination candidate.

[0153] In one embodiment, when there is more than one temporary driving route, the vehicle electronic device (2000) can determine the priority of one or more temporary driving routes based on preset conditions.

[0154] For example, the vehicle electronic device (2000) may determine, among a plurality of temporary driving routes, a temporary driving route that includes a point (e.g., an intersection, etc.) that is closest in distance from the vehicle to a destination candidate (an empty parking slot) as having the highest priority.

[0155] In addition, the vehicle electronic device (2000) may set a temporary driving route including a destination candidate searched from a front camera, a first left-side camera looking at the left front, and a first right-side camera looking at the right front as having a higher priority when the distance from the vehicle to the destination candidate in each of a plurality of temporary driving routes is the same or similar.

[0156] In addition, the vehicle electronic device (2000) can set priorities for temporary driving routes by assigning weights to conditions of preset destination candidates (e.g., a place with a large area of ​​empty parking slots, a place with many empty parking slots, a place with two consecutive empty parking slots, etc.) among a plurality of temporary driving routes.

[0157] The vehicle electronic device (2000) may change the driving route when a new temporary driving route with a higher priority than the current temporary driving route is determined while updating the vehicle's driving route. If the driving route is changed, the vehicle electronic device (2000) may perform operation S910 again.

[0158] In operation S940, the vehicle electronic device (2000) controls vehicle driving based on the driving route. The vehicle electronic device (2000) can control control devices (e.g., powertrain and steering device) embedded in the vehicle to drive the vehicle along the driving route. Until the vehicle reaches the destination, the vehicle electronic device (2000) can repeatedly perform operations S910 to S930.

[0159] In operation S950, the vehicle electronic device (2000) performs a parking operation. When the vehicle arrives at the destination, the vehicle electronic device (2000) may perform a parking operation to automatically park the vehicle in an empty parking slot. Based on the completion of parking of the vehicle, the vehicle electronic device (2000) may provide the user with parking information of the vehicle. The parking information of the vehicle may include, but is not limited to, information such as the location where the vehicle is parked within the parking lot, the parking start time, and the parking fee.

[0160] Figure 10 is a drawing for explaining an operation performed using artificial intelligence technology in the present disclosure.

[0161] The operation of detecting features from an image and the operation of estimating the depth of the image, performed by a vehicle electronic device (2000) according to one embodiment, may be performed using artificial intelligence (AI) technology that performs operations through a neural network (1010).

[0162] Referring to FIG. 10, a neural network (1010) may be trained using training data. In addition, the neural network (1000) may receive input data (1011) as an input layer (1020), and the output layer (1040) may perform an operation to analyze the input data (1011) and output output data (1015), which is the desired result. The operation through the neural network may be performed through a hidden layer (1030). In FIG. 10, for convenience, the hidden layer (1030) is simplified and illustrated as being formed as a single layer, but the hidden layer (1030) may be formed as a plurality of layers.

[0163] In one embodiment, the neural network (1010) may be a feature detection model. The feature detection model may be an artificial intelligence model that inputs an image and outputs data about features within the image. For example, the feature detection model may be, but is not limited to, an artificial intelligence model that outputs the locations and classes of features within the image. The feature detection model may be trained using a training dataset in which feature classes are labeled. The feature detection model may be implemented using a known deep learning architecture and algorithm, or may be implemented through a modification of a known deep learning architecture and algorithm. For example, the feature detection model may use, but is not limited to, a Fully Convolutional Network (FCN) based on a Convolutional Neural Network (CNN), Mask R-CNN, etc.

[0164] In one embodiment, the neural network (1010) may be a depth estimation model. The depth estimation model may be an artificial intelligence model that inputs an image and outputs depth information. For example, the depth estimation model may be an artificial intelligence model that outputs depth information of objects (e.g., walls, pillars, obstacles, etc.) in the image. The depth estimation model may be trained using a training dataset configured so that depth information corresponds to the image. The depth detection model may be implemented using a known deep learning architecture and algorithm, or may be implemented through a modification of a known deep learning architecture and algorithm. For example, the depth estimation model may use, but is not limited to, DepthNet, which is based on a convolutional neural network (CNN).

[0165] In one embodiment, the operations for training and / or inference of the aforementioned neural network (1010) may be performed in the vehicle electronic device (2000) or may be performed in a server.

[0166] FIG. 11 is a block diagram illustrating a configuration of a vehicle electronic device according to one embodiment of the present disclosure.

[0167] Referring to FIG. 11, a vehicle electronic device (2000) according to one embodiment may include cameras (2100), a memory (2200), and a processor (2300).

[0168] In one embodiment, the number of cameras (2100) may be six. Specifically, the cameras (2100) may be configured with one front camera, one rear camera, and two side cameras (first side camera, second side camera) for each of the left and right sides. The first side camera may be a camera that captures the front of each of the left and right sides, and the second side camera may be a camera that captures the rear of each of the left and right sides. The vehicle electronic device (2000) may acquire multiple images capturing the surroundings of the vehicle using the cameras (2100). Specifically, when the cameras (2100) are configured with six cameras, six images captured in six directions from the vehicle may be acquired. The acquired multiple images may be used to analyze the surrounding space from the current location of the vehicle.

[0169] In one embodiment, the cameras (2100) may be configured with cameras having a wider vertical field of view than typical cameras. For example, a camera having an image acquired by the camera having a width:height ratio of 5:4 may be used, but the image ratio is not limited thereto. Surround cameras typically installed in vehicles are generally installed to view the road surface around the vehicle to recognize parking lines, lanes, etc., and thus acquire images having a narrow vertical field of view. In contrast, the cameras (2100) of the present disclosure are configured with cameras having a wide vertical field of view, and thus have a high vertical field of view (e.g., a field of view similar to that of a human), and thus can detect features existing on walls, pillars, ceilings, etc. That is, the cameras (2100) of the present disclosure are configured with cameras having a wide vertical field of view rather than wide-angle cameras having a wide horizontal field of view.

[0170] The memory (2200) may store instructions, data structures, and program codes that can be read by the processor (2300). There may be one or more memories (2200). In the disclosed embodiments, operations performed by the processor (2300) may be implemented by executing instructions or codes of a program stored in the memory (2200).

[0171] The memory (2200) may include non-volatile memory such as read-only memory (ROM) (e.g., programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory (e.g., memory card, solid-state drive (SSD)), and analog recording type (e.g., hard disk drive (HDD), magnetic tape, optical disk), and volatile memory such as random-access memory (RAM) (e.g., dynamic random-access memory (DRAM), static random-access memory (SRAM)).

[0172] The memory (2200) may store one or more instructions and / or programs that cause the vehicle electronic device to operate to control automatic parking of the vehicle. For example, the memory (2200) may store an image processing module (2210), a spatial map processing module (2220), and a vehicle control module (2230).

[0173] The processor (2300) can control the overall operations of the vehicle electronic device (2000). For example, the processor (2300) can control the overall operations of the vehicle electronic device (2000) for controlling automatic parking of the vehicle by executing one or more instructions of a program stored in the memory (2200). There can be one or more processors (2300).

[0174] The processor (2300) may be configured with at least one of, for example, a central processing unit, a microprocessor, a graphic processing unit, an application specific integrated circuits (ASICs), a digital signal processor (DSPs), a digital signal processing device (DSPDs), a programmable logic device (PLDs), a field programmable gate array (FPGAs), an application processor, a neural processing unit, or an artificial intelligence processor designed with a hardware structure specialized for processing an artificial intelligence model, but is not limited thereto.

[0175] In one embodiment, the processor (2300) executes an image processing module (2210) to store / manage images and extract various information from the images. The processor (2300) may extract features from the images and estimate depth information using one or more artificial intelligence models. Since the operations of the image processing module (2210) have been described above in the previous drawings, a repetitive description will be omitted.

[0176] In one embodiment, the processor (2300) may use the spatial map processing module (2220) to generate and manage a spatial map around the vehicle. The processor (2300) may generate the spatial map and map various features detected within the space onto the spatial map. The processor (2300) may update the spatial map based on sensor data (e.g., images) acquired while the vehicle is driving. Since the operations of the spatial map processing module (2220) have been described above in the previous drawings, a repeated description will be omitted.

[0177] In one embodiment, the processor (2300) may control the vehicle using the vehicle control module (2230). The processor (2300) may determine a driving path of the vehicle based on a spatial map and control control devices (e.g., a power unit and a steering device) embedded in the vehicle to drive the vehicle along the driving path. In addition, the processor (2300) may control control devices (e.g., a power unit and a steering device) embedded in the vehicle to perform a parking operation when the vehicle arrives at a destination. Since the description related to the operations of the vehicle control module (2230) has been described above in the previous drawings, a repeated description will be omitted.

[0178] Although not illustrated in FIG. 11, the vehicle electronic device (2000) may further include a communication interface including a communication circuit. The communication interface may perform data communication with other electronic devices under the control of the processor (2300). The communication interface may perform data communication between the electronic device (2000) and other electronic devices (e.g., a server) using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliances (WiGig), and RF communication.

[0179] In one embodiment, if the vehicle electronic device (2000) is capable of communicating with another server, at least some of the operations performed by the vehicle electronic device (2000) may be performed by the server. For example, the vehicle electronic device (2000) may receive an artificial intelligence model (e.g., a feature detection model, a depth estimation model, etc.) from the server. Alternatively, the vehicle electronic device (2000) may transmit images acquired through the cameras (2100) to the server and receive data required for automatic parking control of the vehicle (e.g., features, a spatial map, a driving route, etc.) from the server.

[0180] In addition, although not shown in FIG. 11, the vehicle electronic device (2000) may further include a plurality of sensors. For example, in addition to the cameras (2100), the vehicle electronic device (2000) may further include, but is not limited to, an RGB-depth (RGB-D) sensor, a ToF (Time of Flight) sensor, an IMU (Inertial Measurement Unit) sensor, a LIDAR (Light Detection and Ranging), a RADAR (Radio Detection and Ranging), a GPS (Global Positioning System) sensor, an ultrasonic sensor, and the like. The vehicle electronic device (2000) may acquire sensor data through the sensors and use the sensor data to perform operations of the vehicle electronic device described above (e.g., a feature extraction operation, a depth estimation operation, a spatial map generation operation, a driving operation, a parking operation, and the like).

[0181] Meanwhile, the modules stored in the aforementioned memory (2200) and executed by the processor (2300) are for convenience of explanation and are not limited to the illustrated examples. Other modules may be added to implement the aforementioned embodiments, a single module may be divided into multiple modules with distinct functions, and some of the aforementioned modules may be combined to be implemented as a single module.

[0182] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by a single processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first and second operations may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an AI-specific processor). Here, an AI-specific processor, which is an example of the second processor, may perform operations for training / inference of an AI model. However, the embodiments of the present disclosure are not limited thereto.

[0183] One or more processors (2300) according to the present disclosure may be implemented as a single-core processor or as a multi-core processor.

[0184] When a method according to one embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one core or may be performed by multiple cores included in one or more processors.

[0185] The present disclosure provides a vehicle electronic device that controls a vehicle to automatically navigate a path and park using in-vehicle sensors. The technical challenges addressed by the present disclosure are not limited to those described above, and other technical challenges not mentioned herein will be readily apparent to those skilled in the art, based on the description herein.

[0186] According to one aspect of the present disclosure, a vehicle electronic device may be provided. The vehicle electronic device may include: a plurality of cameras; a memory storing one or more instructions; and at least one processor executing the one or more instructions stored in the memory.

[0187] The at least one processor can acquire a plurality of images of the surroundings of the vehicle by using the plurality of cameras by executing the one or more instructions.

[0188] The at least one processor can detect features from the plurality of images by executing the one or more instructions.

[0189] The at least one processor may generate a spatial map around the vehicle based on the features by executing the one or more instructions.

[0190] The at least one processor can determine a driving path for parking the vehicle based on the spatial map by executing the one or more instructions.

[0191] The at least one processor can control driving of the vehicle based on the driving path by executing the one or more instructions.

[0192] The at least one processor may control the vehicle to perform a parking operation based on the vehicle arriving at a destination of the driving route by executing the one or more instructions.

[0193] The above multiple cameras may include a front camera, a rear camera, a first left-side camera, a second left-side camera, a first right-side camera, and a second right-side camera.

[0194] The at least one processor can detect, from the plurality of images, first features related to a space in which the vehicle is located and second features related to semantic elements related to driving and parking of the vehicle by executing the one or more instructions.

[0195] The above spatial map may include a first layer including spatial information and a second layer including driving and parking-related information.

[0196] The at least one processor can generate the first layer of the spatial map based on the first features by executing the one or more instructions.

[0197] The at least one processor can generate the second layer of the spatial map based on the second features by executing the one or more instructions.

[0198] The at least one processor can set one or more destination candidates based on the first layer of the spatial map by executing the one or more instructions.

[0199] The at least one processor may generate one or more temporary driving routes corresponding to one or more of the destination candidates based on the second layer of the spatial map by executing the one or more instructions.

[0200] The at least one processor may determine the priority of the one or more temporary driving routes based on a preset condition by executing the one or more instructions.

[0201] The at least one processor may determine the driving route of the vehicle based on a priority of the one or more temporary driving routes by executing the one or more instructions.

[0202] The at least one processor may update the spatial map based on images acquired using the plurality of cameras while the vehicle is driving by executing the one or more instructions.

[0203] The at least one processor can change the frame rate at which the plurality of cameras capture images around the vehicle based on the driving path and destination direction of the vehicle by executing the one or more instructions.

[0204] The at least one processor may provide parking information of the vehicle to the user based on the completion of parking of the vehicle by executing the one or more instructions.

[0205] The at least one processor can obtain depth information of objects on the driving path of the vehicle by executing the one or more instructions.

[0206] The at least one processor can control driving of the vehicle based on the depth information by executing the one or more instructions.

[0207] According to one aspect of the present disclosure, a method of operating an electronic device for a vehicle may be provided.

[0208] The above method may include a step of acquiring a plurality of images taken of the surroundings of the vehicle using a plurality of cameras.

[0209] The method may include a step of detecting features from the plurality of images.

[0210] The method may include a step of generating a spatial map around the vehicle based on the features.

[0211] The method may include a step of determining a driving path for parking the vehicle based on the spatial map.

[0212] The method may include a step of controlling driving of the vehicle based on the driving path.

[0213] The method may include a step of controlling the vehicle to perform a parking operation based on the vehicle arriving at a destination of the driving route.

[0214] The above multiple cameras may include a front camera, a rear camera, a first left-side camera, a second left-side camera, a first right-side camera, and a second right-side camera.

[0215] The step of detecting the above features may include a step of detecting, from the plurality of images, first features related to a space where the vehicle is located and second features related to semantic elements related to driving and parking of the vehicle.

[0216] The above spatial map may include a first layer including spatial information and a second layer including driving and parking-related information.

[0217] The step of generating the spatial map may include the step of generating the first layer of the spatial map based on the first features.

[0218] The step of generating the spatial map may include the step of generating the second layer of the spatial map based on the second features.

[0219] The step of determining the driving route may include a step of setting one or more destination candidates based on the first layer of the spatial map.

[0220] The step of determining the driving path may include a step of generating one or more temporary driving paths corresponding to one or more of the destination candidates based on the second layer of the spatial map.

[0221] The step of determining the driving route may include a step of determining priorities of the one or more temporary driving routes based on preset conditions.

[0222] The step of determining the driving route may include a step of determining the driving route of the vehicle based on a priority of the one or more temporary driving routes.

[0223] The method may include a step of updating the spatial map based on images acquired using the plurality of cameras while the vehicle is driving.

[0224] The method may include a step of changing a frame rate at which the plurality of cameras capture images of the surroundings of the vehicle based on the driving path and destination direction of the vehicle.

[0225] The method may include a step of providing parking information of the vehicle to a user.

[0226] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include computer storage media and communication media. Computer storage media include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include computer-readable instructions, data structures, or other data in a modulated data signal, such as program modules.

[0227] Additionally, a computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0228] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0229] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0230] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. In vehicle electronic devices (2000), Multiple cameras (2100); A memory (2200) storing one or more instructions; and At least one processor (2300) for executing one or more instructions stored in the memory, The at least one processor (2300) executes the one or more instructions, Acquire multiple images of the surroundings of the vehicle using the above multiple cameras, Detect features from the above multiple images, Generate a spatial map around the vehicle based on the above features, Determine a driving path for parking the vehicle based on the above space map, Controlling the driving of the vehicle based on the driving path, An electronic device for a vehicle that controls the vehicle to perform a parking operation based on the vehicle arriving at a destination of the driving route.

2. In paragraph 1, The at least one processor, by executing the one or more instructions, An electronic device for a vehicle, which detects, from the plurality of images, first features related to a space in which the vehicle is located and second features related to semantic elements related to driving and parking of the vehicle.

3. In paragraph 2, The above spatial map includes a first layer containing spatial information, and a second layer containing driving and parking-related information. The at least one processor, by executing the one or more instructions, Based on the above first features, the first layer of the spatial map is generated, An electronic device for a vehicle, which generates the second layer of the spatial map based on the second features.

4. In paragraph 3, The at least one processor, by executing the one or more instructions, Based on the first layer of the above space map, one or more destination candidates are set, An in-vehicle electronic device that generates one or more temporary driving routes corresponding to one or more destination candidates based on the second layer of the spatial map.

5. In paragraph 4, The at least one processor, by executing the one or more instructions, Determine the priority of one or more temporary driving routes based on preset conditions, An electronic device for a vehicle, which determines the driving route of the vehicle based on the priorities of one or more temporary driving routes.

6. In paragraph 1, The at least one processor, by executing the one or more instructions, An electronic device for a vehicle that updates the spatial map based on images acquired using the plurality of cameras while the vehicle is driving.

7. In paragraph 6, The at least one processor, by executing the one or more instructions, An electronic device for a vehicle, which changes the frame rate at which the plurality of cameras capture images of the surroundings of the vehicle based on the driving path and destination direction of the vehicle.

8. In the operating method of a vehicle electronic device, Step (S210) of acquiring multiple images captured around a vehicle using multiple cameras; Step of detecting features from the above multiple images (S220); A step of generating a spatial map around the vehicle based on the above features (S230); Step (S240) of determining a driving path for parking the vehicle based on the above space map; A step (S250) of controlling the driving of the vehicle based on the driving path; and A method comprising a step (S260) of controlling the vehicle to perform a parking operation based on the vehicle arriving at a destination of the driving route.

9. In paragraph 8, The step of detecting the above features is, A method comprising the step of detecting, from the plurality of images, first features related to a space in which the vehicle is located and second features related to semantic elements related to driving and parking of the vehicle.

10. In paragraph 9, The above spatial map includes a first layer containing spatial information, and a second layer containing driving and parking-related information. The steps for generating the above spatial map are: A step of generating the first layer of the spatial map based on the first features; and A method comprising the step of generating the second layer of the spatial map based on the second features.

11. In paragraph 10, The step of determining the above driving route is: A step of setting one or more destination candidates based on the first layer of the above space map; and A method comprising the step of generating one or more temporary driving routes corresponding to one or more of the destination candidates based on the second layer of the spatial map.

12. In paragraph 11, The step of determining the above driving route is: A step of determining priorities of one or more temporary driving routes based on preset conditions; and A method comprising the step of determining the driving route of the vehicle based on the priorities of the one or more temporary driving routes.

13. In paragraph 8, The above method, A method further comprising the step of updating the spatial map based on images acquired using the plurality of cameras while the vehicle is driving.

14. In paragraph 13, The above method, A method further comprising the step of changing the frame rate at which the plurality of cameras capture images around the vehicle based on the driving path and destination direction of the vehicle.

15. A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 8 to 14 on a computer.

Images