Vehicle and control method thereof

Abstract

The present disclosure provides a vehicle, comprising: a camera unit disposed in the vehicle, having multiple channels and configured to acquire images of the vehicle's surroundings, the camera unit comprising one or more cameras; a sensing device comprising an ultrasonic sensor and configured to acquire distance information between an object and the vehicle; and a controller configured to match a portion of the image of the vehicle's surroundings with at least one mask, form map information based on the at least one mask and the distance information, determine at least one control point based on the map information, and acquire images of the vehicle's surroundings based on priorities of the camera units corresponding to types of the vehicle's surroundings determined based on the control points.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2019-0166111, filed on December 12, 2019, in the Korean Intellectual Property Office, which is hereby incorporated by reference. Technical Field

[0003] The present disclosure relates to a vehicle for recognizing an image around the vehicle and a method for controlling the vehicle. Background Art

[0004] A vehicle's autonomous driving technology is a technology that allows the vehicle to understand road conditions and drive automatically even if the driver does not control the brakes, steering wheel, or accelerator pedal.

[0005] Autonomous driving technology is the core technology for realizing smart cars. For autonomous driving, autonomous driving technologies can include Highway Driving Assist (HDA), which automatically maintains the distance between vehicles, Back Side Warning (BSD), which senses surrounding vehicles and issues warnings during reversing, Automatic Emergency Braking (AEB), which activates the braking system when the vehicle does not recognize the vehicle in front, Lane Departure Warning (LDWS), Lane Keeping Assist (LKAS), which compensates for lane departure without a turn signal, Advanced Smart Cruise Control (ASCC), which maintains the distance between vehicles at a set speed, Traffic Jam Assist (TJA), Parking Collision Avoidance Assist (PCA), and Remote Smart Parking Assist (RSPA).

[0006] Specifically, the RSPA system uses only ultrasonic sensors to identify parking spaces, and thus can perform automatic parking by generating a control trajectory only when there is a vehicle nearby.

[0007] In order to increase the integrity of the parking space when there is no vehicle or parking device, a recognition system is needed that recognizes the lane type outside the vehicle and transmits the lane type to the control system. Summary of the Invention

[0008] An aspect of an embodiment of the present disclosure provides a vehicle capable of efficiently and automatically parking by changing a recognition area of ​​a camera according to a type of parking and a method of controlling the vehicle.

[0009] Additional embodiments of the disclosure will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the disclosure.

[0010] According to an embodiment of the present disclosure, a vehicle includes: a camera disposed in the vehicle, having multiple channels and configured to acquire images of the vehicle's surroundings; a sensing device including an ultrasonic sensor and configured to acquire distance information between an object and the vehicle; and a controller configured to match a portion of the image of the vehicle's surroundings with at least one mask, form map information based on the at least one mask and the distance information, determine at least one control point based on the map information, and acquire images of the vehicle's surroundings based on a priority of the camera corresponding to a surrounding type of the vehicle determined based on the control point.

[0011] The map information may include distance information corresponding to pixels of an image around the vehicle.

[0012] The controller may be configured to transform an image of the vehicle's surroundings into a vehicle coordinate system to match the at least one mask.

[0013] The controller may be configured to determine the type of surroundings of the vehicle by learning images of the surroundings of the vehicle.

[0014] When parallel parking is performed on one side of the vehicle, the controller may be configured to assign priority to a lane on one side of the vehicle among the plurality of lanes.

[0015] When performing reverse diagonal parking of the vehicle, the controller may be configured to assign priority to a lane in front of the vehicle among the plurality of lanes.

[0016] When performing forward diagonal parking of the vehicle, the controller may be configured to assign priority to a lane on one side of the vehicle among the plurality of lanes.

[0017] When performing rear parking of the vehicle, the controller may be configured to assign priority to a lane behind the vehicle among the plurality of lanes.

[0018] The controller may be configured to change priorities in real time in response to the movement of the vehicle.

[0019] The vehicle may further include a display. The controller may be configured to form a top view image based on map information of the vehicle, and form a boundary line in the top view image based on the priority information and output the boundary line to the display.

[0020] According to another aspect of the present disclosure, a method for controlling a vehicle includes: a camera having multiple channels acquiring an image around the vehicle; a sensing device acquiring distance information between an object and the vehicle; a controller matching a portion of the image around the vehicle with at least one mask; the controller forming map information based on the at least one mask and the distance information; the controller determining at least one control point based on the map information; and the controller acquiring an image around the vehicle based on a priority of the camera corresponding to a surrounding type of the vehicle determined based on the control point.

[0021] The map information may include distance information corresponding to pixels of an image around the vehicle.

[0022] Matching a portion of the image surrounding the vehicle with the at least one mask may include converting the image surrounding the vehicle to a vehicle coordinate system to match with the at least one mask.

[0023] Acquiring the image of the surroundings of the vehicle may include determining the surrounding type of the vehicle by learning the image of the surroundings of the vehicle.

[0024] Acquiring the image around the vehicle may include assigning priority to a lane on one side of the vehicle among the plurality of lanes when parallel parking is performed on one side of the vehicle.

[0025] Acquiring the image around the vehicle may include assigning priority to a lane in front of the vehicle among the plurality of lanes when performing reverse diagonal parking of the vehicle.

[0026] Acquiring the image around the vehicle may include assigning priority to a lane on one side of the vehicle among the plurality of lanes when performing parallel diagonal parking of the vehicle.

[0027] Acquiring the image around the vehicle may include assigning priority to a lane behind the vehicle among the plurality of lanes when performing rear parking of the vehicle.

[0028] The method may further include the controller changing the priority in real time in response to the movement of the vehicle.

[0029] The method may further include: the controller forming a top-view image based on map information of the vehicle; and the controller forming a boundary line in the top-view image based on the priority information and outputting the boundary line to the display. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] These and / or other aspects of the present disclosure will become more apparent and more readily understood through the following description of embodiments in conjunction with the accompanying drawings, in which:

[0031] Figure 1 is a control block diagram according to an embodiment;

[0032] Figure 2is a diagram for describing an operation of determining map information based on an image around a vehicle according to an embodiment;

[0033] Figure 3 is a diagram showing map information according to an embodiment;

[0034] Figure 4 is a diagram illustrating forming a control point according to an embodiment;

[0035] Figures 5 to 8 is a diagram for describing the priorities of cameras assigned according to surrounding types;

[0036] Figure 9 is a diagram illustrating formation of a boundary in a top-view image according to an embodiment; and

[0037] Figure 10 is a flow chart according to an embodiment. DETAILED DESCRIPTION

[0038] Throughout the specification, the same reference numerals refer to the same elements. Not all elements of the embodiments of the present disclosure will be described, and the description of the contents that are well known in the art or that overlap with each other in the embodiments will be omitted. The terms used throughout the specification, such as "~ components", "~ modules", "~ members", "~ blocks", etc., can be implemented with software and / or hardware, and multiple "~ components", "~ modules", "~ members" or "~ blocks" can be implemented in a single element, or a single "~ component", "~ module", "~ member" or "~ block" can include multiple elements.

[0039] It will be further understood that the term "connection" and its derivatives refer to both direct connection and indirect connection, and indirect connection includes the connection through a wireless communication network. Unless otherwise indicated, the terms "include" and "comprising" are inclusive or open, and do not exclude other undescribed elements or method steps. It will be further understood that the term "member" and its derivatives refer to the situation where a member contacts another member, and also refer to the situation where another member exists between two members. It will be further understood that although the terms "first", "second", "third" etc. can be used herein to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish an element, component, region, layer or part from another element, component, region, layer or part.

[0040] It will be understood that the singular forms "a," "an," and "the" also include the plural forms unless the context clearly indicates otherwise. The reference numerals for the method steps are used only for ease of explanation and are not intended to limit the order of the steps. Therefore, unless the context clearly indicates otherwise, the steps may be performed in other orders.

[0041] Hereinafter, operation principles and embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0042] Figure 1 is a control block diagram according to an embodiment.

[0043] Reference Figure 1 , the vehicle 1 according to the embodiment may include a camera unit 300 , a sensing device 100 , a display 400 , and a controller 200 .

[0044] The camera unit 300 may include one or more cameras 300. The camera unit 300 has multiple channels and may acquire images around the vehicle 1. Hereinafter, the term "camera 300" may refer to a camera unit or a single camera or multiple cameras of the camera unit.

[0045] The one or more cameras 300 installed in the vehicle 1 may include a charge-coupled device (CCD) camera or a CMOS color image sensor. Here, CCD and CMOS both refer to sensors that convert light received through the lens of the camera 300 into electrical signals and store the electrical signals.

[0046] The sensing device 100 may include an ultrasonic sensor.

[0047] The ultrasonic sensor may employ a method of emitting ultrasonic waves and detecting the distance to the obstacle using the ultrasonic waves reflected from the obstacle.

[0048] The sensing device 100 may acquire distance information between the vehicle 1 and obstacles disposed around the vehicle 1 .

[0049] The display 400 may be provided as a display device provided in an instrument panel or on a center instrument panel in the vehicle 1 .

[0050] The display 400 may include a cathode ray tube (CRT), a digital light processing (DLP) panel, a plasma display panel (PDP), a liquid crystal display (LCD) panel, an electroluminescent (EL) panel, an electrophoretic display (EPD) panel, an electrochromic display (ECD) panel, a light emitting diode (LED) panel, or an organic light emitting diode (OLED) panel, but is not limited thereto.

[0051] The controller 200 may match a portion of the image around the vehicle 1 with at least one mask. That is, an obstacle, a floor, etc. shown in the image may be matched with a corresponding mask.

[0052] The controller 200 may form map information based on at least one mask and the distance information.

[0053] The map information may refer to information including distance information between the vehicle 1 and surrounding obstacles.

[0054] The controller 200 may acquire an image of the surroundings of the vehicle 1 based on the priorities of the cameras 300 corresponding to the surrounding types of the vehicle 1 determined based on the map information.

[0055] The surrounding type may refer to a relationship between the vehicle 1 and an obstacle, a relationship between the vehicle 1 and a road, and the like.

[0056] The priority may refer to information related to the recognition area of ​​the camera 300 .

[0057] The map information may include distance information corresponding to pixels of an image around the vehicle 1. That is, the map information may be provided as information in which pixels of an image around the vehicle 1 match the distance between the vehicle 1 and an obstacle.

[0058] The controller 200 may convert an image around the vehicle 1 into a vehicle coordinate system to correspond to at least one mask.

[0059] That is, the controller 200 may acquire an image around the vehicle 1 in a coordinate system centered on the camera 300 , but the controller 200 may convert the image around the vehicle 1 into a coordinate system of the vehicle itself to form map information.

[0060] The controller 200 may determine the surrounding type of the vehicle 1 by learning images around the vehicle 1 .

[0061] The learning performed by the controller 200 may be performed through deep learning.

[0062] Deep learning is a field of machine learning and may refer to a form of representing data as a vector or graph, etc. that can be processed by a computer and constructing a model for learning the data.

[0063] A deep learning model may be formed based on a neural network, and in particular, a model may be constructed by stacking multiple layers of neural networks to form a deep learning model.

[0064] When parallel parking is performed on one side of the vehicle 1 , the controller 200 may assign priority to a lane on one side of the vehicle 1 among the plurality of lanes.

[0065] When performing reverse diagonal parking of the vehicle 1 , the controller 200 may assign priority to a lane in front of the vehicle 1 among the plurality of lanes.

[0066] When performing forward diagonal parking of the vehicle 1 , the controller 200 may assign priority to a lane on one side of the vehicle 1 among the plurality of lanes.

[0067] When performing rear parking of the vehicle 1 , the controller 200 may assign priority to a lane behind the vehicle 1 among the plurality of lanes.

[0068] The controller 200 may change the priority in real time in response to the travel of the vehicle 1 .

[0069] The priorities of the cameras 300 that are changed in response to the parking of the vehicle 1 described above will be described in detail below.

[0070] The controller 200 may form a bird's-eye view image based on map information of the vehicle 1 , and may form a boundary line in the bird's-eye view image based on the priority information and output the boundary line to the display 400 .

[0071] The boundary line formed in the top-view image may refer to a recognition area of ​​each of the cameras 300 .

[0072] Controller 200 can be implemented using a memory and a processor. The memory stores an algorithm for controlling the operation of components in vehicle 1 or data related to a program that reproduces the algorithm. The processor uses the data stored in the memory to perform the aforementioned operations. The memory and processor can be implemented as separate chips. Alternatively, the memory and processor can be implemented as a single chip.

[0073] Can be based on Figure 1 The performance of the components of the vehicle 1 shown may include adding or deleting at least one component. It will be readily understood by those skilled in the art that the relative positions of the components may be changed according to the performance or structure of the vehicle 1.

[0074] at the same time, Figure 1 Each of the components shown may refer to software and / or hardware components such as field programmable gate arrays (FPGAs) and application specific integrated circuits (ASICs).

[0075] Figure 2 is a diagram for describing an operation of determining map information based on an image around a vehicle according to an embodiment, Figure 3 is a diagram showing map information according to an embodiment.

[0076] Reference Figure 2 , shows an external image V2 acquired by the vehicle 1. The cameras 300 provided in the vehicle 1 may be provided at the front, rear, and sides to acquire images around the vehicle 1.

[0077] The controller 200 may match the mask with the acquired image of the surroundings of the vehicle 1 .

[0078] Meanwhile, during this process, the controller 200 may convert the coordinates of each of the cameras 300 into the coordinates of the vehicle 1. In particular, the mask M2-1 may be matched to obstacles of the vehicle such as shown in the image around the vehicle 1.

[0079] Additionally, empty spaces on the road can be matched with a different mask M2-2.

[0080] On the other hand, the floor or ground may be matched with another mask M2-3.The controller 200 may match the mask with the distance information corresponding to each pixel.

[0081] The controller 200 can determine the map information F2 based on such operations.

[0082] At the same time, Figure 2 , it is shown that the map information is derived based on the vehicle image acquired by the camera 300 in response to one camera 300 , but each of the cameras 300 provided in the vehicle 1 may perform a corresponding operation.

[0083] At the same time, when the map information of each of the cameras 300 determined as described above is collected, it can be finally obtained that Figure 3 Map information F3 is shown.

[0084] The map information may be determined based on the recognition results and the distance coordinate system of the camera 300 and the ultrasonic sensor. The controller 200 may provide a method for providing a free space and a control point for parking control using distance map data for each pixel of each camera 300 based on the map information.

[0085] The above operation may form map information based on the image of the camera 300 suitable for space recognition according to the form of parking spaces around the vehicle 1 .

[0086] Such an operation minimizes the occlusion of the image and can quickly determine the time point at which the recognition space is occupied.

[0087] In addition, the map information can be used to generate an optimal distance map format for recognition by comparing the recognition results of the cameras 300 at different locations.

[0088] Additionally, control performance can be optimized by generating the coordinates of entry points required for object control.

[0089] at the same time, Figure 3The distance map shown is merely an example of the present disclosure, and the distance map may be represented in various forms and is not limited in form.

[0090] Figure 4 is a diagram illustrating forming control points according to an embodiment.

[0091] Reference Figure 4 As described above, in the case of map information, distance information can be included. The controller 200 can control the vehicle 1 by avoiding each control point by forming control points P41, P42, and P43 for each obstacle rather than the entire obstacle. That is, in the object control of the controller 200, point information close to the progress of the vehicle 1 can be used instead of using information for each obstacle to maximize control efficiency. The surrounding types based on this subdivision will be described below.

[0092] Figures 5 to 8 3 is a diagram for describing priorities of the cameras 300 assigned according to surrounding types.

[0093] Reference Figure 5 , showing a surrounding type in which a parking space S5 is formed on the right side of the vehicle 1.

[0094] In addition, the controller 200 can control the vehicle 1 based on the two control points P51 and P52. Since the two control points are located on one side of the vehicle, the controller 200 can assign a higher priority to the side camera 300. The controller 200 can perform side parking by widening the width of the area Ca51 recognized by the side camera 300.

[0095] Reference Figure 6 , showing the situation of reverse diagonal parking.

[0096] Even in this case, the controller 200 can control the vehicle 1 based on the two control points P61 and P62. Although the two control points are located on one side of the vehicle 1, the two control points are located on different sides. Figure 5 The position of the control point is determined, therefore, it is necessary to determine the obstacles ahead to park the vehicle in the corresponding parking area.

[0097] The controller 200 may assign a higher priority to the front camera 300. The controller 200 may park the vehicle in the area Ca61 recognized by the front camera 300.

[0098] Reference Figure 7 , showing the situation of parking in a forward diagonal direction.

[0099] Even in this case, the controller 200 can control the vehicle 1 based on the two control points P71 and P72. Although the two control points are located on one side of the vehicle 1, the two control points are located on different sides. Figure 6 The position of the control point is determined, therefore, the obstacles on the side need to be determined to park the vehicle in the corresponding parking area.

[0100] The controller 200 may assign a higher priority to the side camera 300. The controller 200 may park the vehicle in the area Ca71 recognized by the side camera 300.

[0101] Reference Figure 8 , showing a surrounding type in which a parking space is formed behind the vehicle 1.

[0102] The controller 200 can control the vehicle 1 based on the two control points P81 and P82. Since the two control points are located behind the vehicle, the controller 200 can assign a higher priority to the rear camera 300. The controller 200 can perform rear parking by widening the width of the area recognized by the rear camera 300.

[0103] On the other hand, when the vehicle 1 is traveling, the surrounding situation may change in real time, and the position and control point of the vehicle 1 may also change in real time. Therefore, the controller 200 may change the priority of the camera 300 in real time by considering the positional relationship between the vehicle 1 and the control point.

[0104] Figure 9 is a diagram illustrating formation of a boundary in a top-view image according to an embodiment.

[0105] Reference Figures 5 to 8 , the recognition area of ​​the camera 300 provided in the vehicle 1 may be changed. At the same time, the controller 200 may use information of the plurality of cameras 300 when forming the overhead view image, and may display the recognition area reflecting the priority of the camera 300 as a boundary in the overhead view image.

[0106] For example, in Figure 5 When the vehicle 1 performs parallel side parking, a boundary line L81 where the recognition area of ​​the side camera 300 is large may be displayed in the top view image.

[0107] On the other hand, when Figure 8 As shown, when the vehicle 1 performs rear parking, a boundary line L82 where the recognition area of ​​the rear camera 300 is relatively large may be displayed in the top-view image.

[0108] Meanwhile, the above operation is merely one embodiment for describing the operation of the present disclosure, and the operation of forming map information according to distance and changing the priority or recognition area of ​​the camera 300 accordingly is not limited.

[0109] Figure 10 is a flow chart according to an embodiment.

[0110] Reference Figure 10 , vehicle 1 can obtain images and distance information around vehicle 1 (1001).

[0111] In addition, the vehicle 1 may form map information based on an image and distance information around the vehicle 1 ( 1002 ).

[0112] In addition, the map information may include distance information of each obstacle, and the controller 200 may determine a control point based on the distance information of each obstacle ( 1003 ).

[0113] In addition, the controller 200 may determine a surrounding type based on a positional relationship between the vehicle 1 and the control point ( 1004 ).

[0114] In response to this type, the controller 200 may assign a priority to each of the cameras 300 and control the vehicle 1 based on the recognition area of ​​the camera 300 to which the priority is assigned ( 1005 ).

[0115] According to an embodiment of the present disclosure, the vehicle 1 and the control method of the vehicle 1 can change the recognition area of ​​the camera according to the type of parking, thereby enabling effective automatic parking.

[0116] The disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions executable by a processor. The instructions may be stored in the form of program code, and when executed by the processor, the instructions may generate a program module to perform the operations of the disclosed embodiments. The recording medium may be implemented as a non-transitory computer-readable recording medium.

[0117] The non-transitory computer-readable recording medium may include all types of recording media that store commands that can be interpreted by a computer. For example, the non-transitory computer-readable recording medium may be a ROM, RAM, magnetic tape, magnetic disk, flash memory, optical data storage device, etc.

[0118] The embodiments of the present disclosure are described above with reference to the accompanying drawings. It should be apparent to those skilled in the art that the present disclosure can be implemented in other forms other than the above-described embodiments without changing the technical ideas or basic features of the present disclosure. The above embodiments are merely exemplary and should not be interpreted in a limiting sense.

Claims

1. A vehicle comprising: a camera unit, disposed in the vehicle, having a plurality of channels and acquiring images around the vehicle, the camera unit comprising one or more cameras; a sensing device comprising an ultrasonic sensor and acquiring distance information between an object and the vehicle; as well as a controller that matches a portion of an image surrounding the vehicle to at least one mask, the portion of the image including obstacles, empty spaces of a road, and a ground surface; forming map information based on the at least one mask and the distance information; determining at least one control point related to the obstacle based on the map information; and acquiring an image of the surroundings of the vehicle based on the priorities of the camera units corresponding to the surrounding types of the vehicle determined based on the control points, The surrounding type of the vehicle includes the parking type of the vehicle.

2. The vehicle according to claim 1, wherein The map information includes the distance information corresponding to pixels of an image around the vehicle.

3. The vehicle according to claim 1, wherein The controller converts the image of the vehicle's surroundings to a vehicle coordinate system to match the at least one mask.

4. The vehicle according to claim 1, wherein The controller determines a surrounding type of the vehicle by learning an image of the surroundings of the vehicle.

5. The vehicle according to claim 1, wherein When parallel parking is performed on one side of the vehicle, the controller assigns the priority to a lane on one side of the vehicle among the plurality of lanes.

6. The vehicle according to claim 1, wherein When performing reverse diagonal parking of the vehicle, the controller assigns the priority to a lane in front of the vehicle among the plurality of lanes.

7. The vehicle according to claim 1, wherein When performing forward diagonal parking of the vehicle, the controller assigns the priority to a lane on one side of the vehicle among the plurality of lanes.

8. The vehicle according to claim 1, wherein When performing rear parking of the vehicle, the controller assigns the priority to a lane behind the vehicle among the plurality of lanes.

9. The vehicle according to claim 1, wherein The controller changes the priority in real time in response to the travel of the vehicle.

10. The vehicle of claim 1, further comprising: monitor, The controller forms a bird's-eye view image based on map information of the vehicle, and forms a boundary line in the bird's-eye view image based on priority information and outputs the boundary line to the display.

11. A method for controlling a vehicle, comprising: A camera unit with multiple channels acquires images around the vehicle, wherein the camera unit includes one or more cameras; The sensing device obtains distance information between the object and the vehicle; matching a portion of an image surrounding the vehicle to at least one mask, the portion of the image including obstacles, empty spaces of a road, and a ground surface; forming map information based on the at least one mask and the distance information; determining at least one control point associated with the obstacle based on the map information; as well as acquiring an image of the surroundings of the vehicle based on priorities of the camera units corresponding to the surrounding types of the vehicle determined based on the control points, The surrounding type of the vehicle includes the parking type of the vehicle.

12. The method according to claim 11, wherein The map information includes the distance information corresponding to pixels of an image around the vehicle.

13. The method according to claim 11, wherein Matching a portion of the image surrounding the vehicle to at least one mask includes: The image of the vehicle's surroundings is transformed into a vehicle coordinate system to match the at least one mask.

14. The method according to claim 11, wherein Acquiring an image around the vehicle includes: The surrounding type of the vehicle is determined by learning images of the surroundings of the vehicle.

15. The method according to claim 11, wherein Acquiring an image around the vehicle includes: When parallel parking is performed on one side of the vehicle, the priority is assigned to a lane on one side of the vehicle among the plurality of lanes.

16. The method according to claim 11, wherein Acquiring an image around the vehicle includes: When reverse diagonal parking of the vehicle is performed, the priority is assigned to a lane in front of the vehicle among the plurality of lanes.

17. The method according to claim 11, wherein Acquiring an image around the vehicle includes: When performing forward diagonal parking of the vehicle, the priority is assigned to a lane on one side of the vehicle among the plurality of lanes.

18. The method according to claim 11, wherein Acquiring an image around the vehicle includes: When rear parking of the vehicle is performed, the priority is assigned to a lane behind the vehicle among the plurality of lanes.

19. The method of claim 11, further comprising: The priority is changed in real time in response to the travel of the vehicle.

20. The method of claim 11, further comprising: forming a bird's-eye view image based on map information of the vehicle; forming a boundary line in the top view image based on the priority information; as well as The boundary line is output to a display.

Images