Method for recognizing a vehicle behind based on an image, vehicle and readable medium

Abstract

This invention discloses a method for vehicle recognition based on images, a vehicle thereof, and a readable medium. A method for identifying a vehicle behind another vehicle includes: acquiring an image captured by a camera while driving; determining a time zone based on the acquired image; performing object recognition based on the determined time zone; determining the image coordinates of the vehicle behind another vehicle based on the result of the object recognition; and converting the determined image coordinates into distance.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of Korean Patent Application No. 10-2022-0053688, filed on April 29, 2022, which is incorporated herein by reference as if fully set forth herein. Technical Field

[0003] This disclosure relates to rear collision avoidance technology, and more specifically, to technology for identifying rear vehicles based on raw images captured by cameras mounted on the vehicle. Background Technology

[0004] The Rear Collision Warning (RCW) system warns the driver of a vehicle when a risk of collision with an oncoming vehicle is detected from behind. RCW systems typically operate at speeds of 30 km / h or lower and aim to minimize injury from a rear-end collision by activating the vehicle's hazard lights or engaging the seatbelts when such a collision is anticipated. In this situation, the vehicle behind, sensing the collision risk, can reduce its forward acceleration by applying its brakes before the impending collision.

[0005] US Patent Publication (US20070296564A1) and US Patent Publication (US006831572B2) disclose a method and device for preventing a collision by using a RADAR mounted on the rear bumper of a vehicle to identify an approaching vehicle from behind and to issue a warning alert to the driver when a collision is anticipated.

[0006] Currently, the detection range of typical Advanced Driver Assistance Systems (ADAS) sensors is concentrated in front of the vehicle. Under normal conditions, the vehicle moves forward, and therefore, when a hazard is detected ahead, the ADAS system can control the vehicle to avoid a frontal collision. However, there is a drawback that collisions caused by approaching vehicles from behind are not actively handled.

[0007] Various sensors, such as RADAR, LiDAR, cameras, and ultrasonic sensors, are installed at the front of the vehicle to enhance the ability to identify hazards ahead and provide a variety of driving assistance functions; however, typically only sensors for parking assistance are installed at the rear of the vehicle.

[0008] When active sensors that transmit actual signals (such as RADAR and LiDAR) are installed at the rear of a vehicle, these sensors may interfere with the front sensors of the following vehicles and degrade the performance of their driver assistance systems. Furthermore, the installation of expensive active sensors such as RADAR and LiDAR for limited rear collision control increases the vehicle's price.

[0009] In the case of typical rear sensors, because the detection distance is relatively short compared to front sensors, a collision may occur before a collision response can be initiated if a vehicle approaches from behind at high speed.

[0010] Therefore, there is a need for technologies that can effectively prevent rear-end collisions without significantly increasing vehicle prices. Summary of the Invention

[0011] The purpose of this disclosure is to provide a method, apparatus, and vehicle for image-based vehicle recognition.

[0012] Another object of this disclosure is to provide a method for vehicle recognition based on camera images, for recognizing vehicles behind a vehicle based on raw images from a camera mounted at the rear of the vehicle, as well as an apparatus and a vehicle for the method.

[0013] Another object of this disclosure is to provide a method, apparatus, and vehicle for image-based vehicle identification using existing cameras to identify rear vehicles without adding rear active sensors and separate hardware, in order to prevent performance degradation of the front sensors of the rear vehicles.

[0014] Another objective of this disclosure is to provide a method, apparatus, and vehicle for image recognition of remotely positioned vehicles based on a proposed object recognition deep learning network.

[0015] Another object of this disclosure is to provide a method, apparatus, and vehicle for image-based vehicle recognition that enhances object recognition performance by classifying and constraining learning parameters applied to a deep learning network for object recognition based on time zone, day / night, and weather.

[0016] Those skilled in the art will recognize that the purposes achievable through this disclosure are not limited to those specifically described above, and that the above and other purposes achievable through this disclosure will become clearer from the following detailed description.

[0017] According to one aspect, a method for identifying a vehicle behind another vehicle while driving includes: acquiring an image captured by a camera while driving; determining a time zone based on the acquired image; performing object recognition based on the determined time zone; determining the image coordinates of the vehicle behind another vehicle based on the result of the object recognition; and converting the determined image coordinates into a distance.

[0018] According to an implementation, determining a time zone based on the acquired image may include: acquiring the output value of a light sensor; and calculating the average brightness of the image by correcting the image, wherein the time zone may be determined based on the output value of the light sensor and the average brightness of the image.

[0019] According to the implementation, the learning parameters of the object recognition deep learning network can be set based on a determined time zone, and the object recognition deep learning network can include: a base network configured to extract features from an image; and an object detection head configured to learn from a dataset to classify anchor boxes into candidate regions with fixed positions, and to adjust the position and size of ground truth bounding boxes with an intersection-over-union (IoU) ratio equal to or greater than a reference value within the classified anchor boxes.

[0020] According to the implementation method, the time zone may include a night zone and a day zone. Based on the determination that the time zone is a day zone, the learning parameters of the object recognition deep learning network can be set to preset day parameters, and based on the determination that the time zone is a night zone, the learning parameters of the object recognition deep learning network can be set to preset night parameters.

[0021] According to the implementation, the nighttime parameters may include a first nighttime parameter for identifying the headlight area and a second nighttime parameter for identifying the rear vehicle area.

[0022] According to an implementation, the method may further include: when the time zone is a night zone, acquiring a fixed exposure image by setting the camera to a fixed exposure time, identifying the headlight area based on the fixed exposure image, acquiring an automatic exposure image by setting the camera to automatic exposure, and identifying the rear vehicle area based on the automatic exposure image.

[0023] According to an implementation, the method may further include determining whether the headlight area is included in the rear vehicle area, wherein when the headlight area is included in the rear vehicle area, the lower end of the rear vehicle area can be determined as the lower end of the rear vehicle, and the distance to the rear vehicle can be calculated based on the determined lower end of the rear vehicle and the camera calibration value.

[0024] According to an implementation, the method may further include, when the headlight area is successfully identified but the rear vehicle area is not identified, calculating the maximum headlight mounting height based on the position of the headlight area, wherein the lower end of the rear vehicle can be determined based on the lower end of the headlight area and the maximum headlight mounting height, and the distance to the rear vehicle can be calculated based on the determined lower end of the rear vehicle and the camera calibration value.

[0025] According to the implementation, the calibration values ​​may include a focal length value as the distance from the camera to the image plane, a vertical mounting height value from the ground to the camera, and a vertical mounting angle value of the camera.

[0026] According to the implementation method, the camera may include a surround view camera.

[0027] According to an implementation, the method may further include calculating the speed of the vehicle behind based on the change in distance over a predetermined time period, and outputting a rear collision warning alarm based on the distance and speed.

[0028] On the other hand, a non-volatile computer-readable storage medium is provided that stores at least one computer program including instructions, which, when executed by at least one processor, cause the at least one processor to perform a rear vehicle recognition operation, the operation including: acquiring an image captured by a camera while driving; determining a time zone based on the acquired image; performing object recognition based on the determined time zone; determining the image coordinates of a rear vehicle based on the result of the object recognition; and converting the determined image coordinates into a distance.

[0029] According to another aspect, a vehicle includes: a camera configured to capture a rear view image while driving; a time zone determiner configured to determine a time zone based on the rear view image; an object recognizer configured to recognize an object based on the determined time zone; and a calculator configured to determine the image coordinates of the rear vehicle based on the result of the object recognition, and convert the determined image coordinates into distance.

[0030] According to an embodiment, the vehicle may further include: a light sensor configured to measure ambient illuminance; and a preprocessor configured to correct an image by correcting only the region of interest of the rear image, wherein a time zone determiner can calculate the average brightness of the corrected image and determine the time zone based on the output value of the light sensor and the average brightness.

[0031] According to an embodiment, the vehicle may further include a memory configured to store a plurality of learning parameters for an object recognition deep learning network, wherein the object recognizer can identify an object by selecting and setting at least one of the plurality of learning parameters according to a determined time zone.

[0032] According to the implementation, the object recognition deep learning network may include: a base network configured to extract features from an image; and an object detection head configured to learn from a dataset to classify anchor boxes into candidate regions with fixed positions, and to adjust the position and size of ground truth bounding boxes with an intersection-over-union (IoU) ratio equal to or greater than a reference value within the classified anchor boxes.

[0033] According to the implementation method, the time zone may include a night zone and a day zone. Based on the determination that the time zone is a day zone, the learning parameters of the object recognition deep learning network can be set to preset day parameters, and based on the determination that the time zone is a night zone, the learning parameters of the object recognition deep learning network can be set to preset night parameters.

[0034] According to the implementation, the nighttime parameters may include a first nighttime parameter for identifying the headlight area and a second nighttime parameter for identifying the rear vehicle area.

[0035] According to the implementation, when the time zone is a night zone, the object recognizer can identify the headlight area based on a fixed-exposure image obtained by setting the camera to a fixed exposure time, and identify the rear vehicle area based on an automatic-exposure image obtained by setting the camera to automatic exposure.

[0036] According to the implementation method, when the headlight area is included in the rear vehicle area, the calculator can determine the lower end of the rear vehicle area as the lower end of the rear vehicle, and calculate the distance to the rear vehicle based on the lower end of the rear vehicle and the camera calibration value.

[0037] According to the implementation method, when the headlight area is successfully identified but the rear vehicle area is not identified, the calculator can calculate the maximum headlight installation height based on the position of the headlight area, determine the lower end of the rear vehicle based on the lower end of the headlight area and the maximum headlight installation height, and calculate the distance to the rear vehicle based on the lower end of the rear vehicle and the camera calibration value.

[0038] According to the implementation, the calibration values ​​may include a focal length value as the distance from the camera to the image plane, a vertical mounting height value from the ground to the camera, and a vertical mounting angle value of the camera.

[0039] According to one implementation, the camera may include a surround-view camera.

[0040] According to the implementation, the calculator can calculate the speed of the vehicle behind based on the change in distance over a predetermined time period, and the vehicle can further include a warning alarm component configured to generate a rear collision warning alarm based on distance and speed.

[0041] The aspects disclosed herein are only a part of the preferred embodiments of the present disclosure, and those skilled in the art can design and understand various embodiments based on the technical features of the present disclosure based on the detailed description of the present disclosure. Attached Figure Description

[0042] The accompanying drawings are included to provide a further understanding of this disclosure and are incorporated in and constitute a part of this application. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of this disclosure.

[0043] Figure 1 This is a block diagram illustrating the configuration of a rear vehicle recognition device according to an embodiment of the present disclosure.

[0044] Figure 2 This is a flowchart of a method for recognizing vehicles based on images according to embodiments of the present disclosure.

[0045] Figure 3 This is a flowchart of a method for determining the time zone for setting learning parameters of an object recognition deep learning network according to an embodiment of the present disclosure.

[0046] Figure 4 This is a flowchart illustrating a method for calculating the distance and speed of a vehicle behind by setting learning parameters of an object recognition deep learning network according to an embodiment of the present disclosure, based on a determined time zone.

[0047] Figure 5 This is a flowchart illustrating the process of calculating the distance to a vehicle behind by setting learning parameters corresponding to a night zone, according to an embodiment of this disclosure.

[0048] Figure 6 The process of recognizing rear objects based on camera images according to an embodiment of the present disclosure is illustrated.

[0049] Figure 7 A method for object recognition in nighttime areas according to an embodiment of the present disclosure is shown.

[0050] Figure 8 An IoU calculation method according to an embodiment of the present disclosure is shown.

[0051] Figure 9 A method for calculating the distance to a vehicle behind is shown according to an embodiment of the present disclosure. Detailed Implementation

[0052] In the following description, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, similar reference numerals denote similar elements, although the elements are shown in different drawings. Furthermore, in the following description of embodiments of the present disclosure, detailed descriptions of known functions and configurations incorporated herein will be omitted for clarity and brevity.

[0053] It should be understood that while the terms first, second, A, B, (a), (b), etc., may be used herein to describe various elements of this disclosure, these terms are only used to distinguish one element from another, and the nature, order, or sequence of the corresponding elements is not limited by these terms. Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will be further understood that terms such as those defined in common dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant field, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0054] In the various examples disclosed herein, " / " and "," should be interpreted as indicating "and / or". For example, "A / B" can mean "A and / or B". Furthermore, "A, B" can mean "A and / or B". Additionally, "A / B / C" can mean "at least one of A, B, and / or C". Furthermore, "A, B, and C" can mean "at least one of A, B, and / or C".

[0055] In the various examples of this disclosure, "or" needs to be interpreted as indicating "and / or". For example, "A or B" can include "A only", "B only", and / or "both A and B". In other words, "or" needs to be interpreted as indicating "additionally or alternatively".

[0056] In the following text, reference will be made to Figures 1 to 9 The embodiments of this disclosure are described in detail.

[0057] Figure 1 This is a block diagram illustrating the configuration of a rear vehicle recognition device according to an embodiment of the present disclosure.

[0058] In the following implementation, the rear vehicle identification device is described as being installed on a vehicle as an example, but this is only one implementation and can be applied to other transportation equipment, such as UAM, drones and ships.

[0059] Reference Figure 1 The rear vehicle recognition device 100 may generally include a sensing component 10, a controller 20, and an output interface 30.

[0060] The sensing component 10 can provide various sensing information to the controller 20.

[0061] For example, sensing component 10 may include camera 11 and light sensor 12.

[0062] Camera 11 can capture rear images through a fisheye lens and can provide the captured rear images (in the following text, used in conjunction with the original images) to controller 20.

[0063] The light sensor 12 can measure ambient illuminance and provide the measured illuminance value to the controller 20.

[0064] The controller 20 may include at least one of a preprocessor 21, a time zone determiner 22, an object recognizer 23, a memory 24, an alarm component 25, and a calculator 26.

[0065] The preprocessor 21 can preprocess the sensing information acquired from the sensing unit 10.

[0066] For example, preprocessor 21 can generate a corrected image by correcting the original image captured by camera 11. Specifically, preprocessor 21 can generate the corrected image by removing distorted areas of the original image based on lens information from camera 11 and correcting only the region of interest. The region of interest (i.e., the area behind the vehicle) is an area with low distortion and can therefore be identified by distant vehicles during correction.

[0067] The time zone determiner 22 can calculate the average brightness of the corrected image and determine the time zone based on the output value of the light sensor 12 and the average brightness value of the corrected image.

[0068] According to the implementation method, the time zone can be divided into a night zone and a day zone.

[0069] For example, when the output value of the light sensor corresponds to "nighttime" and the average brightness value of the corrected image is equal to or less than the reference value, the time zone determiner 22 can determine the time zone as a nighttime zone. If there are many streetlights around the road and the average brightness value of the corrected image exceeds the reference value even at night, the time zone can be determined as a daytime zone.

[0070] According to the implementation method, when the determined time zone is a daytime zone, the learning parameters of the object recognition deep learning network used for object recognition can be set to preset daytime parameters, and when the determined time zone is a nighttime zone, the learning parameters of the object recognition deep learning network can be set to preset nighttime parameters. By classifying and setting the learning parameters optimized for daytime and nighttime, object recognition performance can be advantageously improved.

[0071] According to the implementation, the nighttime parameters may include a first nighttime parameter for identifying the headlight area of ​​a vehicle behind and a second nighttime parameter for identifying the area of ​​a vehicle behind.

[0072] The memory 24 can hold multiple learning parameters used in the aforementioned object recognition deep learning network.

[0073] When the time zone is a night zone, the object recognizer 23 can acquire a fixed exposure image by setting the camera 11 to a fixed exposure time, and identify the headlight area based on the acquired fixed exposure image.

[0074] When the time zone is nighttime, the object recognizer 23 can acquire an auto-exposure image by setting the camera 11 to auto-exposure, and identify the vehicle area behind based on the acquired auto-exposure image.

[0075] As a result, even when the time zone is nighttime, the object recognizer 23 can also use short-exposure images and automatic-exposure images to identify the headlight area and the area of ​​the vehicle behind an approaching vehicle.

[0076] When the object recognizer 23 successfully identifies both the headlight area and the rear vehicle area, the calculator 26 can determine whether the headlight area is included in the rear vehicle area. As a result, if the headlight area is included in the rear vehicle area, the calculator 26 can determine the lower edge of the rear vehicle area as the lower edge of the rear vehicle. The calculator 26 can calculate the distance to the rear vehicle based on the lower edge of the rear vehicle and the camera's calibration value. Here, the specific method for calculating the distance to the rear vehicle based on image coordinates and calibration values ​​becomes clearer through the explanation of the accompanying drawings, which will be described later.

[0077] When the object recognizer 23 successfully identifies the headlight area but fails to identify the rear vehicle area, the calculator 26 can calculate the maximum headlight mounting height based on the position of the identified headlight area, and determine the lower end of the rear vehicle based on the lower end of the headlight area and the calculated maximum headlight mounting height. The calculator 26 can calculate the distance to the rear vehicle based on the lower end of the rear vehicle and the calibration value of the rear camera. Here, the specific method for calculating the distance to the rear vehicle based on image coordinates and calibration values ​​becomes clearer with the explanation of the accompanying drawings, which will be described later.

[0078] According to the implementation, the calibration values ​​may include: focal length, i.e., the distance from the camera to the image plane; vertical mounting height from the ground to the camera; and vertical mounting angle of the camera.

[0079] In this specification, a camera is a device for acquiring images, and the name is not intended to limit the scope of this disclosure. For example, a camera may be a rear-view camera for capturing images of the area behind a vehicle. Alternatively, a camera may include at least one camera configured as an image sensor or at least one surround-view camera. A surround-view camera includes at least one camera or at least one image sensor configured to monitor the area around a vehicle.

[0080] The calculator 26 according to the implementation method can also calculate the speed of the vehicle behind based on the change in distance from the vehicle behind within a predetermined time.

[0081] The warning alarm component 25 can adaptively generate a rear collision warning alarm based on the distance to the vehicle behind and / or the speed of the vehicle behind calculated by the calculator 26, and can output the rear collision warning alarm through the output interface 30.

[0082] The output interface 30 may include a speaker, a display and a vibration element, and output various warning and alarm messages in the form of voice / image / text / vibration modes.

[0083] Figure 2 This is a flowchart illustrating a method for image-based vehicle recognition according to embodiments of the present disclosure.

[0084] Reference Figure 2 The vehicle can acquire raw images captured by the camera while driving (S210).

[0085] The vehicle can determine the time zone based on the output value of the light sensor and the acquired raw image (S220). Here, the determined time zone can be either a daytime zone or a nighttime zone. The vehicle can generate a corrected image for a central region (i.e., a region of interest) with relatively small distortion between the output value of the light sensor and the raw image, and calculate the average brightness value of the generated corrected image. For example, the vehicle can determine the time zone as a nighttime zone based on the light sensor value being "nighttime" and the average brightness value being less than or equal to a predetermined reference value, and in other cases, the vehicle can determine the time zone as a daytime zone.

[0086] The vehicle can perform object recognition based on the determined time zone (S230).

[0087] According to the implementation, when the time zone is nighttime, the vehicle can acquire a fixed-exposure image by setting camera 11 to a fixed exposure time, and identify the headlight area based on the acquired fixed-exposure image. Furthermore, when the time zone is nighttime, the vehicle can acquire an automatic-exposure image by setting camera 11 to automatic exposure, and identify the rear vehicle area based on the acquired automatic-exposure image. As a result, in the case of nighttime, the vehicle can use both short-exposure images and automatic-exposure images to identify the headlight area of ​​an approaching vehicle and the rear vehicle area.

[0088] The vehicle can determine the image coordinates of the vehicle behind it based on the result of object recognition (S240).

[0089] The vehicle can convert the determined image coordinates into distance (S250).

[0090] When both the headlight area and the rear vehicle area are successfully identified, the vehicle can determine whether the headlight area is included within the rear vehicle area. As a result, if the headlight area is included within the rear vehicle area, the vehicle can determine the lower edge of the rear vehicle area as the lower edge of the rear vehicle. The vehicle can then calculate the distance to the rear vehicle based on the lower edge of the rear vehicle and the camera's calibration values. The specific method for calculating the distance to the rear vehicle based on image coordinates and calibration values ​​will become clearer through the accompanying drawings described later.

[0091] When the headlight area is successfully identified but the rear vehicle area fails to be identified, the vehicle can calculate the maximum headlight mounting height based on the position of the identified headlight area, and determine the lower end of the rear vehicle based on the lower end of the headlight area and the calculated maximum headlight mounting height. The vehicle can then calculate the distance to the rear vehicle based on the lower end of the rear vehicle as determined by the camera's calibration values. The specific method for calculating the distance to the rear vehicle based on image coordinates and calibration values ​​will become clearer with the accompanying drawings described later.

[0092] According to the implementation, the calibration values ​​may include: focal length, i.e., the distance from the camera to the image plane; vertical mounting height from the ground to the camera; and vertical mounting angle of the camera.

[0093] In this specification, a camera is a device for acquiring images, and the name is not intended to limit the scope of this disclosure. For example, a camera may be a rear-view camera for photographing the area behind a vehicle. Alternatively, a camera may include at least one camera configured as an image sensor or at least one surround-view camera. Figure 3 This is a flowchart illustrating a method for determining the time zone for setting learning parameters of an object recognition deep learning network according to embodiments of the present disclosure.

[0094] refer to Figure 3 The vehicle can acquire raw images captured by the camera while driving (S310).

[0095] The vehicle can generate a corrected image by correcting the region of interest in the original image and calculating the average brightness of the generated corrected image (S320).

[0096] The vehicle can acquire the output value of the light sensor (S330).

[0097] The vehicle can determine the time zone based on the output value of the light sensor and the calculated average brightness value (S340).

[0098] According to the implementation method, the time zone can be classified into night zone and day zone.

[0099] According to the implementation method, when the determined time zone is a daytime zone, the learning parameters of the object recognition deep learning network used for object recognition can be set to preset daytime parameters, and when the determined time zone is a nighttime zone, the learning parameters of the object recognition deep learning network can be set to preset nighttime parameters. The performance of subsequent object recognition can be improved by adaptively setting the learning parameters optimized for daytime and nighttime.

[0100] According to the implementation, the nighttime parameters may include a first nighttime parameter for identifying the headlight area and a second nighttime parameter for identifying the rear vehicle area.

[0101] Figure 4 This is a flowchart illustrating a method for calculating the distance and speed of a vehicle behind by setting learning parameters of an object recognition deep learning network according to a determined time zone, as described in embodiments of this disclosure.

[0102] refer to Figure 4 The vehicle can determine whether the determined time zone is a daytime zone (S410).

[0103] As a result, when the time zone is the daytime zone, the vehicle can set the learning parameters of the object recognition deep learning network to the daytime parameters (S420).

[0104] As a result of operation 410, when the time zone is a night zone, the vehicle can set the learning parameters of the object recognition deep learning network to night parameters (S430).

[0105] The vehicle can determine (or estimate) the image coordinates of the identified rear vehicle based on the output value of the object recognition deep learning network (S440).

[0106] The vehicle can calculate the distance to the identified rear vehicle based on the determined (or estimated) image coordinates and the camera's calibration values ​​(S450).

[0107] The vehicle can calculate the speed of the identified following vehicle based on the change in distance per unit time (S460).

[0108] The vehicle can output the distance and speed calculated for the vehicle behind it via an output device included in the vehicle (S470).

[0109] For example, a vehicle determines the likelihood of a rear-end collision based on the distance to an approaching vehicle and the speed of that vehicle. As a result of this determination, when the risk of a rear-end collision is detected, a predefined warning message can be generated and output.

[0110] According to the implementation method, when a risk of rear-end collision is detected, the vehicle can control the vehicle's braking / driving speed / hazard warning lights / active seat belts by controlling the corresponding electronic control components, thereby reducing the risk of additional collisions with the vehicle in front and user injury in the event of a rear-end collision.

[0111] Figure 5 This is a flowchart illustrating the process of calculating the distance to a vehicle behind by setting learning parameters corresponding to a night zone, according to an embodiment of the present disclosure.

[0112] refer to Figure 5 When the identification process for vehicles behind in the night zone begins, the vehicle can acquire a fixed-exposure image by setting the camera to a fixed exposure time (S501).

[0113] The vehicle can be set with first night learning parameters for nighttime headlight recognition (S502).

[0114] The vehicle can identify the headlight area based on a fixed-exposure image (S503). That is, the vehicle can identify the headlight area by inputting a fixed-exposure image into an object recognition deep learning network.

[0115] The vehicle can acquire an auto-exposure image by setting the camera to auto-exposure (S504).

[0116] The vehicle can be set with a second nighttime learning parameter (S505) for the recognition of vehicles behind.

[0117] The vehicle can identify the area of ​​vehicles behind it based on the automatically exposed image (S506). That is, the vehicle can identify the area of ​​vehicles behind it by inputting the automatically exposed image into an object recognition deep learning network.

[0118] The vehicle can determine whether the headlight area is included in the identified rear vehicle area (S507).

[0119] As a result, when the headlight area is not included in the rear vehicle area, the vehicle can calculate the maximum headlight mounting height based on the position of the headlight area (S508). In this case, the maximum headlight height value occupies multiple pixels in the corresponding image but few pixels at a distance, and therefore the vehicle can compensate for this by applying a predefined lookup table (LUT) filter based on the position of the headlight area.

[0120] The vehicle can use the sum of the lower end of the headlight area and the maximum headlight mounting height to determine the lower end of the vehicle behind it (S509).

[0121] Conversely, as a result of operation 507, when the headlight area is included in the rear vehicle area, the vehicle can determine the lower end of the identified rear vehicle area as the lower end of the rear vehicle (S510).

[0122] The vehicle can calculate the distance to the vehicle behind based on the camera's calibration value and the determined lower end of the vehicle behind (S511).

[0123] As described above, the vehicle according to this disclosure can estimate the position of the vehicle behind using two corrected images (i.e., a fixed exposure image and an automatically exposed image obtained by setting the exposure time in the night zone differently), thereby advantageously halving the output frame rate and improving object recognition performance in the night zone.

[0124] Figure 6 The process of recognizing rear objects based on camera images according to an embodiment of the present disclosure is illustrated.

[0125] Reference Figure 6 The camera's raw image 610 can be input to the preprocessor. In this case, the camera's raw image 610 can be corrected by the preprocessor and converted into a corrected image 620, with the distorted parts removed from the corrected image 620 by a fisheye lens.

[0126] The base network of the object recognition deep learning network can generate an anchor-box-based learning image 630 by extracting features from the input correction image 620.

[0127] The object recognition head can classify anchor boxes, candidate regions, or predicted bounding boxes based on features extracted by the base network and input from the anchor box-based learning image 630, and adjust the position and size of the ground reality bounding boxes.

[0128] According to the implementation method, a classification network such as ResNet or VGG16 can be applied as the base network, and learning can begin using weight values ​​determined by learning from a dataset such as ILSVRC2012.

[0129] The object recognition head can use the network output values ​​of anchor boxes with an intersection-over-union (IoU) ratio of 50% or greater to perform learning to obtain location and classification values, where the ground reality bounding boxes are within anchor boxes with fixed locations.

[0130] Here, we can use Formula 1 below to represent the network output value.

[0131] [Formula 1]

[0132]

[0133] Here, x a y a h a and w a It can be the center point (x) a y a The constant values ​​for the height and width of the anchor frame are indicated by reference numeral 630. x, y, w, and h can represent the center point (x, y), height (h), and width (w) of the identified object, and can be obtained using the network output values ​​of Equation 1. c can be a classification value relating to the vehicle's background and state. According to the implementation, L1 loss can be used to learn the location values, and cross-entropy loss can be used to learn the classification values. The network weights can be updated using a stochastic gradient descent method. As a result, an object recognition result image 640 can be obtained.

[0134] Figure 7 A method for identifying objects in nighttime areas according to an embodiment of this disclosure is shown.

[0135] refer to Figure 7 Reference number 710 can identify the headlight area of ​​an approaching vehicle from behind by learning from fixed exposure images.

[0136] Referring to reference number 720, the area behind vehicles can be identified by learning from automatically exposed images.

[0137] Figure 8 An IoU calculation method according to an embodiment of the present disclosure is shown.

[0138] refer to Figure 8 IoU can be calculated by dividing the overlapping area between the anchor frame and the ground reality bounding box by the union area of ​​the anchor frame and the ground reality bounding box.

[0139] Figure 9 A method for calculating the distance to a vehicle behind is shown according to an embodiment of the present disclosure.

[0140] refer to Figure 9 When assuming the ground is level, the distance (d) to the identified vehicle behind can be calculated using Formula 2 below.

[0141] [Formula 2]

[0142]

[0143] Here, y is the image coordinate, f is the focal distance from the camera to the image plane, and Z is the focal distance. cam It refers to the vertical installation height of the camera above the ground, and also to the vertical installation angle of the camera.

[0144] Here, the focal distance can be calculated in image coordinates.

[0145] Surround-view monitoring (SVM) systems, which include surround-view cameras, can be calibrated during vehicle manufacturing to transform the bird's-eye view, and thus information such as the mounting location, mounting angle, and focal distance of each camera can be predetermined and stored in memory included in the vehicle.

[0146] The operation of the methods or algorithms described in the embodiments disclosed herein can be implemented in hardware and software modules, or a combination of both, executed by a processor. The software modules may always reside in storage media (i.e., memory and / or data storage) such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, or CD-ROM.

[0147] An exemplary storage medium can be coupled to a processor, and the processor can read information from and write information to the storage medium. Alternatively, the storage medium can be integrated with the processor. The processor and storage medium can always reside within an application-specific integrated circuit (ASIC). The ASIC can always reside within the user equipment (UE). Alternatively, the processor and storage medium can always be separate components within the UE.

[0148] According to embodiments of this disclosure, methods, apparatus, and vehicles based on image recognition of vehicles can be advantageously provided.

[0149] According to embodiments of the present disclosure, a method for image-based vehicle recognition, as well as apparatus and vehicle for the method, can be advantageously provided for recognizing a vehicle behind the vehicle based on raw images from an SVM camera mounted at the rear of the vehicle.

[0150] According to embodiments of this disclosure, a method, apparatus, and vehicle thereof can be advantageously provided for identifying vehicles based on images using existing cameras to prevent performance degradation of front sensors of rear vehicles without adding rear active sensors and separate hardware.

[0151] According to embodiments of this disclosure, a remote rear vehicle can be identified using an object recognition deep learning network based on an SVM camera image, thereby advantageously and effectively preventing collisions with rear vehicles.

[0152] According to embodiments of this disclosure, an image-based vehicle recognition method, apparatus, and vehicle can be advantageously provided to improve object recognition performance and RCW performance by dynamically determining the learning parameters applied to a proposed object recognition deep learning network based on time zone, day / night, and weather conditions.

[0153] In addition, various effects that can be directly or indirectly identified through this document may be provided.

[0154] The above description is merely an illustration of the technical concept of this disclosure, and those skilled in the art to which this disclosure pertains can make various modifications and changes without departing from the basic characteristics of this disclosure.

[0155] Therefore, the embodiments disclosed herein are not intended to limit the technical spirit of this disclosure, but rather to provide an interpretation, and the scope of the technical spirit of this disclosure is not limited to these embodiments. The scope of this disclosure should be interpreted by the appended claims, and all technical concepts within the scope of the appended claims should be interpreted as being included within the scope of this disclosure.

Claims

1. A method for identifying vehicles behind, comprising: Acquire images captured by a camera while driving a vehicle; Determine the time zone based on the acquired images; Perform object identification based on a defined time zone; The image coordinates of the vehicle behind are determined based on the result of the object recognition. The determined image coordinates are converted into distances from the vehicles behind. Determining the time zone includes: Obtain the output value of the light sensor; Correct the image; Calculate the average brightness of the image based on the corrected image; and The time zone is determined based on the output value of the light sensor and the average brightness of the image. The method further includes setting the learning parameters of the object recognition deep learning network based on a determined time zone, and The object recognition deep learning network includes: A basic network is configured to extract features from the image; and The object detection head is configured to learn from the dataset to classify each of the multiple anchor boxes as a candidate region with a fixed position, and to adjust the position and size of the ground truth bounding boxes in the classified anchor boxes that have an intersection-union ratio equal to or greater than a reference value.

2. The method according to claim 1, wherein: The time zone includes a night zone and a day zone, and The method further includes: In response to determining that the determined time zone is the daytime zone, the parameters of the object recognition deep learning network are set to predefined daytime parameters; and In response to determining that the determined time zone is the night zone, the parameters of the object recognition deep learning network are set to preset night parameters.

3. The method of claim 2, wherein, The nighttime parameters include a first nighttime parameter for identifying the headlight area and a second nighttime parameter for identifying the area of ​​vehicles behind.

4. The method of claim 3, further comprising, in response to determining that the time zone is the night zone, performing: Obtain a fixed-exposure image by setting the camera to a fixed exposure time; The headlight area is identified based on the fixed exposure image; Automatic exposure images are acquired by setting the camera to automatic exposure; and The area of ​​the vehicle behind is identified based on the automatically exposed image.

5. The method of claim 4, further comprising: Determine whether the headlight area is included in the area of ​​the rear vehicles; as well as In response to determining that the headlight area is included in the rear vehicle area, perform: Determine that the lower end of the rear vehicle area is the lower end of the rear vehicle; and The distance to the vehicle behind is calculated based on the determined lower end of the vehicle behind and the camera's calibration value.

6. The method of claim 4, further comprising: In response to the successful recognition of the headlight area and the failure to recognize the rear vehicle area, the maximum headlight installation height is calculated based on the position of the headlight area; The lower end of the vehicle behind is determined based on the lower end of the headlight area and the maximum headlight mounting height; and The distance to the vehicle behind is calculated based on the determined lower end of the vehicle behind and the camera's calibration value.

7. The method of claim 5, wherein, The calibration values ​​include: The focal length value represents the distance from the camera to the image plane; The vertical installation height value from the ground to the camera; and The vertical installation angle value of the camera.

8. The method of claim 1, wherein, The camera includes a surround-view camera.

9. The method of claim 1, further comprising: The speed of the vehicle behind is calculated based on the change in distance within a predefined time period; as well as A rear collision warning alarm is output based on the distance and speed of the vehicle behind.

10. A non-volatile computer-readable medium comprising instructions that, when executed by a processor, cause the processor to control a rear vehicle recognition system to perform: Acquire images captured by a camera while driving a vehicle; Determine the time zone based on the acquired images; Perform object identification based on a defined time zone; determining an image coordinate of the rear vehicle based on a result of performing the object recognition; as well as The determined image coordinates are converted into distances from the vehicles behind. in, To determine the time zone, the instructions, when executed by the processor, further cause the processor to control the rear vehicle recognition system to execute: Obtain the output value of the light sensor; Correct the image; Calculate the average brightness of the image based on the corrected image; and The time zone is determined based on the output value of the light sensor and the average brightness of the image. When executed by the processor, the instructions further instruct the processor to control the rear vehicle recognition system to set the learning parameters of the object recognition deep learning network based on a defined time zone. The object recognition deep learning network includes: A basic network is configured to extract features from the image; as well as The object detection head is configured to learn from the dataset to classify each of multiple anchor boxes as a candidate region with a fixed position, and to adjust the position and size of the ground truth bounding boxes in the classified anchor boxes that have an intersection-union ratio equal to or greater than a reference value.

11. A vehicle comprising: A camera is configured to capture rearward images while the vehicle is in motion; processor; as well as A computer-readable medium that communicates with the processor and stores instructions that, when executed by the processor, cause the processor to control the vehicle to perform the following: The time zone is determined based on the captured rear images; Identify objects based on a defined time zone; Determine the image coordinates of the vehicle behind based on the result of object recognition; and Convert the determined image coordinates into distance. In order to determine the time zone, when the instruction is executed by the processor, the processor further controls the vehicle to perform the following: Obtain the output value of the light sensor; Correct the image; Calculate the average brightness of the image based on the corrected image; and The time zone is determined based on the output value of the light sensor and the average brightness of the image. When executed by the processor, the instructions further cause the processor to control the vehicle to set the learning parameters of the object recognition deep learning network based on a defined time zone. The object recognition deep learning network includes: A basic network is configured to extract features from the image; and The object detection head is configured to learn from the dataset to classify each of multiple anchor boxes as a candidate region with a fixed position, and to adjust the position and size of the ground truth bounding boxes in the classified anchor boxes that have an intersection-union ratio equal to or greater than a reference value.

12. The vehicle of claim 11, further comprising: A data storage device is configured to store multiple learning parameters for the object recognition deep learning network. In order to identify the object, when the instruction is executed by the processor, the instruction further causes the processor to control the vehicle to select and set at least one of the plurality of learning parameters according to the determined time zone.

Images