Lane-changing method and system based on deep learning object detection

Abstract

The present invention provides a lane-changing method and system based on deep learning object detection. The method includes: obtaining a first image representing the environment behind the vehicle; inputting the first image into a vehicle detection model for model inference to obtain an inference result; when the inference result includes detecting other vehicles, counting the environment of other vehicles in each region of the first image; the environment of other vehicles includes the moving direction of other vehicles and the parts shown in the first image; determining the lane-changing state of the vehicle according to the environment of other vehicles in each region of the first image. This method determines the lane-changing state of the vehicle through the environment of other vehicles in each region of the first image, thereby judging whether the vehicle can change lanes normally, and improving the accuracy of vehicle detection and lane-changing reminder in adjacent lanes. This lane-changing method is applicable to determining the states of the vehicle and vehicles in adjacent lanes in a weakly calibrated scenario, reducing the difficulty of customer installation and debugging, and enhancing the user experience.

Description

Technical Field

[0001] The present invention belongs to the technical field of in-vehicle driving assistance, and particularly relates to a lane-changing method and system based on deep learning object detection. Background Art

[0002] Lane-changing of a vehicle is one of the driving operations most likely to cause traffic accidents. Therefore, before a vehicle changes lanes, the driver must observe the road conditions in advance and select an appropriate time to change lanes. Currently, most lane-changes still rely on the driver's experience.

[0003] With the rapid development of in-vehicle electronic devices, there is currently an in-vehicle assisted lane-changing system on the market that can determine whether there are vehicles in the adjacent lane during lane-changing, thereby reminding the driver whether lane-changing is possible. However, due to inaccurate segmentation of lane lines, the current in-vehicle assisted lane-changing system will have problems such as missed reporting and false reporting of vehicles in the adjacent lane, and inaccurate reminders. Summary of the Invention

[0004] Aiming at the defects in the prior art, the present invention provides a lane-changing method and system based on deep learning object detection, which improves the accuracy of vehicle detection and lane-changing reminder in the adjacent lane.

[0005] In a first aspect, a lane-changing method based on deep learning object detection includes:

[0006] Obtaining a first image representing the environment behind the vehicle;

[0007] Inputting the first image into a vehicle detection model for model inference to obtain an inference result;

[0008] When the inference result includes detecting other vehicles, counting the environment of other vehicles in each region of the first image; the environment of other vehicles includes the moving direction of other vehicles and the parts shown in the first image;

[0009] Determining the lane-changing state of the vehicle according to the environment of other vehicles in each region of the first image.

[0010] Preferably, the first image is obtained by a rear pull camera disposed at the rear of the vehicle.

[0011] Preferably, the calibration method of the rear pull camera includes:

[0012] Obtaining a second image captured by the rear pull camera in the calibration state;

[0013] Using a lane line detection algorithm to detect two lane lines adjacent to the vehicle in the second image and identify the vanishing points of the two lane lines;

[0014] Obtaining the height of the rear pull camera from the ground and the pixel focal length of the rear pull camera;

[0015] Calculate the true distance corresponding to any image coordinate in the second image based on the height, pixel focal length, and vanishing point.

[0016] Preferably, the calibration state includes parking the vehicle in the exact middle of the lane.

[0017] Preferably, the method for calculating the true distance d corresponding to any image coordinate in the second image includes:

[0018]

[0019] In the formula, θ is the pitch angle of the rear camera, F c is the pixel focal length, H c is the height, y b is the vertical component of the image coordinate in the second image, y h is the vertical component of the vanishing point in the second image.

[0020] Preferably, the training method of the vehicle detection model includes:

[0021] Collect the third image captured by the rear camera;

[0022] Annotate the third image, and the annotation content includes other vehicles in the adjacent lane;

[0023] Use the object detection model to train the annotated third image to obtain the vehicle detection model.

[0024] Preferably, determining the lane-changing state of the vehicle according to the other-vehicle environment in each area of the first image specifically includes:

[0025] When there are no other vehicles in the front half area of the first image, there are other vehicles in the rear half area, and the other vehicles are in the forward direction, the lane-changing state of the vehicle is the state of being overtaken;

[0026] When there are other vehicles in the front half area of the first image, and the displayed part is the rear of the other vehicle, the lane-changing state of the vehicle is the state of meeting;

[0027] When there are other vehicles in the front half area of the first image, and the displayed part is the front of the other vehicle, the lane-changing state of the vehicle is the state of overtaking;

[0028] When the lane-changing state of the vehicle is not the state of being overtaken, the state of meeting, and the state of overtaking, determine the lane-changing state of the vehicle as the safe state.

[0029] Preferably, the front half area and the rear half area are divided according to the true distance d corresponding to any image coordinate in the second image.

[0030] In a second aspect, a lane-changing system based on deep learning object detection includes:

[0031] Collection unit: configured to obtain a first image representing the environment behind the vehicle;

[0032] Model inference unit: configured to input the first image into a vehicle detection model for model inference to obtain an inference result;

[0033] Detection unit: configured to, when the inference result includes detection of another vehicle, count the environment of the other vehicle in each region of the first image; the environment of the other vehicle includes the moving direction of the other vehicle and the part shown in the first image; determine the lane-changing state of the vehicle according to the environment of the other vehicle in each region of the first image.

[0034] Preferably, the collection unit includes a rear-pulling camera disposed at the rear of the vehicle.

[0035] As can be seen from the above technical solutions, the lane-changing method and system based on deep learning object detection provided by the present invention determine the lane-changing state of the vehicle according to the environment of the other vehicle in each region of the first image, so as to determine whether the vehicle can change lanes normally, and improve the accuracy of vehicle detection and lane-changing reminder in adjacent lanes. The lane-changing method and system are applicable to determining the states of the vehicle and vehicles in adjacent lanes in a weakly calibrated scenario, reducing the difficulty of customer installation and debugging, and enhancing the user experience. Description of the Drawings

[0036] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0037] Figure 1 It is a flowchart of the lane-changing method provided for the embodiment.

[0038] Figure 2 It is a flowchart of the calibration method for the rear-pulling camera provided for the embodiment.

[0039] Figure 3 It is a schematic diagram of the image captured by the rear-pulling camera provided for the embodiment.

[0040] Figure 4 It is a flowchart of the vehicle detection model training method provided for the embodiment.

[0041] Figure 5 It is a flowchart of the lane-changing state determination method for the vehicle provided for the embodiment.

[0042] Figure 6 It is a block diagram of the lane-changing system provided for the embodiment. Detailed Embodiments

[0043] The embodiments of the technical solution of the present invention will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, so they are only examples and cannot be used to limit the protection scope of the present invention. It should be noted that unless otherwise specified, the technical terms or scientific terms used in this application should be the ordinary meanings understood by those skilled in the art to which the present invention belongs.

[0044] It should be understood that when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0045] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in this specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0046] As used in this specification and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" depending on the context.

[0047] Embodiment:

[0048] A lane-changing method based on deep learning object detection, see Figure 1 , including:

[0049] S1: Obtain a first image representing the environment behind the vehicle;

[0050] S2: Input the first image into a vehicle detection model for model inference to obtain an inference result;

[0051] S3: When the inference result includes detecting other vehicles, count the environment of other vehicles in each area of the first image; the environment of other vehicles includes the moving direction of other vehicles and the parts shown in the first image;

[0052] S4: Determine the lane-changing state of the vehicle according to the environment of other vehicles in each area of the first image.

[0053] In this embodiment, the first image can be obtained by shooting with a rear-pull camera set at the rear of the vehicle. For example, the rear-pull camera can be set above the license plate at the rear of the vehicle and close to the center of the vehicle body. In order to ensure that the shooting angle of the rear-pull camera is consistent during use, the installed rear-pull camera should try to ensure that there is no obvious rotation or deflection during use. After obtaining the first image, the lane changing method can scale the first image to a fixed size (for example, 320*320). Since different vehicles have different sizes and heights, in order to improve the applicability of the lane changing method, the rear-pull camera can be installed to ensure that its pitch angle is adjustable within plus or minus 10 degrees. In this way, the pitch angle of the rear-pull camera can be adjusted according to the specific situation of the vehicle, so that the rear-pull camera can capture an image with a wider field of view.

[0054] In this embodiment, the environment behind the vehicle includes whether there are other vehicles behind, whether there are other vehicles in the adjacent lane, the distance between the vehicle and other vehicles, etc. After acquiring the first image, the lane changing method inputs the first image into the vehicle detection model for model reasoning to obtain a reasoning result. Among them, the vehicle detection model can adopt an existing model, or it can train a model according to the situation of the vehicle, so that the trained model can improve the accuracy of the vehicle detection. The reasoning result may include whether there are other vehicles in the adjacent lane, the distance between the other vehicle and the vehicle, etc. If the reasoning result is that no other vehicle is detected in the adjacent lane, then the processing of the first image collected this time is abandoned, and the first image at the next moment is continuously collected, and the first image at the next moment is judged.

[0055] In this embodiment, when the inference result of the lane changing method includes detecting another vehicle, it indicates that there is another vehicle in the adjacent lane. At this time, it is necessary to determine the lane changing state of the vehicle to determine whether the lane change can be performed normally. For example, if the lane changing state of the vehicle is a safe state, it means that the vehicle can change lanes normally at this time. If the lane changing state of the vehicle is a state of being overtaken, returning or overtaking, it means that the vehicle cannot change lanes normally at this time and needs to wait for the lane changing state of the vehicle to change to a safe state before changing lanes normally. The lane changing state of the vehicle is determined according to the environment of other vehicles in each area of ​​the first image. The area in the first image can be determined according to the size and height of the vehicle, or according to the safe distance for lane change. The safe distance can be determined by the user or determined uniformly. For example, the area close to the vehicle in the image can be set as the front half area, and the area other than the front half area in the image can be set as the back half area. For example, the 1 / 2 area close to the vehicle in the image is set as the front half area, and the remaining 1 / 2 area is set as the back half area. For example, the lane change status of the vehicle is determined based on whether there is another vehicle in each area of ​​the first image, the moving direction of the other vehicle, and which part of the other vehicle is displayed.

[0056] A lane-changing method based on deep learning object detection determines the lane-changing state of the host vehicle through the surrounding vehicle environment in each region of the first image, thereby determining whether the host vehicle can change lanes normally, improving the accuracy of vehicle detection and lane-changing reminder in adjacent lanes. This lane-changing method is applicable to determining the state of the host vehicle and vehicles in adjacent lanes in a weakly calibrated scenario, reducing the difficulty of customer installation and debugging, and enhancing the user experience.

[0057] Further, in some embodiments, refer to Figure 2 , the calibration method of the rear pull camera includes:

[0058] S11: Obtain a second image captured by the rear pull camera in the calibrated state;

[0059] S12: Use a lane line detection algorithm to detect two lane lines adjacent to the host vehicle in the second image and identify the vanishing point of the two lane lines;

[0060] S13: Obtain the height of the rear pull camera from the ground and the pixel focal length of the rear pull camera;

[0061] S14: Calculate the real distance corresponding to any image coordinate in the second image according to the height, pixel focal length, and vanishing point.

[0062] In this embodiment, after the rear pull camera is installed, the rear pull camera needs to be calibrated. When calibrating the rear pull camera, first collect a second image captured by the rear pull camera in the calibrated state, then use a lane line detection algorithm to detect two lane lines adjacent to the host vehicle in the second image and identify the vanishing point of the two lane lines, refer to Figure 3 . When the host vehicle is parked on a straight lane, then the two lane lines adjacent to the host vehicle in the second image can intersect at a point, that is, the vanishing point is obtained. Finally, calculate the real distance corresponding to any image coordinate in the second image according to the height, pixel focal length, and vanishing point. Wherein the real distance can represent the distance in the real environment corresponding to each pixel point in the second image. Refer to Figure 3 , for example, the vertical component of pixel point A in the second image is 3 cm, indicating that the vertical distance between pixel point A and the edge L closest to the host vehicle in the second image is 3 cm. Find the area corresponding to pixel point A in the second image in the real environment and measure the distance S between this area and the host vehicle. Thus, the real distance corresponding to pixel point A can be obtained as the distance S.

[0063] Further, in some embodiments, the calibrated state includes parking the host vehicle in the middle of the lane, and the lane is a straight lane.

[0064] In this embodiment, the calibration state can be determined in the following way: The vehicle can be parked on the lane beside the road. The lane should preferably be a straight lane. The vehicle body should be straightened to ensure that the distances between the left and right sides of the vehicle body and the adjacent lane lines are approximately equal. At the same time, it is also necessary to ensure that the rear camera can clearly capture the adjacent lane lines on both sides of the vehicle body.

[0065] Further, in some embodiments, the calculation method of the true distance d corresponding to any image coordinate in the second image includes:

[0066]

[0067] In the formula, θ is the pitch angle of the rear camera, F c is the pixel focal length, H c is the height, y b is the vertical component of the image coordinate in the second image, y h is the vertical component of the vanishing point in the second image.

[0068] In this embodiment, the pitch angle θ of the rear camera can be calculated based on the height H c , the pixel focal length F c , and the vanishing point, or it can be read by a detection device connected to the rear camera. The true distance d reflects the true physical distance corresponding to the image coordinate in the second image.

[0069] Further, in some embodiments, referring to Figure 4 , the training method of the vehicle detection model includes:

[0070] S21: Collect the third image captured by the rear camera;

[0071] S22: Annotate the third image, and the annotation content includes other vehicles in the adjacent lanes;

[0072] S23: Use the object detection model to train the annotated third image to obtain the vehicle detection model.

[0073] In this embodiment, the lane-changing method can also train the vehicle detection model based on the deep learning object detection algorithm. During the training process, collect the third image captured by the rear camera, mark the other vehicles in the adjacent lanes of the vehicle in the third image, and finally use the object detection model yolov5 of the pytorch framework to train the annotated third image to obtain the vehicle detection model.

[0074] Further, in some embodiments, referring to Figure 5 , determining the lane-changing state of the vehicle according to the other-vehicle environment in each area of the first image specifically includes:

[0075] When there is no other vehicle in the front half area of the first image, there is an other vehicle in the rear half area, and the other vehicle is moving forward, the lane-changing state of the host vehicle is the overtaken state;

[0076] When there is an other vehicle in the front half area of the first image, and the displayed part is the rear of the other vehicle, the lane-changing state of the host vehicle is the meeting state;

[0077] When there is an other vehicle in the front half area of the first image, and the displayed part is the front of the other vehicle, the lane-changing state of the host vehicle is the overtaking state;

[0078] When the lane-changing state of the host vehicle is not the overtaken state, the meeting state, or the overtaking state, it is determined that the lane-changing state of the host vehicle is the safe state.

[0079] In this embodiment, when determining the lane-changing state of the host vehicle, the lane-changing method may determine the lane-changing state of the host vehicle only when multiple frames of images are detected to meet the same situation, improving the accuracy of state judgment. The lane-changing method has the following situations:

[0080] 1) If there is no other vehicle in the front half area, an other vehicle is detected in the rear half area in multiple frames of images, and the other vehicle is moving forward, then it is determined that the relationship between the host vehicle and the other vehicle is "the host vehicle is in the overtaken state and should change lanes carefully".

[0081] 2) If an other vehicle is detected in the front half area in multiple frames of images, and the image of the rear of the other vehicle is displayed, then it is determined that the relationship between the host vehicle and the other vehicle is the "meeting state".

[0082] 3) If an other vehicle is detected in the front half area in multiple frames of images, and the image of the front of the other vehicle is displayed, then it is determined that the relationship between the host vehicle and the other vehicle is "the host vehicle is in the overtaking state".

[0083] 4) When it does not belong to the above three situations, it is determined that the relationship between the host vehicle and the other vehicle is the "safe state".

[0084] Further, in some embodiments, the front half area and the rear half area are divided according to the real distance d corresponding to any image coordinate in the second image.

[0085] In this embodiment, the lane-changing method may divide the second image into a front half area and a rear half area according to the distance of the real physical distance. For example, the set of pixel points corresponding to the area representing the distance within 50 meters from the host vehicle is divided into the front half area, and the set of pixel points corresponding to the area representing the distance greater than 50 meters from the host vehicle is divided into the rear half area.

[0086] A lane-changing system based on deep learning object detection, see Figure 6 , including:

[0087] Acquisition unit 1: used to obtain the first image representing the environment behind the host vehicle;

[0088] Model inference unit 2: configured to input the first image into a vehicle detection model for model inference to obtain an inference result;

[0089] Detection unit 3: configured to, when the inference result includes detecting another vehicle, count the environment of the other vehicle in each area of the first image; the environment of the other vehicle includes the moving direction of the other vehicle and the part shown in the first image; determine the lane-changing state of the vehicle according to the environment of the other vehicle in each area of the first image.

[0090] Further, in some embodiments, the acquisition unit 1 includes a rear-pulling camera disposed at the rear of the vehicle.

[0091] For the system provided by the embodiments of the present invention, for the sake of brief description, for parts not mentioned in the embodiments, reference may be made to the corresponding content in the foregoing embodiments.

[0092] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered by the scope of the claims and the specification of the present invention.

Claims

1. A lane-changing method based on deep learning object detection, characterized in that, it includes: Obtain a first image representing the environment behind the vehicle; Input the first image into a vehicle detection model for model inference to obtain an inference result; When the inference result includes detecting other vehicles, count the environment of other vehicles in each area of the first image; the environment of other vehicles includes the moving direction of other vehicles and the parts shown in the first image; Determine the lane-changing state of the vehicle according to the environment of other vehicles in each area of the first image; The determining the lane-changing state of the vehicle according to the environment of other vehicles in each area of the first image specifically includes: When there are no other vehicles in the front half area of the first image, there are other vehicles in the rear half area, and the other vehicles are moving forward, the lane-changing state of the vehicle is the being-overtaken state; When there are other vehicles in the front half area of the first image, and the part shown is the rear of the other vehicle, the lane-changing state of the vehicle is the meeting-vehicle state; When there are other vehicles in the front half area of the first image, and the part shown is the front of the other vehicle, the lane-changing state of the vehicle is the overtaking state; When the lane-changing state of the vehicle is not the being-overtaken state, the meeting-vehicle state, and the overtaking state, determine the lane-changing state of the vehicle as the safe state; The front half area and the rear half area are divided according to the real distance d corresponding to any image coordinate in the second image; the second image is obtained by a rear-pulling camera under the calibration state.

2. The lane-changing method based on deep learning object detection according to claim 1, characterized in that, The first image is obtained by a rear-pulling camera arranged at the rear of the vehicle.

3. The lane-changing method based on deep learning object detection according to claim 2, characterized in that, The calibration method of the rear-pulling camera includes: Use a lane line detection algorithm to detect two lane lines adjacent to the vehicle in the second image and identify the vanishing points of the two lane lines; Obtain the height of the rear-pulling camera from the ground and the pixel focal length of the rear-pulling camera; Calculate the real distance corresponding to any image coordinate in the second image according to the height, pixel focal length and vanishing point.

4. The lane-changing method based on deep learning object detection according to claim 3, characterized in that, The calibration state includes parking the vehicle in the middle of the lane.

5. The lane-changing method based on deep learning object detection according to claim 3, characterized in that, The calculation method of the real distance d corresponding to any image coordinate in the second image includes: where θ is the pitch angle of the rear-pulling camera, F c is the pixel focal length, H c is the height, y b is the vertical component of the image coordinates in the second image, y h is the vertical component of the vanishing point in the second image.

6. The lane-changing method based on deep learning object detection according to claim 2, characterized in that, The training method of the vehicle detection model includes: Collect a third image captured by the rear-pulling camera; Annotate the third image, and the annotation content includes other vehicles in the adjacent lane; Use a target detection model to train the annotated third image to obtain the vehicle detection model.

7. A lane-changing system based on deep learning object detection, characterized in that, it includes: An acquisition unit: used to obtain a first image representing the environment behind the vehicle; Model inference unit: configured to input the first image into a vehicle detection model for model inference to obtain an inference result; Detection unit: configured to, when the inference result includes detection of other vehicles, count the environment of other vehicles in each region of the first image; the environment of other vehicles includes the moving direction of other vehicles and the parts shown in the first image; determine the lane-changing state of the host vehicle according to the environment of other vehicles in each region of the first image; The determining the lane-changing state of the host vehicle according to the environment of other vehicles in each region of the first image specifically includes: When there are no other vehicles in the front half region of the first image, there are other vehicles in the rear half region, and the other vehicles are moving forward, the lane-changing state of the host vehicle is the being-overtaken state; When there are other vehicles in the front half region of the first image and the part shown is the rear of the other vehicle, the lane-changing state of the host vehicle is the meeting-vehicle state; When there are other vehicles in the front half region of the first image and the part shown is the front of the other vehicle, the lane-changing state of the host vehicle is the overtaking state; When the lane-changing state of the host vehicle is not the being-overtaken state, the meeting-vehicle state, and the overtaking state, determine that the lane-changing state of the host vehicle is the safe state; The front half region and the rear half region are divided according to the real distance d corresponding to any image coordinate in the second image; the second image is obtained by a rear-pulling camera in a calibrated state.

8. The lane-changing system based on deep learning object detection according to claim 7, wherein, The acquisition unit includes a rear-pulling camera disposed at the rear of the host vehicle.

Images