Detection method and device of stop line, storage medium and electronic device

Abstract

The embodiment of the invention provides a stop line detection method and device, a storage medium and an electronic device, and the method comprises the steps: inputting a current vehicle-mounted front-view image into a stop line instance segmentation model, and obtaining a group of segmentation masks, a view field region of a vehicle-mounted front-view camera comprising a road stop line; determining a group of target stop lines based on the group of segmentation masks, and determining the distance between the current vehicle and each target stop line in the group of target stop lines to obtain a group of stop line distances; estimating the distance between the current vehicle and the target stop line based on the historical stop line distance and the vehicle speed of the current vehicle to obtain an estimated stop line distance between the current vehicle and the target stop line at the current moment; a target stop line distance is determined according to the set of stop line distances and the estimated stop line distance, and the distance difference between the target stop line distance and the estimated stop line distance is smaller than or equal to a preset distance difference value.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of intelligent driving perception algorithms, and more specifically, to a stop line detection method and device, a storage medium, and an electronic device. Background Art

[0002] In the field of intelligent driving, the stop line is an important road marking, and the detection and distance measurement of the stop line is an important part of intelligent driving perception. The stop line detection method in the related art is to use a pre-trained stop line detection model to obtain the stop line detection result, and obtain the distance between the vehicle and the stop line based on the inverse perspective transformation algorithm. However, the distance measurement method used in the above method is limited by the fact that the camera optical axis is parallel to the road surface, and does not take into account the continuity between the frames of the vehicle-mounted camera, which will lead to large distance fluctuations and inaccuracy.

[0003] It can be seen that the stop line detection method in the related art has the problem of low accuracy of the detection result. Summary of the invention

[0004] Embodiments of the present invention provide a stop line detection method and device, a storage medium and an electronic device, so as to at least solve the problem that the stop line detection method in the related art has a low accuracy of the detection result.

[0005] According to one embodiment of the present invention, a stop line detection method is provided, comprising: inputting a current vehicle-mounted front view image into a stop line instance segmentation model to obtain a set of segmentation masks, wherein the current vehicle-mounted front view image is an image captured by a vehicle-mounted front view camera of a current vehicle at a current moment, and the field of view of the vehicle-mounted front view camera includes a road stop line; based on the set of segmentation masks, determining a set of target stop lines, and determining the distance between the current vehicle and each target stop line in the set of target stop lines to obtain a set of stop line distances, wherein each target stop line is associated with at least one segmentation mask in the set of segmentation masks. The method comprises the steps of: estimating the distance between the current vehicle and the target stop line based on the historical stop line distances and the vehicle speed of the current vehicle to obtain an estimated stop line distance, wherein the historical stop line distances include the distances between the current vehicle and the target stop line at a set of historical moments, and the estimated stop line distance is the estimated distance between the current vehicle and the target stop line at the current moment; determining the target stop line distance based on the set of stop line distances and the estimated stop line distance, wherein the distance difference between the target stop line distance and the estimated stop line distance is less than or equal to a preset distance difference.

[0006] According to another embodiment of the present invention, a stop line detection device is provided, comprising: an input unit, configured to input a current vehicle-mounted front view image into a stop line instance segmentation model to obtain a set of segmentation masks, wherein the current vehicle-mounted front view image is an image captured by a vehicle-mounted front view camera of a current vehicle at a current moment, and the field of view of the vehicle-mounted front view camera includes a road stop line; a first determination unit, configured to determine a set of target stop lines based on the set of segmentation masks, and determine a distance between the current vehicle and each target stop line in the set of target stop lines to obtain a set of stop line distances, wherein each target stop line is associated with at least one segmentation mask pair in the set of segmentation masks. The invention relates to a method for determining a target stop line distance according to the set of stop line distances and the estimated stop line distance. The method comprises the following steps: an estimating unit for estimating the distance between the current vehicle and the target stop line based on the historical stop line distances and the vehicle speed of the current vehicle to obtain an estimated stop line distance, wherein the historical stop line distance includes the distance between the current vehicle and the target stop line at a set of historical moments, and the estimated stop line distance is the estimated distance between the current vehicle and the target stop line at the current moment; and a second determining unit for determining a target stop line distance according to the set of stop line distances and the estimated stop line distance, wherein the distance difference between the target stop line distance and the estimated stop line distance is less than or equal to a preset distance difference.

[0007] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0008] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0009] According to yet another embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0010] Through the present invention, after acquiring the vehicle-mounted front view image data, firstly, the current vehicle-mounted front view image is input based on the pre-trained stop line instance segmentation model to obtain the stop line segmentation result, and the stop line distance at the current moment is estimated by using the stop line distance at the historical moment, so as to improve the accuracy and stability of the distance between the vehicle and the stop line. Therefore, it is possible to solve the problem of low accuracy of the detection result of the stop line detection method in the related art, thereby achieving the effect of improving the accuracy of the stop line detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic flow chart of a stop line detection method according to an embodiment of the present invention;

[0012] Figure 2 is a schematic diagram of an original image of a vehicle-mounted camera according to an embodiment of the present invention;

[0013] Figure 3 is a schematic diagram of a stop line segmentation result after an example segmentation according to an embodiment of the present invention;

[0014] Figure 4 is a schematic diagram of a stop line extraction result according to an embodiment of the present invention;

[0015] Figure 5 is a flow chart of another stop line detection method according to an embodiment of the present invention;

[0016] Figure 6 4 is a structural block diagram of a stop line detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] Figure 1 FIG. 1 is a flow chart of a method for detecting a stop line according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0020] Step S102, inputting the current vehicle-mounted front view image into the stop line instance segmentation model to obtain a set of segmentation masks, wherein the current vehicle-mounted front view image is an image captured by the vehicle-mounted front view camera of the current vehicle at the current moment, and the field of view area of ​​the vehicle-mounted front view camera includes the road stop line;

[0021] A stop line detection method in an embodiment of the present invention can be applied to a scenario where a vehicle captured image is processed to confirm the position of a stop line and the distance between the vehicle and the stop line.

[0022] In the field of intelligent driving, stop lines are important road markings, and detecting and measuring the distance to stop lines is an important part of intelligent driving perception.

[0023] The stop line detection method in the related art is to obtain the stop line detection result by using a pre-trained stop line detection model, and obtain the distance between the vehicle and the stop line based on an inverse perspective transformation algorithm.

[0024] However, the distance measurement method used in the above method is limited by the fact that the optical axis of the camera is parallel to the road surface, and does not take into account the continuity between frames of the on-board camera, which will lead to large distance fluctuations and inaccuracies.

[0025] Here, the continuity between frames of the vehicle-mounted camera can refer to the smooth transition in time or space between each frame (i.e., a single image) in the video stream captured by the vehicle-mounted camera. This continuity is very important for observing and analyzing video content, which can assist in understanding the movement of the vehicle, environmental changes, and potential abnormal situations. Exemplarily, the vehicle-mounted camera can capture images at a certain frame rate (such as 30 frames per second). This means that the time interval between each frame is fixed, such as 1 / 30 second. Temporal continuity means that the frames are evenly distributed in time without skipping any time periods. And the image content between frames is consistent in space. For example, if the vehicle is moving, the vehicle position in the previous frame should have a predictable change with the position in the next frame. This spatial continuity helps to understand the vehicle's trajectory and speed.

[0026] In order to at least partially solve the above problem, in this embodiment, a stop line instance segmentation model is used to perform stop line segmentation on the input image, and the distances between a set of segmented stop lines and the current vehicle are compared with the current stop line distance estimated based on the historical stop line distances to obtain the distance between the actual road stop line and the current vehicle. This can effectively avoid jumps in the stop line distance and make the distance measurement smoother and more accurate.

[0027] Optionally, in this embodiment, the vehicle-mounted forward-looking camera is a camera device installed at the front of the car, mainly used to capture real-time images and videos in front of the vehicle.

[0028] Instance segmentation aims to distinguish different object instances in an image and generate a segmentation mask for each instance. Optionally, in this embodiment, the stop line instance segmentation model is a pre-trained instance segmentation model for segmenting stop lines from an input image, and the stop line instance segmentation model includes but is not limited to:

[0029] Mask R-CNN: This is a deep learning model developed by Facebook AI Research that combines object detection and image segmentation techniques to generate masks for each object and can recognize and localize multiple objects.

[0030] FCN (Fully Convolutional Network): This is a deep learning model for image segmentation that uses a fully convolutional network structure to predict pixel-level category labels. With modification, FCN can also be used for instance segmentation.

[0031] YOLACT: This is a single-stage instance segmentation model that leverages the idea of ​​anchor-based detection models to quickly generate high-quality segmentation masks.

[0032] SOLO: This is a single-stage instance segmentation model that leverages the idea of ​​anchor-free detection models to generate segmentation masks by predicting the center point and scale of each object.

[0033] PolarMask: This is a single-stage instance segmentation model that leverages the idea of ​​anchor-free detection models to generate segmentation masks by predicting the center point and angle of each object.

[0034] AdaptIS: This is a single-stage instance segmentation model that leverages the idea of ​​anchor-free detection models to generate segmentation masks by predicting the center point and angle of each object.

[0035] For the stop line instance segmentation model, Figure 2 The current vehicle front view image shown in the figure is input into the stop line instance segmentation model, and the model will output the following Figure 3 Multiple segmentation masks are shown, each mask corresponds to an object instance, that is, each segmentation mask corresponds to a stop line recognized by the model.

[0036] Here, the current vehicle-mounted front view image is an image captured by the vehicle-mounted front view camera of the current vehicle at the current time t, the field of view of the vehicle-mounted front view camera includes the actual road stop line, and each segmentation mask in a set of segmentation masks is used to mark a stop line section identified from the current vehicle-mounted front view image.

[0037] Step S104, based on the set of segmentation masks, determining a set of target stop lines, and determining the distance between the current vehicle and each target stop line in the set of target stop lines, to obtain a set of stop line distances, wherein each target stop line corresponds to at least one segmentation mask in the set of segmentation masks;

[0038] Taking into account that the stop line instance segmentation model may identify other lines in the current vehicle-mounted front view image as stop lines, here, based on a set of segmentation masks, a set of target stop lines is determined. The target stop lines may be other lines that are mistakenly identified as stop lines by the model, or due to occlusion by elements such as vehicles, the target stop lines may be a section of a complete road stop line. This is not limited in the present embodiment.

[0039] Optionally, each target stop line may correspond to at least one segmentation mask in a set of segmentation masks.

[0040] Optionally, in this embodiment, the distance between the current vehicle and each target stop line in a set of target stop lines may be determined based on a distance formula between a point and a straight line, which is not limited in this embodiment.

[0041] Step S106, based on the historical stop line distances and the vehicle speed of the current vehicle, the distance between the current vehicle and the target stop line is estimated to obtain an estimated stop line distance, wherein the historical stop line distance includes the distance between the current vehicle and the target stop line at a set of historical moments, and the estimated stop line distance is the estimated distance between the current vehicle and the target stop line at the current moment.

[0042] In order to avoid a sudden change in the outputted distance between the current vehicle and the stop line, in this embodiment, the distance between the current vehicle and the target stop line may be estimated based on the historical stop line distances and the vehicle speed of the current vehicle.

[0043] The historical stop line distance includes the distance between the current vehicle and the target stop line at a set of historical moments, and the estimated stop line distance is the estimated distance between the current vehicle and the target stop line at the current moment.

[0044] Step S108, determining a target stop line distance according to a set of stop line distances and an estimated stop line distance, wherein a distance difference between the target stop line distance and the estimated stop line distance is less than or equal to a preset distance difference.

[0045] When the distance difference between the stop line distance determined based on the current vehicle-mounted front view image and the estimated stop line distance is within a certain error range, the target stop line distance can be determined based on a set of stop line distances and the estimated stop line distances, and the corresponding target stop line can be output as the position of the actual road stop line.

[0046] Through the above steps, the current vehicle-mounted front view image is input into the stop line instance segmentation model to obtain a set of segmentation masks, wherein the current vehicle-mounted front view image is an image collected by the vehicle-mounted front view camera of the current vehicle at the current moment, and the field of view area of ​​the vehicle-mounted front view camera includes the road stop line; based on a set of segmentation masks, a set of target stop lines is determined, and the distance between the current vehicle and each target stop line in the set of target stop lines is determined to obtain a set of stop line distances, wherein each target stop line corresponds to at least one segmentation mask in the set of segmentation masks; based on the historical stop line distances and the vehicle speed of the current vehicle, the distance between the current vehicle and the target stop line is estimated to obtain an estimated stop line distance, wherein the historical stop line distance includes the distance between the current vehicle and the target stop line at a set of historical moments, and the estimated stop line distance is the estimated distance between the current vehicle and the target stop line at the current moment; according to a set of stop line distances and the estimated stop line distance, the target stop line distance is determined, wherein the distance difference between the target stop line distance and the estimated stop line distance is less than or equal to the preset distance difference, thereby solving the problem that the detection method of the stop line in the related art has a low accuracy of the detection result.

[0047] In an exemplary embodiment, based on a set of segmentation masks, a set of target stop lines is determined, and a distance between the current vehicle and each target stop line in the set of target stop lines is determined to obtain a set of stop line distances, including:

[0048] S11, extracting the minimum bounding rectangle for each segmentation mask to obtain the minimum bounding rectangle corresponding to each segmentation mask;

[0049] S12, determining a set of target stop lines based on left and right endpoints of a minimum circumscribed rectangle corresponding to each segmentation mask, wherein each target stop line is determined based on left and right endpoints of a minimum circumscribed rectangle corresponding to at least one segmentation mask in the set of segmentation masks;

[0050] S13, converting the image coordinates of the left and right end points of each target stop line into world coordinates to obtain the world coordinates of the left and right end points of each target stop line, and determining the straight line equation of each target stop line according to the world coordinates of the left and right end points of each target stop line;

[0051] S14, calculating the distance from the current vehicle to each target stop line according to the vehicle coordinates of the current vehicle and the straight line equation of each target stop line, to obtain a set of stop line distances.

[0052] Optionally, in this embodiment, the minimum bounding rectangle is used to describe the minimum boundary of a point set or object. It is done by finding a rectangle on a two-dimensional plane so that the point set or object is completely contained in the rectangle and the area of ​​the rectangle is minimized. The area of ​​the minimum bounding rectangle is the smallest among all possible rectangles that enclose the point set; each point of the point set or object is located inside or on the boundary of the minimum bounding rectangle.

[0053] Optionally, in this embodiment, the method for determining the minimum circumscribed rectangle may include but is not limited to:

[0054] 1. Check all possible rectangles and find the smallest one. This method works when the point set is small, but it is inefficient for a large number of point sets.

[0055] 2. Rotating Calipers: This is an algorithm for finding the minimum circumscribed rectangle in linear time, which is suitable for situations where the point set is large.

[0056] 3. Divide and conquer method: divide the point set into smaller subsets, then recursively find the minimum enclosing rectangle of each subset, and finally merge these rectangles.

[0057] For example, in this embodiment, the stop line detection is performed on the input image based on the pre-trained stop line instance segmentation model to obtain the stop line segmentation result. The connected region extraction is performed on the stop line segmentation result, so a complete stop line may be segmented into one connected region or multiple connected regions; the minimum bounding rectangle extraction is performed on all the extracted connected regions.

[0058] Based on the left and right endpoints of the minimum circumscribed rectangle corresponding to each segmentation mask, a set of target stop lines is determined. The left and right endpoints of the minimum circumscribed rectangle can be the two intersection points of the horizontal midline of the minimum circumscribed rectangle and the minimum circumscribed rectangle, or can be the two endpoints of other line segments used to determine a left or right line (for example, the target stop line). This is not limited in this embodiment.

[0059] Optionally, in this embodiment, considering that the stop line instance segmentation model may have misidentification and monocular ranging (a method of distance measurement using a single camera (for example, a vehicle-mounted forward-looking camera)) is easily affected by factors such as ground flatness and vehicle posture, a set of segmentation masks can be screened, filtered, and merged to improve the accuracy of the measurement results. Here, each target stop line is determined based on the left and right endpoints of the minimum circumscribed rectangle corresponding to at least one segmentation mask in a set of segmentation masks.

[0060] Generating a stop line expression in a world coordinate system, that is, converting the image coordinates of the left and right endpoints of each target stop line into world coordinates to obtain the world coordinates of the left and right endpoints of each target stop line, and determining the straight line equation of each target stop line according to the world coordinates of the left and right endpoints of each target stop line, may include the following steps:

[0061] 1) Get the image coordinates x of the left and right end points of each target stop line pl (x l ,y l ), x pr (x r ,y r );

[0062] 2) Based on the calibration parameters such as camera extrinsics, camera intrinsics, the transformation matrix from camera to ego vehicle IMU (Inertial Measurement Unit), and the world coordinates obtained by ego vehicle IMU, the image coordinates x pl 、x pr Convert to world coordinates Xpl, Ypr;

[0063] 3) According to the two points Xpl and Ypr in the world coordinate system, the straight line equation of the target stop line can be obtained as shown in formula (1), which is the target stop line expression:

[0064] ax+by+c=0 (1)

[0065] The distance between the vehicle and the stop line is calculated according to the distance formula from a point to a straight line. That is, according to the vehicle coordinates of the current vehicle and the straight line equation of each target stop line, the distance from the current vehicle to each target stop line is calculated to obtain a set of stop line distances. The calculation formula is as shown in formula (2):

[0066]

[0067] Where (X pose , Y pose ) is the world coordinate of the ego-vehicle IMU, and a, b, c are the coefficients of the stop line equation.

[0068] Through this embodiment, an instance segmentation model is used to perform stop line segmentation on the input image, and the connected area and minimum circumscribed rectangle are extracted from the segmented area (segmentation mask) to extract the stop line, which is more accurate and faster than traditional methods.

[0069] In an exemplary embodiment, the method further comprises:

[0070] S21, before determining a set of target stop lines based on the left and right end points of the minimum circumscribed rectangle corresponding to each segmentation mask, the method further includes: filtering out the segmentation masks that meet the filtering conditions in the set of segmentation masks to obtain a filtered set of segmentation masks, wherein the filtering conditions include: the area of ​​the corresponding minimum circumscribed matrix is ​​less than a preset area threshold, and the aspect ratio of the corresponding minimum circumscribed matrix is ​​less than a preset aspect ratio threshold;

[0071] S22, based on the left and right endpoints of the minimum circumscribed rectangle corresponding to each segmentation mask, determine a group of target stop lines, including: determining the connecting lines of the left and right endpoints of the minimum circumscribed rectangle corresponding to each segmentation mask as the initial stop lines corresponding to each segmentation mask, to obtain a group of initial stop lines; merging the initial stop lines that meet the merging conditions in the group of initial stop lines, and determining the remaining stop lines after the merging as a group of target stop lines, wherein the merging condition is: the weighted sum of the absolute value of the slope difference and the lateral distance is less than a preset threshold.

[0072] A segmentation mask of an instance output by the stop line instance segmentation model may correspond to a connected area, and the segmentation masks that meet the filtering conditions in a set of segmentation masks are filtered out to obtain a filtered set of segmentation masks. This may refer to filtering out the connected areas that meet the filtering conditions in a set of connected areas corresponding to a set of segmentation masks to obtain a filtered set of connected areas.

[0073] It should be noted that the aforementioned determination of a set of target stop lines based on the left and right end points of the minimum circumscribed rectangle corresponding to each segmentation mask may be performed based on each segmentation mask in the filtered set of segmentation masks.

[0074] For example, in this embodiment, connected areas that meet the following conditions are filtered and deleted:

[0075] 1) The minimum circumscribed rectangle area of ​​the connected region is less than the threshold r1;

[0076] 2) The ratio of the length to the width of the minimum circumscribed rectangle of the connected area is less than the threshold r2.

[0077] The extracted initial stop line segments are merged, and the stop lines that meet formula (3) are merged using the slope and the horizontal distance of the segments:

[0078] value=α×|k i -k j |+β×|x i -x j | (3)

[0079] where k i and k jRespectively represent the slopes of the i-th and j-th initial stop lines, x i Indicates the horizontal coordinate of the right endpoint of the i-th stop line, x j represents the horizontal coordinate of the left endpoint of the jth stop line, and α and β are normalization coefficients.

[0080] When value is less than the preset threshold t value , merge stop lines i and j, and update the left and right end points of the stop line at the same time.

[0081] Here, α and β may be preset fixed values, which are not limited in this embodiment.

[0082] In an exemplary embodiment, based on a set of segmentation masks, determining a set of target stop lines, and determining a distance between a current vehicle and each of the target stop lines in the set of target stop lines includes:

[0083] S31, input the historical stop line distance and the vehicle speed of the current vehicle into the target state transfer equation to obtain the estimated stop line distance output by the target state transfer equation, wherein the target state transfer equation is a state transfer equation used by the Kalman filter model with the distance between the current vehicle and the target stop line as the state quantity.

[0084] The state transition equation of the Kalman filter describes how the state of the system changes from one time step to the next. The state vector corresponding to the current state is the output of the equation, that is, it is obtained by combining the state of the previous time step, the control input, and taking into account the process noise.

[0085] The state vector corresponding to the state at the previous time step is one of the inputs to the equation, which combines the dynamic characteristics of the system and the control input, as well as the effects of process noise, to calculate the state at the current time step.

[0086] That is, the state transition equation establishes the relationship between the current state vector and the previous state vector, the control input, and the process noise, and is the basis for the Kalman filter to predict (or estimate) the next state.

[0087] For example, in this embodiment, the target state transfer equation is a state transfer equation used by a Kalman filter model that takes the distance between the current vehicle and the target stop line as a state quantity.

[0088] The historical stop line distance (eg, the stop line distance corresponding to the previous frame) and the vehicle speed of the current vehicle are input into the target state transfer equation to obtain the estimated stop line distance output by the target state transfer equation.

[0089] Through this embodiment, the kinematic equation is used to predict the stop line distance at the current frame time using the historical frame data, which can effectively avoid the jump of the stop line distance and make the distance measurement smoother and more accurate.

[0090] In an exemplary embodiment, the target state transfer equation is the sum of the distance between the current vehicle and the target stop line at time t-1, the product of the vehicle speed of the current vehicle at time t-1 and the specified time step, and the process noise, wherein t is a positive integer greater than or equal to 1;

[0091] The historical stop line distance and the vehicle speed of the current vehicle are input into the target state transfer equation to obtain the estimated stop line distance output by the target state transfer equation, including:

[0092] S41, inputting the distance between the current vehicle and the target stop line at the previous moment before the current moment and the vehicle speed of the current vehicle at the previous moment into the target state transfer equation to obtain an estimated stop line distance output by the target state transfer equation.

[0093] Taking the current moment as time t and the previous moment as time t-1 as an example, the kinematic equation is used to predict the stopping line distance at the current frame using the historical frame data. Given the vehicle speed v at time t-1, t-1 In the case of , assuming that the distance between the vehicle and the stop line is the state quantity d, the state transfer equation can be shown as formula (4):

[0094] d t|t-1 =d t-1|t-1 +v t-1 Δt+w t (4)

[0095] Among them, d t|t-1 To estimate the stop line distance, the time step Δt is specified as the time difference between the current time t and the previous time t-1, and wt is the process noise.

[0096] In an exemplary embodiment, determining a target stop line distance based on a set of stop line distances and an estimated stop line distance includes:

[0097] S51, determining an absolute difference between each stop line distance in a set of stop line distances and the estimated stop line distance, obtaining an absolute difference corresponding to each stop line distance, and determining a stop line distance with the smallest corresponding absolute difference as a candidate stop line distance;

[0098] S52, when the absolute difference corresponding to the candidate stop line distance is less than or equal to the preset distance difference, the estimated stop line distance is input as a state quantity into the observation model of the Kalman filter model to obtain the target stop line distance output by the observation model, wherein the observation model is used to linearly map the input state quantity through the observation matrix, and the sum of the linear mapping result and the measured sound is determined as the output observation quantity.

[0099] Since there is a certain error in detecting the stop line from the image, and the vehicle may shake, brake suddenly, etc. during driving, there will be a certain error in directly measuring the distance to the stop line. In order to reduce the distance measurement error, in this embodiment, the distance between the vehicle and the stop line is updated based on the Kalman filter method.

[0100] Kalman filter is an effective recursive filter, which is mainly used for state estimation of linear dynamic systems. In Kalman filter, the observation model describes the observation process of the system, the input value is the state quantity, and the output result is the observation vector. Kalman filter uses this observation model to update the estimate of the system state, taking into account the influence of observation noise.

[0101] For example, in this embodiment, z is calculated t Distance d from estimated stop line t|t-1 The absolute difference Δd is obtained, and the stop line with the smallest Δd value is retained. The minimum value is denoted as Δd min .

[0102] When Δd min When it is less than or equal to the preset distance difference, the stop line is output, such as Figure 4 shown.

[0103] At this time, the distance z between the output stop line and the vehicle is t As the observed quantity, the observation model is established as formula (5):

[0104] z t =Hd t|t-1 +v t (5)

[0105] Among them, v t To measure noise, H is a one-dimensional identity matrix.

[0106] Through this embodiment, the distance between the vehicle and the stop line is updated according to the Kalman filter model, which can effectively avoid the jump of the stop line distance and make the distance measurement smoother and more accurate.

[0107] In an exemplary embodiment, determining a target stop line distance based on a set of stop line distances and an estimated stop line distance further includes:

[0108] S61: When the absolute difference corresponding to the candidate stop line distance is greater than a preset distance difference, the estimated stop line distance is determined as the target stop line distance.

[0109] For example, in this embodiment, when the absolute difference Δd corresponding to the distance from the candidate stop line min When the value is greater than or equal to a certain threshold (for example, a preset distance difference), it is determined that there is no stop line output in the current frame. At this time, the estimated stop line distance dt|t-1 The value of is used as the distance between the vehicle and the stop line in the current frame.

[0110] As an optional exemplary embodiment, a road stop line detection and ranging method based on monocular vision is taken as an example to explain the stop line detection method of the present invention.

[0111] like Figure 5 As shown, a road stop line detection and ranging method based on monocular vision in this embodiment may include the following steps:

[0112] Step S501, acquiring image data of a vehicle-mounted front-view camera;

[0113] Step S502, obtaining a stop line segmentation area based on the instance segmentation model;

[0114] Step S503, stop line extraction;

[0115] Step S504, estimating the distance between the vehicle and the stop line based on Kalman filtering.

[0116] Through this embodiment, after acquiring the vehicle-mounted front view image data, firstly, the current vehicle-mounted front view image is input based on the pre-trained stop line instance segmentation model to obtain the stop line segmentation result; secondly, based on the stop line geometric features, the straight line equation of the stop line in the world coordinate system is extracted; finally, a stop line ranging model is established in the world coordinate system, and the Kalman filter algorithm is used to estimate the stop line distance, thereby improving the accuracy and stability of the distance between the vehicle and the stop line, and the stop line detection result and the distance between the vehicle and the stop line can be obtained, which is more accurate and stable than direct ranging. The results can be applied to the fields of vehicle positioning, mapping, and autonomous driving planning and control.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0118] In the present embodiment, a detection device for a stop line is also provided, and the device is used to implement the above-mentioned embodiment and the preferred embodiment, and the description thereof will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented with software, the implementation of hardware, or a combination of software and hardware is also possible and contemplated.

[0119] Figure 6 is a structural block diagram of a stop line detection device provided by an embodiment of the present invention, such as Figure 6 As shown, the device comprises:

[0120] An input unit 602 is used to input the current vehicle-mounted front view image into the stop line instance segmentation model to obtain a set of segmentation masks, wherein the current vehicle-mounted front view image is an image collected by the vehicle-mounted front view camera of the current vehicle at the current moment, and the field of view area of ​​the vehicle-mounted front view camera includes the road stop line;

[0121] A first determining unit 604 is used to determine a set of target stop lines based on a set of segmentation masks, and determine a distance between the current vehicle and each target stop line in the set of target stop lines to obtain a set of stop line distances, wherein each target stop line corresponds to at least one segmentation mask in the set of segmentation masks;

[0122] an estimating unit 606, configured to estimate the distance between the current vehicle and the target stop line based on the historical stop line distances and the vehicle speed of the current vehicle, to obtain an estimated stop line distance, wherein the historical stop line distance includes the distance between the current vehicle and the target stop line at a set of historical moments, and the estimated stop line distance is the estimated distance between the current vehicle and the target stop line at the current moment;

[0123] The second determining unit 608 is configured to determine a target stop line distance according to a set of stop line distances and an estimated stop line distance, wherein a distance difference between the target stop line distance and the estimated stop line distance is less than or equal to a preset distance difference.

[0124] It should be noted that the input unit 602 in this embodiment can be used to execute the above step S102, the first determination unit 604 in this embodiment can be used to execute the above step S104, the estimation unit 606 in this embodiment can be used to execute the above step S106, and the second determination unit 608 in this embodiment can be used to execute the above step S108.

[0125] According to the embodiment of the present invention, the current vehicle-mounted front view image is input into the stop line instance segmentation model to obtain a set of segmentation masks, wherein the current vehicle-mounted front view image is an image collected by the vehicle-mounted front view camera of the current vehicle at the current moment, and the field of view of the vehicle-mounted front view camera includes the road stop line; based on the set of segmentation masks, a set of target stop lines is determined, and the distance between the current vehicle and each target stop line in the set of target stop lines is determined to obtain a set of stop line distances, wherein each target stop line corresponds to at least one segmentation mask in the set of segmentation masks; based on the historical stop line distances and the vehicle speed of the current vehicle, the distance between the current vehicle and the target stop line is estimated to obtain an estimated stop line distance, wherein the historical stop line distance includes the distance between the current vehicle and the target stop line at a set of historical moments, and the estimated stop line distance is the estimated distance between the current vehicle and the target stop line at the current moment; according to the set of stop line distances and the estimated stop line distance, the target stop line distance is determined, wherein the distance difference between the target stop line distance and the estimated stop line distance is less than or equal to the preset distance difference, thereby solving the problem that the detection method of the stop line has low accuracy of the detection result.

[0126] In an exemplary embodiment, the first determining unit includes:

[0127] An extraction module is used to extract the minimum bounding rectangle of each segmentation mask to obtain the minimum bounding rectangle corresponding to each segmentation mask;

[0128] A first determination module is used to determine a set of target stop lines based on left and right endpoints of a minimum circumscribed rectangle corresponding to each segmentation mask, wherein each target stop line is determined based on left and right endpoints of a minimum circumscribed rectangle corresponding to at least one segmentation mask in the set of segmentation masks;

[0129] An execution module, used for converting the image coordinates of the left and right end points of each target stop line into world coordinates, obtaining the world coordinates of the left and right end points of each target stop line, and determining the straight line equation of each target stop line according to the world coordinates of the left and right end points of each target stop line;

[0130] The calculation module is used to calculate the distance from the current vehicle to each target stop line according to the vehicle coordinates of the current vehicle and the straight line equation of each target stop line, so as to obtain a set of stop line distances.

[0131] In an exemplary embodiment, the apparatus further includes: a filtering unit, configured to filter out the segmentation masks that meet the filtering conditions in the group of segmentation masks before determining a group of target stop lines based on the left and right end points of the minimum circumscribed rectangle corresponding to each segmentation mask, so as to obtain a filtered group of segmentation masks, wherein the filtering conditions include: the area of ​​the corresponding minimum circumscribed matrix is ​​less than a preset area threshold, and the aspect ratio of the corresponding minimum circumscribed matrix is ​​less than a preset aspect ratio threshold;

[0132] The first determination module includes: a determination submodule, which is used to determine the lines connecting the left and right end points of the minimum circumscribed rectangle corresponding to each segmentation mask as the initial stop lines corresponding to each segmentation mask, so as to obtain a group of initial stop lines; an execution submodule, which is used to merge the initial stop lines that meet the merging conditions in the group of initial stop lines, and determine the remaining stop lines after the merging as a group of target stop lines, wherein the merging condition is: the weighted sum of the absolute value of the slope difference and the lateral distance is less than a preset threshold.

[0133] In an exemplary embodiment, the first determining unit includes:

[0134] The first input module is used to input the historical stop line distance and the vehicle speed of the current vehicle into the target state transfer equation to obtain the estimated stop line distance output by the target state transfer equation, wherein the target state transfer equation is a state transfer equation used by the Kalman filter model with the distance between the current vehicle and the target stop line as the state quantity.

[0135] In an exemplary embodiment, the target state transfer equation is the sum of the distance between the current vehicle and the target stop line at time t-1, the product of the vehicle speed of the current vehicle at time t-1 and the specified time step, and the process noise, wherein t is a positive integer greater than or equal to 1;

[0136] The first input module comprises:

[0137] The input submodule is used to input the distance between the current vehicle and the target stop line at the previous moment before the current moment and the vehicle speed of the current vehicle at the previous moment into the target state transfer equation to obtain the estimated stop line distance output by the target state transfer equation.

[0138] In an exemplary embodiment, the second determining unit includes:

[0139] a second determination module, configured to determine an absolute difference between each stop line distance in a set of stop line distances and the estimated stop line distance, obtain an absolute difference corresponding to each stop line distance, and determine the stop line distance with the smallest corresponding absolute difference as a candidate stop line distance;

[0140] The second input module is used to input the estimated stop line distance as a state quantity into the observation model of the Kalman filter model when the absolute difference corresponding to the candidate stop line distance is less than or equal to the preset distance difference, so as to obtain the target stop line distance output by the observation model, wherein the observation model is used to linearly map the input state quantity through the observation matrix, and determine the sum of the linear mapping result and the measured sound as the output observation quantity.

[0141] In an exemplary embodiment, the second determining unit further includes:

[0142] The third determination module is configured to determine the estimated stop line distance as the target stop line distance when the absolute difference corresponding to the candidate stop line distance is greater than a preset distance difference.

[0143] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0144] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0145] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0146] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0147] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0148] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0149] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0150] An embodiment of the present invention further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0151] An embodiment of the present invention also provides a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above-mentioned method embodiments.

[0152] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0153] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting a stop line, characterized in that: include: Inputting the current vehicle-mounted front view image into the stop line instance segmentation model to obtain a set of segmentation masks, wherein the current vehicle-mounted front view image is an image captured by the vehicle-mounted front view camera of the current vehicle at the current moment, and the field of view area of ​​the vehicle-mounted front view camera includes the road stop line; Based on the set of segmentation masks, a set of target stop lines is determined, and a distance between the current vehicle and each target stop line in the set of target stop lines is determined to obtain a set of stop line distances, wherein each target stop line corresponds to at least one segmentation mask in the set of segmentation masks; estimating the distance between the current vehicle and the target stop line based on historical stop line distances and the vehicle speed of the current vehicle to obtain an estimated stop line distance, wherein the historical stop line distance includes the distance between the current vehicle and the target stop line at a set of historical moments, and the estimated stop line distance is the estimated distance between the current vehicle and the target stop line at the current moment; A target stop line distance is determined according to the set of stop line distances and the estimated stop line distance, wherein a distance difference between the target stop line distance and the estimated stop line distance is less than or equal to a preset distance difference value.

2. The method according to claim 1, characterized in that The step of determining a set of target stop lines based on the set of segmentation masks, and determining a distance between the current vehicle and each target stop line in the set of target stop lines to obtain a set of stop line distances includes: Extracting the minimum bounding rectangle of each segmentation mask to obtain the minimum bounding rectangle corresponding to each segmentation mask; Determining the set of target stop lines based on left and right endpoints of a minimum circumscribed rectangle corresponding to each segmentation mask, wherein each target stop line is determined based on left and right endpoints of a minimum circumscribed rectangle corresponding to at least one segmentation mask in the set of segmentation masks; Convert the image coordinates of the left and right end points of each target stop line into world coordinates to obtain the world coordinates of the left and right end points of each target stop line, and determine the straight line equation of each target stop line according to the world coordinates of the left and right end points of each target stop line; According to the vehicle coordinates of the current vehicle and the straight line equation of each target stop line, the distance from the current vehicle to each target stop line is calculated to obtain the set of stop line distances.

3. The method according to claim 2, characterized in that Before determining the group of target stop lines based on the left and right end points of the minimum circumscribed rectangle corresponding to each segmentation mask, the method further includes: filtering out the segmentation masks that meet the filtering conditions in the group of segmentation masks to obtain the filtered group of segmentation masks, wherein the filtering conditions include: the area of ​​the corresponding minimum circumscribed matrix is ​​less than a preset area threshold, and the aspect ratio of the corresponding minimum circumscribed matrix is ​​less than a preset aspect ratio threshold; The method of determining the group of target stop lines based on the left and right endpoints of the minimum circumscribed rectangle corresponding to each segmentation mask includes: determining the lines connecting the left and right endpoints of the minimum circumscribed rectangle corresponding to each segmentation mask as the initial stop lines corresponding to each segmentation mask, so as to obtain a group of initial stop lines; merging the initial stop lines in the group of initial stop lines that meet the merging condition, and determining the remaining stop lines after the merging as the group of target stop lines, wherein the merging condition is: the weighted sum of the absolute value of the slope difference and the lateral distance is less than a preset threshold.

4. The method according to claim 1, characterized in that: The step of determining a set of target stop lines based on the set of segmentation masks, and determining a distance between the current vehicle and each target stop line in the set of target stop lines, comprises: The historical stop line distance and the vehicle speed of the current vehicle are input into a target state transfer equation to obtain the estimated stop line distance output by the target state transfer equation, wherein the target state transfer equation is a state transfer equation used by a Kalman filter model that uses the distance between the current vehicle and the target stop line as a state quantity.

5. The method according to claim 4, characterized in that The target state transfer equation is the sum of the distance between the current vehicle and the target stop line at time t-1, the product of the vehicle speed of the current vehicle and the specified time step at time t-1, and the process noise, wherein t is a positive integer greater than or equal to 1; The step of inputting the historical stop line distance and the vehicle speed of the current vehicle into a target state transfer equation to obtain the estimated stop line distance output by the target state transfer equation includes: The distance between the current vehicle and the target stop line at the previous moment before the current moment and the vehicle speed of the current vehicle at the previous moment are input into the target state transfer equation to obtain the estimated stop line distance output by the target state transfer equation.

6. The method according to claim 4, characterized in that The step of determining a target stop line distance according to the set of stop line distances and the estimated stop line distance comprises: determining an absolute difference between each stop line distance in the set of stop line distances and the estimated stop line distance, obtaining an absolute difference corresponding to each stop line distance, and determining a stop line distance having a minimum corresponding absolute difference as a candidate stop line distance; When the absolute difference corresponding to the candidate stop line distance is less than or equal to the preset distance difference, the estimated stop line distance is input as a state quantity into the observation model of the Kalman filter model to obtain the target stop line distance output by the observation model, wherein the observation model is used to linearly map the input state quantity through an observation matrix, and the sum of the result of the linear mapping and the measured sound is determined as the output observation quantity.

7. The method according to claim 6, characterized in that The step of determining a target stop line distance according to the set of stop line distances and the estimated stop line distance further comprises: In a case where the absolute difference corresponding to the candidate stop line distance is greater than the preset distance difference value, the estimated stop line distance is determined as the target stop line distance.

8. A stop line detection device, characterized in that: include: An input unit, used to input a current vehicle-mounted front view image into a stop line instance segmentation model to obtain a set of segmentation masks, wherein the current vehicle-mounted front view image is an image captured by a vehicle-mounted front view camera of the current vehicle at a current moment, and the field of view of the vehicle-mounted front view camera includes the road stop line; A first determining unit is used to determine a set of target stop lines based on the set of segmentation masks, and determine a distance between the current vehicle and each target stop line in the set of target stop lines to obtain a set of stop line distances, wherein each target stop line corresponds to at least one segmentation mask in the set of segmentation masks; an estimating unit, configured to estimate the distance between the current vehicle and the target stop line based on historical stop line distances and a vehicle speed of the current vehicle, to obtain an estimated stop line distance, wherein the historical stop line distance includes the distance between the current vehicle and the target stop line at a set of historical moments, and the estimated stop line distance is the estimated distance between the current vehicle and the target stop line at the current moment; The second determining unit is used to determine a target stop line distance according to the set of stop line distances and the estimated stop line distance, wherein a distance difference between the target stop line distance and the estimated stop line distance is less than or equal to a preset distance difference.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images