Multi-source lane line fusion method, device, vehicle and storage medium

Abstract

The embodiment of the present invention discloses a method, device, unmanned vehicle and storage medium for fusing multi-source lane lines. The method includes: acquiring the detection lane lines corresponding to each visual sensor in the vehicle in real time, and multiple visual sensors are arranged in the vehicle; if a lane line update is detected for the detection lane line corresponding to the target visual sensor, the predicted lane line of the fused lane line at the historical detection moment at the current detection moment is calculated; according to the detection lane line of the target visual sensor at the current detection moment and the predicted lane line, the fused lane line at the current detection moment is updated. The technical solution of the embodiment of the present invention provides a new method for sequentially fusing non-homologous lane lines corresponding to multiple visual sensors, thereby improving the stability and accuracy of lane line detection.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, specifically to unmanned driving and artificial intelligence technology, and more particularly to a method, device, vehicle and storage medium for fusing multi-source lane lines. Background Art

[0002] With the advancement of technology, the installation volume of L2-level assisted driving systems defined by the American Society of Automotive Engineers (SAE) has grown rapidly in recent years. Most of the functions include lane center keeping and full-speed adaptive cruise control, which together achieve lateral and longitudinal control of the vehicle within its lane.

[0003] The lane keeping system mainly relies on lane lines for control. As the requirements for system reliability become higher and higher, systems with redundant architecture will become more and more popular. For single-camera systems, the camera is directly used to sense lane lines for lateral control. Currently, most lateral controls on the market rely on the same visual perception module to provide lane lines.

[0004] In the process of implementing the present invention, the inventors found that the prior art has the following defects: when only the only visual perception module in the vehicle is used to provide lane lines, the stability and accuracy of the detection cannot be guaranteed. In particular, when the only visual perception module temporarily fails, it will bring safety driving risks to the autonomous driving vehicle. Summary of the invention

[0005] Embodiments of the present invention provide a multi-source lane line fusion method, device, vehicle and storage medium to provide a new way to sequentially fuse non-homologous lane lines corresponding to multiple visual sensors, thereby improving the stability and accuracy of lane line detection.

[0006] In a first aspect, an embodiment of the present invention provides a method for fusing multi-source lane lines, which includes: acquiring in real time the detection lane lines corresponding to each visual sensor in the vehicle, wherein multiple visual sensors are arranged in the vehicle; if a lane line update is detected for the detection lane line corresponding to the target visual sensor, calculating the predicted lane line of the fused lane line at the historical detection moment at the current detection moment; updating the fused lane line at the current detection moment according to the detection lane line of the target visual sensor at the current detection moment and the predicted lane line.

[0007] Further, it is determined whether there is an unused target detection lane line associated with the current predicted lane line, including: obtaining a current detection lane line in turn from all unused detection lane lines; obtaining various lane line description parameters corresponding to the current predicted lane line; determining whether the parameter differences between various lane line description parameters of the current detection lane line and various lane line description parameters of the current predicted lane line are all less than or equal to a preset difference threshold; if so, determining that the current detection lane line is associated with the current predicted lane line; if not, returning to execute the operation of obtaining a current detection lane line in turn from all unused detection lane lines, until the processing of all unused detection lane lines is completed.

[0008] Furthermore, according to the Kalman filter algorithm and the target detected lane line, the fused lane line of the currently predicted lane line at the current detection moment is updated, including: calculating the Kalman gain matrix of the currently predicted lane line according to the covariance matrix prediction value of the currently predicted lane line at the current detection moment, the preset observation matrix and the measurement noise covariance matrix; updating the fused lane line of the currently predicted lane line at the current detection moment according to the current predicted lane line, the Kalman gain matrix, the observation matrix and the target detected lane line; calculating the final value of the covariance matrix of the currently updated fused lane line at the current detection moment according to the covariance matrix prediction value of the currently predicted lane line at the current detection moment, the Kalman gain matrix and the observation matrix; wherein the final value of the covariance matrix is ​​used to predict the covariance matrix prediction value of the currently updated fused lane line at the next detection moment.

[0009] Furthermore, according to the Kalman filter algorithm and the current predicted lane line, the fused lane line of the current predicted lane line at the current detection moment is updated, including: calculating the Kalman gain matrix of the current predicted lane line according to the covariance matrix prediction value of the current predicted lane line at the current detection moment, the preset observation matrix and the measurement noise covariance matrix; updating the fused lane line of the current predicted lane line at the current detection moment according to the current predicted lane line, the Kalman gain matrix and the observation matrix; calculating the final value of the covariance matrix of the currently updated fused lane line at the current detection moment according to the covariance matrix prediction value of the current predicted lane line at the current detection moment, the Kalman gain matrix and the observation matrix; wherein the final value of the covariance matrix is ​​used to predict the covariance matrix prediction value of the currently updated fused lane line at the next detection moment.

[0010] Furthermore, after updating the fused lane line at the current detection moment based on the detected lane line of the target visual sensor at the current detection moment and the predicted lane line, it also includes: screening and obtaining the target lane line based on the fused lane line at the current detection moment as a reference for driving decisions during the vehicle's automatic driving process.

[0011] Furthermore, based on the fused lane line at the current detection moment, the target lane line is screened, including: increasing the credibility weight of the fused lane line updated using the detection lane line; reducing the credibility weight of the fused lane line updated without using the detection lane line; and screening out the fused lane line whose credibility weight is greater than or equal to the set weight threshold as the target lane line.

[0012] Furthermore, after reducing the credibility weight of the fused lane line updated by the unused detection lane line, the method further includes: deleting the fused lane line whose credibility weight is less than or equal to the set weight threshold.

[0013] In the second aspect, an embodiment of the present invention further provides a multi-source lane line fusion device, which includes: a detection lane line acquisition module, which is used to acquire in real time the detection lane lines corresponding to each visual sensor in the vehicle, and multiple visual sensors are arranged in the vehicle; a prediction lane line acquisition module, which is used to calculate the predicted lane line of the fused lane line at the historical detection moment at the current detection moment if a lane line update occurs when the detection lane line corresponding to the target visual sensor is detected; a fusion lane line update module, which is used to update the fused lane line at the current detection moment according to the detection lane line of the target visual sensor at the current detection moment and the predicted lane line.

[0014] In the third aspect, an embodiment of the present invention further provides a vehicle, comprising: a plurality of visual sensors for acquiring environmental feature information around the vehicle in real time; one or more controllers; a storage device for storing one or more programs; when the one or more programs are executed by the one or more controllers, the one or more controllers implement the multi-source lane line fusion method as described in any embodiment of the present invention.

[0015] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the multi-source lane line fusion method as described in any embodiment of the present invention.

[0016] The technical solution provided by the embodiment of the present invention obtains the detected lane lines corresponding to each visual sensor in the vehicle in real time; if a lane line update is detected for the detected lane line corresponding to the target visual sensor, the predicted lane line of the fused lane line at the historical detection moment at the current detection moment is calculated; and the fused lane line at the current detection moment is updated according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line. This technical solution provides a new method for sequentially fusing non-homologous lane lines corresponding to multiple visual sensors. Compared with the implementation method of using a single visual perception module in the vehicle to provide lane lines in the prior art, this solution improves the stability and accuracy of lane line detection. Moreover, when a visual sensor fails, the lane lines corresponding to other visual sensors can also be output stably and effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1a A flowchart of a method for fusing multi-source lane lines provided in Embodiment 1 of the present invention;

[0018] Figure 1b A schematic diagram of a vehicle provided with two visual sensors in a multi-source lane line fusion method provided in Embodiment 1 of the present invention;

[0019] Figure 1c A schematic diagram of a sequential lane line fusion method in a multi-source lane line fusion method provided in Embodiment 1 of the present invention;

[0020] Figure 2a is a flow chart of another multi-source lane line fusion method provided in Embodiment 2 of the present invention;

[0021] Figure 2b A schematic diagram of a lane line coordinate system and expression in a multi-source lane line fusion method provided in Embodiment 2 of the present invention;

[0022] Figure 2c This is a specific implementation flow chart applicable to the second embodiment of the present invention for matching each lane line description parameter between the current detected lane line and the current predicted lane line one by one;

[0023] Figure 3a is a flow chart of another multi-source lane line fusion method provided in Embodiment 3 of the present invention;

[0024] Figure 3b It is a flowchart of a specific application scenario applicable to the embodiment of the present invention;

[0025] Figure 4 is a structural diagram of a multi-source lane line fusion device provided in a fourth embodiment of the present invention;

[0026] Figure 5 It is a structural schematic diagram of a vehicle provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0028] It should also be noted that, for ease of description, only the part relevant to the present invention but not all content is shown in the accompanying drawings. It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0029] Embodiment 1

[0030] Figure 1a A flowchart of a multi-source lane line fusion method provided in Embodiment 1 of the present invention. This embodiment is applicable to a situation where a vehicle equipped with multiple visual sensors performs lane line fusion on detected lane lines corresponding to each visual sensor. The method of this embodiment can be executed by a multi-source lane line fusion device, which can be implemented by software and / or hardware, and can generally be integrated in the vehicle's vehicle terminal, typically, in the vehicle terminal of a vehicle with an autonomous driving function.

[0031] Accordingly, the method specifically comprises the following steps:

[0032] S110, acquiring in real time the detection lane lines corresponding to each visual sensor in the vehicle, wherein a plurality of visual sensors are arranged in the vehicle.

[0033] Among them, the visual sensor can have the ability to capture thousands of pixels of light from an entire image. Its main component is a camera or video camera. After capturing the image, the visual sensor compares it with the reference image stored in the memory for analysis. Visual sensors are usually the best choice due to their accuracy, ease of use, rich functions and reasonable cost. Among them, lane lines are solid or dotted lines that separate different driving lanes.

[0034] It is understandable that a vehicle is provided with multiple visual sensors, and different visual sensors correspond to respective detection lane lines. The multiple visual sensors provided in the vehicle may be two or more visual sensors installed in the vehicle. This embodiment does not limit the number and type of visual sensors (for example, two-dimensional visual sensors and three-dimensional visual sensors). Optionally, when the vehicle is provided with two visual sensors, the schematic diagram may be as follows: Figure 1b shown.

[0035] In this embodiment, the visual sensor may be a sensor with an independent image recognition and processing function, which may directly identify the detection lane lines within the detection area based on the collected environmental feature information (environmental image), and then the vehicle computer may directly obtain the detection lane lines collected by each visual sensor output; or, the visual sensor may only have the function of collecting environmental feature information, and then, after the vehicle computer obtains the collected environmental feature information output by each visual sensor, it is necessary to identify it based on the environmental feature information to obtain the detection lane lines corresponding to each visual sensor.

[0036] Optionally, the method of acquiring the detection lane lines corresponding to each visual sensor in the vehicle in real time may be: controlling each visual sensor to acquire environmental feature information of the surrounding environment at a preset detection frequency, and acquiring the detection lane lines of each visual sensor at each detection moment matching the detection frequency by calculation by the vehicle terminal or by independent calculation by the visual sensor. The detection frequency of each visual sensor may be the same or different, and this embodiment does not limit this.

[0037] At the same time, it should be noted that each visual sensor may or may not detect a lane line at each detection moment. Accordingly, the detected lane lines corresponding to each visual sensor in the vehicle obtained in real time may be empty or may be specific lane line equations.

[0038] S120: If a lane line update is detected for the detection lane line corresponding to the target visual sensor, a predicted lane line of the fused lane line at the historical detection time at the current detection time is calculated.

[0039] As mentioned above, each visual sensor in the vehicle corresponds to a matching detection lane line, and further, the target visual sensor may be a visual sensor whose corresponding detection lane line changes.

[0040] In this embodiment, since multiple visual sensors are used for image acquisition, a detected lane line corresponding to each visual sensor will be obtained for each actual lane line. Therefore, it is necessary to fuse the detected lane lines corresponding to each visual sensor to obtain the final fused lane line.

[0041] This embodiment adopts a sequential lane line fusion method, in which Figure 1c A schematic diagram of a sequential lane line fusion method in a multi-source lane line fusion method is shown in FIG. Figure 1c Taking the two visual sensors in the vehicle (camera A and camera B) as an example, whenever the detection lane line corresponding to a camera is updated (the circle position of the row where camera A or the camera is located), it means that the fused lane line in the vehicle needs to be maintained and updated. Therefore, it is necessary to use the detection lane line corresponding to the camera at the current detection moment to update the fused lane line and obtain a new fused lane line (the circle position of the row where the fused lane line is located).

[0042] It should be noted that the technical solution of the embodiment of the present invention adopts a sequential lane line fusion architecture, which has good scalability. That is to say, when one or more new visual sensors are added to the vehicle, the fusion framework does not need to be improved and can continue to be used normally. At the same time, when one or more visual sensors fail, it will not affect the normal visual sensor result output at all, thereby ensuring the real-time update status of the fused lane line.

[0043] As shown above, the sequential lane fusion method is triggered whenever a lane line update occurs when a detection lane line corresponding to a visual sensor is detected. Therefore, the above state needs to be detected.

[0044] Accordingly, the lane line detected by the target visual sensor is updated as follows:

[0045] Each visual sensor provided in the vehicle is triggered to obtain the environmental feature information around the vehicle in real time; based on the environmental feature information, the detection lane lines corresponding to each visual sensor at different detection times are obtained; if it is determined that the detection lane line of the target visual sensor among the visual sensors at the current detection time is inconsistent with the detection lane line at the previous historical detection time, it is determined that the detection lane line corresponding to the target visual sensor is detected and a lane line update occurs.

[0046] Specifically, the environmental feature information may be image information corresponding to the surrounding environment collected by each visual sensor. The inconsistency between the lane line detected by the target visual sensor at the current detection moment and the lane line detected at the previous historical detection moment means that the lane line detection result at the current detection moment is not completely consistent with the lane line detection result at the previous historical detection moment, which may include a variety of situations, such as: the number of lane lines detected at the current detection moment does not match the number of lane lines detected at the previous historical detection moment, the lane line equation of the lane line detected at the current detection moment does not match the lane line equation of the lane line detected at the previous historical detection moment, etc.

[0047] Furthermore, when it is determined that a lane line update has occurred in the detection lane line corresponding to the target visual sensor, it indicates that the fused lane line in the vehicle needs to be updated using the detection lane line corresponding to the target visual sensor. At this time, considering that the most recent update time of the fused lane line is associated with the historical detection time when the lane line update occurred in the last time, it is not associated with the current detection time (i.e., the time point of the current detection lane line update). Based on this, it is necessary to obtain the predicted lane line of the fused lane line corresponding to the historical detection time at the current detection time, and then based on the predicted lane line and the detection lane line, accurately and effectively update the predicted lane line of the fused lane line.

[0048] Optionally, a method for calculating the predicted lane line of the fused lane line at the historical detection moment at the current detection moment may be: using a Kalman filter algorithm to predict and calculate the fused lane line at the historical detection moment to obtain the predicted lane line of the fused lane line at the current detection moment.

[0049] Alternatively, the method for calculating the predicted lane line of the fused lane line at the historical detection moment at the current detection moment can also be: obtain a time period determined by the historical detection moment and the current detection moment, obtain the vehicle's operating status information (for example, speed and yaw angular velocity, etc.) during the time period, and predict the predicted lane line of the fused lane line at the historical detection moment at the current detection moment based on the above operating status information.

[0050] It is understandable that the number of fused lane lines at the historical detection moment may be 0, 1, or more.

[0051] S130. Update the fused lane line at the current detection moment according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line.

[0052] In this embodiment, after calculating the predicted lane line of the fused lane line at the historical detection moment at the current detection moment, the fused lane line at the current detection moment can be updated based on the above predicted lane line and the detected lane line of the target visual sensor at the current detection moment.

[0053] In an optional implementation of this embodiment, the fused lane line at the current detection moment may be updated according to the association result between the detected lane line and the predicted lane line.

[0054] It is understandable that the number of lane lines detected by the target visual sensor at the current detection moment can be 0, 1 or more, and the number of fused lane lines at the historical detection moment can also be 0, 1 or more. Therefore, it is necessary to first establish the association result between the detected lane line and the predicted lane line, that is, establish an association relationship pair between the detected lane line and the predicted lane line pointing to the same actual lane line, and use the detected lane line in the same association relationship pair to update the fused lane line, and obtain the fused lane line of the fused lane line in the association relationship pair at the current detection moment.

[0055] In an optional implementation of the present embodiment, Kalman filtering may also be used to fuse the detected lane line and the predicted lane line to obtain the fused lane line at the current detection moment, or other data fusion methods, such as weighted summation, etc., may be used to fuse the detected lane line and the predicted lane line to obtain the fused lane line at the current detection moment. The present embodiment does not limit this.

[0056] The technical solution provided by the embodiment of the present invention obtains the detected lane lines corresponding to each visual sensor in the vehicle in real time; if a lane line update is detected for the detected lane line corresponding to the target visual sensor, the predicted lane line of the fused lane line at the historical detection moment at the current detection moment is calculated; and the fused lane line at the current detection moment is updated according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line. This technical solution provides a new method for sequentially fusing non-homologous lane lines corresponding to multiple visual sensors. Compared with the implementation method of using a single visual perception module in the vehicle to provide lane lines in the prior art, this solution improves the stability and accuracy of lane line detection. Moreover, when a visual sensor fails, the lane lines corresponding to other visual sensors can also be output stably and effectively.

[0057] Embodiment 2

[0058] Figure 2aA flowchart of another multi-source lane line fusion method provided in Embodiment 2 of the present invention. This embodiment is optimized based on the above embodiments. In this embodiment, a Kalman filter is used to calculate the predicted lane line of the fused lane line at the historical detection moment at the current detection moment, and the fused lane line at the current detection moment is updated.

[0059] Accordingly, the method specifically comprises the following steps:

[0060] S210, acquiring in real time the detection lane lines corresponding to each visual sensor in the vehicle, wherein a plurality of visual sensors are arranged in the vehicle.

[0061] S220, determine whether a lane line update occurs when a detection lane line corresponding to the target visual sensor is detected: if so, execute S230; otherwise, return to execute S210.

[0062] S230. Predict the predicted lane line of the fused lane line at the current detection time according to the vehicle state information at the current detection time, the time interval between the current detection time and the historical detection time, and the fused lane line at the historical detection time.

[0063] Specifically, the vehicle state information may include: the vehicle speed and the vehicle yaw rate. The vehicle state information may be detected by a motion detection sensor configured in the vehicle.

[0064] S240, calculating a Kalman state matrix of the predicted lane line according to the vehicle state information at the current detection time and the time interval between the current detection time and the historical detection time.

[0065] S250, calculating a predicted value of the covariance matrix of the predicted lane line at the current detection time based on the Kalman state matrix of the predicted lane line and the final value of the covariance matrix of the fused lane line at the historical detection time.

[0066] In the above S230-S250, the predicted lane line of each fused lane line at the current detection moment and the covariance matrix predicted value of the predicted lane line at the current detection moment are calculated by using the Kalman filter algorithm.

[0067] Among them, the covariance matrix prediction value is used to update the fused lane line at the current detection moment. That is, the above covariance matrix prediction value is used in the subsequent S280, specifically, in S2801-S2803. By using the covariance matrix prediction value of the predicted lane line at the current detection moment, the final value of the covariance matrix of the currently updated fused lane line at the current detection moment can be calculated in the execution of S2801-S2803 operations, and the final value of the covariance matrix of the fused lane line at the current detection moment will be used as the final value of the covariance matrix of the fused lane line at the next detection moment at the historical detection moment, so as to re-complete the operations of S230-S250 at the new detection moment.

[0068] In this embodiment, taking a fused lane line A at a historical detection moment as an example, the specific process of calculating the predicted lane line of the fused lane line A at the current detection moment and the covariance matrix predicted value of the predicted lane line at the current detection moment is described in detail.

[0069] First, the prediction formula in the Kalman filter algorithm is shown in formulas (1) and (2);

[0070]

[0071]

[0072] in, is the predicted value of the state variable, is the final result of the state variable at the previous moment, F is the Kalman state matrix, which can be extracted from the state equation, and u is the external input, where it can be assumed that there is no external input. is the predicted value of the covariance matrix at the current detection moment, is the final value of the covariance matrix at the historical detection moment, and Q is the system noise covariance matrix.

[0073] In this embodiment, state variables are used to describe the fused lane line A at the historical detection moment and the predicted lane line of the fused lane line A at the current detection moment.

[0074] Among them, through Specifically describe the fusion lane line A at the historical detection time, and through Describes the predicted lane line of the fused lane line A at the current detection time.

[0075] In order to construct the state variable X corresponding to the lane line, it is necessary to first describe the lane line expression. Figure 2b A schematic diagram of the lane line coordinate system and expression in a multi-source lane line fusion method is shown in FIG. Figure 2bAs shown, the lane line coordinate system can be defined as follows: the center of the front bumper of the vehicle is the coordinate origin, the forward direction is the positive direction of the x-axis, and the left direction is the positive direction of the y-axis. Based on this, the lane line equation can be defined in the form of formula (3).

[0076] y=a+b*x+c*x*x+d*x*x*x (3)

[0077] Where x is a point on the lane line (for example, Figure 2b The lateral distance between the point M in the middle and the front of the vehicle (for example, Figure 2b x in M ), y is the longitudinal distance between a point on the lane line and the center of the vehicle's front bumper (e.g. Figure 2b The y in M ).

[0078] Where a, b, c, d are coefficients expressing the lane line equation. Based on the above form, each state variable used in each embodiment of the present invention can be defined as X = [a, b, c, d].

[0079] In this embodiment, the predicted lane line of the fused lane line A at the current detection time is not directly calculated using formula (1), but is calculated by using the vehicle speed v and yaw rate ω, as well as the fused lane line at the historical detection time. Use the following formula (4)-formula (7) to predict the predicted lane line of the fused lane line at the current detection time: in, ΔT is the time interval between the current detection time and the historical detection time.

[0080] a k =a k-1 +b k *v*ΔT+c k-1 *(v*ΔT) 2 +d k-1 *(v·ΔT) 3 (4)

[0081] b k =b k-1 +2*c k-1 *(v*ΔT)+3*d k-1 *(v·ΔT) 2 +ω·ΔT (5)

[0082] c k =c k-1 +3*d k-1 *(v*ΔT) (6)

[0083] dk =d k-1 (7)

[0084] At the same time, based on the above formulas (4) to (7), the Kalman state matrix F shown in formula (8) can be constructed, that is:

[0085]

[0086] According to F. And Q, the covariance matrix prediction value at the current detection time can be calculated based on formula (2) Among them, Q is the experience matrix obtained based on actual experience, and Practical uses, and The method of obtaining it will be described in detail later.

[0087] Therefore, based on the Kalman state matrix of the predicted lane line and the final value of the covariance matrix of the fused lane line at the historical detection moment, the covariance matrix prediction value can be used to update the fused lane line at the current detection moment.

[0088] It should be noted again that the number of fused lane lines at the historical detection time can be one or more, so the above S230-S250 generates matching predicted lane lines for all fused lane lines. For the sake of convenience, the subsequent flowchart only describes the update process using one predicted lane line as an example. In fact, the same update operation needs to be performed for each predicted lane line.

[0089] S260: Obtain the current predicted lane line.

[0090] In this embodiment, taking a predicted lane line at the current detection moment (that is, the current predicted lane line) as an example, a specific implementation method of updating the fused lane line of the current predicted lane line at the current detection moment is introduced. In fact, for each current predicted lane line at the current detection moment, S270-S290 can be repeatedly executed to obtain a matching fused lane line at the current detection moment.

[0091] S270. Determine whether there is an unused target detection lane line associated with the current predicted lane line: if so, execute S280; otherwise, execute S290.

[0092] Among them, after determining that the detection lane line corresponding to the target visual sensor has been updated, the detection lane line of the target visual sensor at the current detection time obtained can be one or more. Therefore, for each current predicted lane line, it is necessary to traverse each detection lane line, and select a detection lane line (that is, the target detection lane line) that meets the matching condition of the current predicted lane line, update the predicted lane line, and obtain the fused lane line.

[0093] In this embodiment, the target detection lane line refers to a detection lane line that is associated with the current predicted lane line and has not been used. Each detection lane line is only used to update one predicted lane line. In order to ensure that each detection lane line is only used once, it is necessary to mark the used status of each detection lane line after it is used.

[0094] Correspondingly, judging whether there is an unused target detection lane line associated with the current predicted lane line means detecting whether there is a detection lane line that points to the same actual lane line (associated) as the current predicted lane line among all the detection lane lines that are not marked as used.

[0095] In an optional implementation of this embodiment, a method for determining whether there is an unused target detection lane line associated with the current predicted lane line may be:

[0096] Among all the unused detection lane lines, obtain a current detection lane line in turn; obtain various lane line description parameters corresponding to the current predicted lane line; determine whether the parameter differences between various lane line description parameters of the current detection lane line and various lane line description parameters of the current predicted lane line are all less than or equal to a preset difference threshold; if so, determine that the current detection lane line is associated with the current predicted lane line; if not, return to execute the operation of obtaining a current detection lane line in turn among all the unused detection lane lines until the processing of all the unused detection lane lines is completed.

[0097] In this embodiment, obtaining an unused target detection lane line associated with the current predicted lane line refers to obtaining a target detection lane line that points to the same actual lane line as the current predicted lane line. Therefore, the difference between the current predicted lane line and the target detection lane line should be very small, or in other words, the lane line equations of the two should be very close. Therefore, various lane line description parameters between the current detection lane line and the current predicted lane line can be obtained respectively for comparison one by one, and based on the comparison results, it is determined whether there is an unused target detection lane line associated with the current predicted lane line.

[0098] In an optional implementation of this embodiment, the lane line description parameters may include: various coefficients in the lane line description equation, that is, one or more of a, b, c and d. At the same time, the lane line description parameters may also include: a certain calibration distance D in front of the vehicle cal , the longitudinal distance between the lane line and the center of the vehicle's front bumper.

[0099] Among them, Figure 2c A specific implementation flow chart of matching each lane line description parameter between the current detected lane line and the current predicted lane line one by one applicable to the second embodiment of the present invention is shown in FIG.

[0100] like Figure 2c As shown, the specific association step may include: when determining whether the currently predicted lane line A and a detected lane line B can be matched and associated, first determine the coefficient a in the lane line equation of the currently predicted lane line A. k and the coefficient a in the lane line equation for detecting lane line B j Is the absolute value of the difference greater than the preset first calibration parameter a? th If it is satisfied, it is directly determined that the association is unsuccessful, otherwise it continues to determine the coefficient b in the lane line equation of the current predicted lane line A. k and the coefficient b in the lane line equation for detecting lane line B j Is the absolute value of the difference greater than the preset second calibration parameter b? th If it is satisfied, it is directly determined that the association is unsuccessful, otherwise it continues to determine the coefficient c in the lane line equation of the current predicted lane line A. k and the coefficient c in the lane line equation for detecting lane line B j Is the absolute value of the difference greater than the preset third calibration parameter c? th If it is satisfied, it is directly determined that the association is unsuccessful, otherwise it continues to determine the coefficient d in the lane line equation of the current predicted lane line A. k and the coefficient d in the lane line equation for detecting lane line B j Is the absolute value of the difference greater than the preset fourth calibration parameter d th If it is satisfied, it is directly determined that the association is unsuccessful, otherwise it proceeds to calculate the front calibration distance D cal The lateral distance deviation D between the two lane lines Hd , as shown in formula (9), then judge D Hd Is the absolute value of greater than the preset fifth calibration parameter D th If it is satisfied, it is directly determined that the association is unsuccessful, otherwise it is determined that the current predicted lane line A and the detected lane line B are successfully associated.

[0101]

[0102] It should be noted that the first calibration parameter a th It is used to define the maximum tolerance threshold value of the difference between the two coefficients a in the lane line equation of the predicted lane line and the detected lane line (see Formula 3), and the second calibration parameter b th The third calibration parameter c is used to define the maximum tolerance threshold value of the difference between the two coefficients b in the lane line equation of the predicted lane line and the detected lane line. th The fourth calibration parameter d is used to define the maximum tolerance threshold value of the difference between the two coefficients c in the lane line equation of the predicted lane line and the detected lane line. th The fifth calibration parameter is used to limit the maximum tolerance threshold value of the difference between the two coefficients d in the lane line equation of the predicted lane line and the detected lane line. The fifth calibration parameter is used to limit the lateral distance deviation D between the predicted lane line and the detected lane line. Hd The maximum tolerance threshold.

[0103] In practical applications, those skilled in the art may preset the above five calibration parameters, and this embodiment does not limit this.

[0104] The advantage of this setting is that by updating the fused lane line at the current detection moment according to the correlation result between the detected lane line and the predicted lane line, the fused lane line obtained in this way is more accurate, which further determines the accuracy and precision of the detection.

[0105] S280, according to the Kalman filter algorithm and the target detection lane line, update the fused lane line of the current predicted lane line at the current detection moment, and mark the target detection lane line as being in use.

[0106] In an optional implementation of this embodiment, according to the Kalman filter algorithm and the target detected lane line, the fused lane line of the current predicted lane line at the current detection time is updated according to the current predicted lane line and can be:

[0107] S2801. Calculate the Kalman gain matrix of the current predicted lane line using the covariance matrix prediction value at the current detection moment, the preset observation matrix and the measurement noise covariance matrix.

[0108] Specifically, the Kalman gain matrix K of the current predicted lane line can be calculated according to formula (10): K ;

[0109]

[0110] in, is the predicted value of the covariance matrix at the current detection moment, which is calculated by the aforementioned formula (2); H is the preset observation matrix, which can be determined by experiments or empirical constants; R is the measurement noise covariance matrix, which can be determined by experiments or empirical constants.

[0111] S2802: Update the fused lane line of the currently predicted lane line at the current detection moment according to the currently predicted lane line, the Kalman gain matrix, the observation matrix and the target detected lane line.

[0112] Specifically, the fused lane line of the current predicted lane line at the current detection time can be calculated according to formula (11) and formula (12):

[0113]

[0114] z k =Hx k (12)

[0115] in, Used to describe the current predicted lane line, x k Used to describe the target detection lane line, where: a k1 ,b k1 ,c k1 ,d k1 The coefficients of the lane line equation for the target detection lane line.

[0116] S2803. Calculate the final value of the covariance matrix of the currently updated fused lane line at the current detection time based on the predicted value of the covariance matrix of the currently predicted lane line at the current detection time, the Kalman gain matrix and the observation matrix.

[0117] Specifically, the final value of the covariance matrix of the currently updated fused lane line at the current detection time can be calculated according to formula (13):

[0118]

[0119] Among them, the final value of the covariance matrix It is used to predict the covariance matrix prediction value of the currently updated fused lane line at the next detection time. That is, in S250, the covariance matrix prediction value of the current detection time is calculated using formula (2) at the current detection time. When used That is, it is calculated according to formula (13) at the last historical detection moment. In this embodiment, after the current predicted lane line is updated to obtain the fused lane line at the current detection moment according to the Kalman filter algorithm and the target detection lane line, it is necessary to mark the target detection lane line as being in use to prevent other predicted lane lines from using the target detection lane line again when performing the association calculation in S270.

[0120] S290: Update the fused lane line of the currently predicted lane line at the current detection moment according to the Kalman filter algorithm and the currently predicted lane line.

[0121] In an optional implementation of this embodiment, according to the Kalman filter algorithm and the current predicted lane line, the method of updating the fused lane line of the current predicted lane line at the current detection time may be:

[0122] The Kalman gain matrix of the currently predicted lane line is calculated based on the covariance matrix prediction value of the currently predicted lane line at the current detection moment, the preset observation matrix and the measurement noise covariance matrix;

[0123] According to the current predicted lane line, the Kalman gain matrix and the observation matrix, a fused lane line of the current predicted lane line at the current detection moment is updated;

[0124] According to the predicted value of the covariance matrix of the current predicted lane line at the current detection time, the Kalman gain matrix and the observation matrix, the final value of the covariance matrix of the currently updated fused lane line at the current detection time is calculated;

[0125] The final value of the covariance matrix is ​​used to predict the predicted value of the covariance matrix of the currently updated fused lane line at the next detection moment.

[0126] When there is no unused target detection lane line associated with the current predicted lane line, the method of updating the fused lane line of the current predicted lane line at the current detection time is similar to S280, the only difference is that z is calculated using formula (12) k When x is substituted k Used to describe the current predicted lane line replace.

[0127] The technical solution of the embodiment of the present invention can make the obtained fused lane line more accurate by updating the fused lane line at the current detection moment according to the detected lane line and predicted lane line of the target visual sensor at the current detection moment, thereby improving the accuracy of the system and making it more stable and practical. Among them, the Kalman filter algorithm does not simply select the lane line, but tracks it through the Kalman filter algorithm. Therefore, the Kalman filter algorithm takes into account the vehicle motion parameters and tracks the detectable lane line, further improving reliability.

[0128] Based on the above embodiments, the method may further include: after determining that the update of the fused lane line at the current detection moment is completed for all predicted lane lines, determining whether there is an unused detection lane line;

[0129] If so, the detected lane line is used as a new fused lane line at the current detection moment.

[0130] The reason for this setting is that if it is determined that there is an unused detection lane line, it means that the detection lane line is the lane line that has just been detected recently and has not been detected at other historical detection moments. Therefore, the detection lane line needs to be initialized and stored as a fusion lane line. Then, when other visual sensors detect the lane line again, they can be subsequently fused with the initialized and stored fusion lane line. Then, this new fusion lane line can be tracked to further improve detection efficiency and accuracy.

[0131] It is understandable that after the detected lane line is used as a new fused lane line at the current detection moment, the final value of the initialized covariance matrix can be set for the new fused lane line for Kalman filter calculation at the next detection moment.

[0132] Embodiment 3

[0133] Figure 3a A flowchart of another multi-source lane line fusion method provided for Embodiment 3 of the present invention. This embodiment is optimized based on the above embodiments. In this embodiment, after updating the fused lane line at the current detection moment according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line, it further includes: screening the target lane line according to the fused lane line at the current detection moment as a reference for driving decision-making during the vehicle's automatic driving process.

[0134] Accordingly, the method specifically comprises the following steps:

[0135] S310, acquiring in real time the detection lane lines corresponding to the respective visual sensors in the vehicle, wherein a plurality of visual sensors are arranged in the vehicle.

[0136] S320: If a lane line update is detected for the detection lane line corresponding to the target visual sensor, a predicted lane line of the fused lane line at the historical detection time at the current detection time is calculated.

[0137] S330: Update the fused lane line at the current detection moment according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line.

[0138] S340: Filter and obtain a target lane line according to the fused lane line at the current detection moment, so as to serve as a reference for driving decision during the automatic driving of the vehicle.

[0139] The target lane line may be one or more fused lane lines updated at the current detection time, and the lane line with the highest reliability selected by comparison. Furthermore, the target lane line may be output to the vehicle decision unit for reference in the vehicle decision unit's driving decision.

[0140] For example, the vehicle decision unit can determine whether to move the vehicle to the left or right by a certain distance based on the lateral distance between the target lane line and the vehicle to control the vehicle to travel in the middle position of the lane.

[0141] In an optional implementation of this embodiment, screening and obtaining the target lane line according to the fused lane line at the current detection moment may include:

[0142] The credibility weight of the fused lane line updated by the detected lane line is increased;

[0143] The credibility weight of the fused lane line updated by the unused detection lane line is reduced;

[0144] The fused lane lines whose credibility weight is greater than or equal to the set weight threshold are selected as the target lane lines.

[0145] It can be understood that if a fused lane line is updated using the detection lane line at the current detection moment, it means that the fused lane line is "seen" by the visual sensor at the current detection moment, which further indicates that at the current detection moment, the probability of the fused lane line appearing around the vehicle will increase, and thus the credibility weight of the fused lane line needs to be increased (for example, +1 processing).

[0146] Similarly, if a fused lane line is not updated using the detection lane line at the current detection moment, it means that the fused lane line is not "seen" by the visual sensor at the current detection moment, which means that the probability of the fused lane line appearing around the vehicle at the current detection moment is reduced, and the credibility weight of the fused lane line needs to be reduced (for example, -1 processing).

[0147] Furthermore, by screening out the fused lane lines whose credibility weight is greater than or equal to the set weight threshold, the target lane line with the highest credibility at the current detection moment can be obtained for reference in making driving decisions. Based on the above embodiments, after reducing the credibility weight of the fused lane line updated by the unused detection lane line, the following may also be included:

[0148] Delete the fused lane lines whose credibility weight is less than or equal to the set weight threshold.

[0149] The technical solution of the embodiment of the present invention needs to process the credibility weight of the fused lane line according to the fused lane line at the current detection moment. When the detected lane line is used to update the fused lane line, it means that the credibility of the detected lane line is high, so the credibility weight is increased; when the detected lane line is not used to update the fused lane line, it means that the credibility of the detected lane line is low and there are other better options, so the credibility weight is reduced. Furthermore, when the credibility weight is not greater than the set weight threshold, it means that the lane line has not been detected by the visual sensor for a long time, so it is deleted to reduce the workload of subsequent unnecessary associated calculations and reduce the calculation and storage pressure on the vehicle side.

[0150] At the same time, by giving priority to lane lines with higher credibility weights, lane lines with lower credibility weights can be excluded, and lane lines below the credibility weight threshold can be deleted. Therefore, the lane line detection stability and detection accuracy can be improved, and the target lane line can also be selected quickly, saving calculation time.

[0151] In this embodiment, a flowchart of a specific application scenario applicable to the embodiment of the present invention is as follows: Figure 3bAs shown in FIG. 1 , the specific process includes: at the current detection time, whether it is determined that the lane line data corresponding to camera A is updated based on the environmental feature information collected by camera A, or when it is determined that the lane line data corresponding to camera B is updated based on the environmental feature information collected by camera B, the prediction process of the fused lane line at the current detection time will be triggered. In the prediction process, it is necessary to refer to the vehicle speed v and the vehicle's yaw rate ω to use the extrapolation equation to extrapolate the fused lane line to the visual sensor measurement time to achieve time synchronization between the fusion system and the visual sensor. The extrapolation equation is the prediction formula in the Kalman filter algorithm, see formulas (1) and (2), to achieve the prediction of state variables and covariance matrices. After predicting the predicted lane lines of the fused lane line at the current detection time, by judging whether the current predicted lane line k exists, all predicted lane lines can be traversed. If there are still predicted lane lines that have not been traversed, the current predicted lane line k is extracted.

[0152] After extracting the current predicted lane line k, by judging whether the detection lane line j corresponding to camera A or camera B exists, each detection lane line can be traversed for the current predicted lane line k. If the detection lane line exists, the detection lane line j is extracted, and it is judged whether the detection lane line j has been used and updated. If it has been used, the next detection lane line is traversed. If it has not been used and updated, it is necessary to judge whether the current predicted lane line k and the detection lane line j can be associated. If it is determined that the association is unsuccessful, the next detection lane line j is traversed. If it is determined that the association is successful, the detection lane line j is used to update the fused lane line corresponding to the current predicted lane line k, and the target detection lane line j is recorded as being in use, and then the next current predicted lane line k is traversed. If it has been determined that there is no detection lane line j for matching, it is recorded that the current predicted lane line k has not been updated, and the fused lane line of the current predicted lane line at the current detection time is updated according to the current predicted lane line k, and then the next current predicted lane line k is traversed. When all predicted lane lines are judged, it is necessary to determine whether there is an unused detection lane line j. If so, the detection lane line needs to be used as a new fused lane line at the current detection moment, and the credibility weight corresponding to the new fused lane line needs to be initialized; if not, the credibility weight of each fused lane line needs to be increased or decreased. Among them, the credibility of the fused lane line updated using the detection lane line is increased; the credibility of the fused lane line updated without the detection lane line is reduced. Finally, the fused lane lines whose credibility weights are less than or equal to the set weight threshold need to be deleted, and the high-credibility lane lines are output as a reference for driving decisions.

[0153] Embodiment 4

[0154] Figure 4 : is a structural diagram of a multi-source lane line fusion device provided in the fourth embodiment of the present invention. The multi-source lane line fusion device provided in this embodiment can be implemented by software and / or hardware, and can generally be integrated in the vehicle's vehicle terminal. Typically, the vehicle terminal of a vehicle with an automatic driving function can implement a multi-source lane line fusion method in the embodiment of the present invention. Figure 4 As shown, the device may specifically include: a detected lane line acquisition module 410, a predicted lane line acquisition module 420 and a fused lane line update module 430.

[0155] The detection lane line acquisition module 410 is used to acquire the detection lane lines corresponding to each visual sensor in the vehicle in real time, and multiple visual sensors are provided in the vehicle;

[0156] The predicted lane line acquisition module 420 is used to calculate the predicted lane line of the fused lane line at the historical detection time at the current detection time if a lane line update is detected for the detection lane line corresponding to the target visual sensor;

[0157] The fused lane line updating module 430 is used to update the fused lane line at the current detection moment according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line.

[0158] The technical solution provided by the embodiment of the present invention obtains the detected lane lines corresponding to each visual sensor in the vehicle in real time; if a lane line update is detected for the detected lane line corresponding to the target visual sensor, the predicted lane line of the fused lane line at the historical detection moment at the current detection moment is calculated; and the fused lane line at the current detection moment is updated according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line. This technical solution provides a new method for sequentially fusing non-homologous lane lines corresponding to multiple visual sensors. Compared with the implementation method of using a single visual perception module in the vehicle to provide lane lines in the prior art, this solution improves the stability and accuracy of lane line detection. Moreover, when a visual sensor fails, the lane lines corresponding to other visual sensors can also be output stably and effectively.

[0159] Based on the above embodiments, the predicted lane line acquisition module 420 can be specifically used to: trigger each visual sensor set in the vehicle to obtain the environmental feature information around the vehicle in real time; according to the environmental feature information, obtain the detection lane lines corresponding to each visual sensor at different detection times; if it is determined that the detection lane line of the target visual sensor among the visual sensors at the current detection time is inconsistent with the detection lane line at the previous historical detection time, it is determined that the detection lane line corresponding to the target visual sensor has been detected and a lane line update has occurred.

[0160] On the basis of the above embodiments, the predicted lane line acquisition module 420 can be specifically used to: use the Kalman filter algorithm to predict and calculate the fused lane line at the historical detection moment, and obtain the predicted lane line of the fused lane line at the current detection moment.

[0161] On the basis of the above embodiments, the predicted lane line acquisition module 420 adopts the Kalman filter algorithm, which may specifically include: predicting the predicted lane line of the fused lane line at the current detection time according to the vehicle state information at the current detection time, the time interval between the current detection time and the historical detection time, and the fused lane line at the historical detection time; calculating the Kalman state matrix of the predicted lane line according to the vehicle state information at the current detection time and the time interval between the current detection time and the historical detection time; calculating the predicted value of the covariance matrix of the predicted lane line at the current detection time according to the Kalman state matrix of the predicted lane line and the final value of the covariance matrix of the fused lane line at the historical detection time;

[0162] The covariance matrix prediction value is used to update the fused lane line at the current detection moment.

[0163] On the basis of the above embodiments, the fused lane line updating module 430 may specifically include: a fused lane line updating unit, configured to update the fused lane line at the current detection moment according to the association result between the detected lane line and the predicted lane line.

[0164] On the basis of the above embodiments, the fused lane line update unit may specifically include: a target detection lane line recognition subunit, which is used to obtain the current predicted lane line and determine whether there is an unused target detection lane line associated with the current predicted lane line; a first fused lane line update subunit, which is used to update the fused lane line of the current predicted lane line at the current detection moment according to the Kalman filter algorithm and the target detection lane line, and mark the target detection lane line as being in use; a second fused lane line update subunit, which is used to update the fused lane line of the current predicted lane line at the current detection moment according to the Kalman filter algorithm and the current predicted lane line if not.

[0165] On the basis of the above embodiments, it can also include a fused lane line updating unit, which is used to: after determining that the update of the fused lane line at the current detection moment is completed for all predicted lane lines, determine whether there is an unused detection lane line; if so, use the detection lane line as a new fused lane line at the current detection moment.

[0166] On the basis of the above embodiments, the target detection lane line identification subunit can be specifically used to: obtain a current detection lane line in turn from all unused detection lane lines; obtain various lane line description parameters corresponding to the current predicted lane line; determine whether the parameter differences between the various lane line description parameters of the current detection lane line and the various lane line description parameters of the current predicted lane line are all less than or equal to a preset difference threshold; if so, determine that the current detection lane line is associated with the current predicted lane line; if not, return to execute the operation of obtaining a current detection lane line in turn from all unused detection lane lines until the processing of all unused detection lane lines is completed.

[0167] Based on the above embodiments, the first fused lane line updating subunit may be specifically used for:

[0168] According to the predicted value of the covariance matrix of the current predicted lane line at the current detection moment, the preset observation matrix and the measurement noise covariance matrix, the Kalman gain matrix of the current predicted lane line is calculated; according to the current predicted lane line, the Kalman gain matrix, the observation matrix and the target detected lane line, the fused lane line of the current predicted lane line at the current detection moment is updated; according to the predicted value of the covariance matrix of the current predicted lane line at the current detection moment, the Kalman gain matrix and the observation matrix, the final value of the covariance matrix of the currently updated fused lane line at the current detection moment is calculated; wherein, the final value of the covariance matrix is ​​used to predict the predicted value of the covariance matrix of the currently updated fused lane line at the next detection moment.

[0169] Based on the above embodiments, the first fused lane line updating subunit may be specifically used for:

[0170] According to the predicted value of the covariance matrix of the current predicted lane line at the current detection moment, the preset observation matrix and the measurement noise covariance matrix, the Kalman gain matrix of the current predicted lane line is calculated; according to the current predicted lane line, the Kalman gain matrix and the observation matrix, the fused lane line of the current predicted lane line at the current detection moment is updated; according to the predicted value of the covariance matrix of the current predicted lane line at the current detection moment, the Kalman gain matrix and the observation matrix, the final value of the covariance matrix of the currently updated fused lane line at the current detection moment is calculated; wherein, the final value of the covariance matrix is ​​used to predict the predicted value of the covariance matrix of the currently updated fused lane line at the next detection moment.

[0171] On the basis of the above embodiments, it may further specifically include: a target lane line screening unit, which is used to screen and obtain the target lane line according to the fused lane line at the current detection moment after updating the fused lane line at the current detection moment according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line, so as to serve as a reference for driving decisions during the vehicle's automatic driving process.

[0172] On the basis of the above embodiments, the target lane line screening unit may specifically include: a weight increasing subunit, which is used to increase the credibility weight of the fused lane line updated using the detection lane line; a weight reducing subunit, which is used to reduce the credibility weight of the fused lane line updated without using the detection lane line; and a target lane line screening subunit, which is used to screen out the fused lane line whose credibility weight is greater than or equal to the set weight threshold as the target lane line.

[0173] On the basis of the above embodiments, it may also include: a fused lane line deletion subunit, which is used to delete the fused lane lines whose credibility weight is less than or equal to the set weight threshold after reducing the credibility weight of the fused lane lines updated by the unused detection lane lines.

[0174] The above-mentioned multi-source lane line fusion device can execute the multi-source lane line fusion method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0175] Embodiment 5

[0176] Figure 5 Schematic diagram of the structure of a vehicle provided by Embodiment 5 of the present invention. Figure 5 As shown, the device includes a plurality of visual sensors 510, a controller 520, a storage device 530, an input device 540 and an output device 550; the number of the plurality of visual sensors 510 and the controller 520 in the vehicle can be one or more, Figure 5In the figure, multiple visual sensors 510 and controller 520 are taken as an example; multiple visual sensors 510, controller 520, storage device 530, input device 540 and output device 550 in the vehicle can be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0177] Multiple visual sensors 510 are used to obtain environmental feature information around the vehicle in real time.

[0178] The storage device 530, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the multi-source lane line fusion method in the embodiment of the present invention (for example, the detection lane line acquisition module 410, the prediction lane line acquisition module 420 and the fusion lane line update module 430). The controller 520 executes various functional applications and data processing of the vehicle by running the software programs, instructions and modules stored in the storage device 530, that is, to implement the above-mentioned multi-source lane line fusion method, which includes:

[0179] Real-time acquisition of detection lane lines corresponding to each visual sensor in the vehicle, where multiple visual sensors are arranged;

[0180] If a lane line update is detected for the detection lane line corresponding to the target visual sensor, the predicted lane line of the fused lane line at the historical detection time at the current detection time is calculated;

[0181] According to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line, the fused lane line at the current detection moment is updated.

[0182] The storage device 530 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the storage device 530 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some examples, the storage device 530 may further include a memory remotely arranged relative to the controller 520, and these remote memories may be connected to the vehicle via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0183] The input device 540 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the vehicle. The output device 550 may include a display device such as a display screen.

[0184] Embodiment 6

[0185] Embodiment 6 of the present invention further provides a storage medium including computer executable instructions, wherein the computer executable instructions are used to execute a method for fusing multi-source lane lines when executed by a computer processor, the method comprising:

[0186] Real-time acquisition of detection lane lines corresponding to each visual sensor in the vehicle, where multiple visual sensors are arranged;

[0187] If a lane line update is detected for the detection lane line corresponding to the target visual sensor, the predicted lane line of the fused lane line at the historical detection time at the current detection time is calculated;

[0188] According to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line, the fused lane line at the current detection moment is updated.

[0189] Of course, the computer executable instructions of a storage medium including computer executable instructions provided in an embodiment of the present invention are not limited to the method operations described above, and can also execute related operations in the multi-source lane line fusion method provided in any embodiment of the present invention.

[0190] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0191] It is worth noting that in the embodiment of the above-mentioned search device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0192] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for fusing multi-source lane lines, characterized in that: include: Real-time acquisition of detection lane lines corresponding to each visual sensor in the vehicle, where multiple visual sensors are arranged; If a lane line update is detected for the detection lane line corresponding to the target visual sensor, the predicted lane line of the fused lane line at the historical detection time at the current detection time is calculated; According to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line, updating the fused lane line at the current detection moment; Wherein, updating the fused lane line at the current detection moment according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line includes: Obtain the current predicted lane line, and determine whether there is an unused target detection lane line associated with the current predicted lane line; If yes, then update the fused lane line of the current predicted lane line at the current detection time according to the Kalman filter algorithm and the target detection lane line, and mark the target detection lane line as being in use; If not, then based on the Kalman filter algorithm and the current predicted lane line, the fused lane line of the current predicted lane line at the current detection moment is updated.

2. The method according to claim 1, characterized in that The lane line update occurs when the detected lane line corresponding to the target visual sensor is detected, including: Trigger each visual sensor set in the vehicle to obtain environmental feature information around the vehicle in real time; According to the environmental feature information, a detection lane line corresponding to each visual sensor at different detection times is obtained; If it is determined that the lane line detected by the target visual sensor among the visual sensors at the current detection moment is inconsistent with the lane line detected at the previous historical detection moment, it is determined that a lane line change has been detected in the detection lane line corresponding to the target visual sensor.

3. The method according to claim 1, characterized in that Calculate the predicted lane line at the current detection time based on the fused lane line at the historical detection time, including: The Kalman filter algorithm is used to predict and calculate the fused lane line at the historical detection moment, and the predicted lane line of the fused lane line at the current detection moment is obtained.

4. The method according to claim 3, characterized in that The Kalman filter algorithm is used to predict and calculate the fused lane line at the historical detection time to obtain the predicted lane line of the fused lane line at the current detection time, including: According to the vehicle state information at the current detection time, the time interval between the current detection time and the historical detection time, and the fused lane line at the historical detection time, the predicted lane line of the fused lane line at the current detection time is predicted; The Kalman state matrix of the predicted lane line is calculated based on the vehicle state information at the current detection time and the time interval between the current detection time and the historical detection time; According to the Kalman state matrix of the predicted lane line and the final value of the covariance matrix of the fused lane line at the historical detection time, the predicted value of the covariance matrix of the predicted lane line at the current detection time is calculated; The covariance matrix prediction value is used to update the fused lane line at the current detection moment.

5. The method according to claim 1, characterized in that After updating the fused lane line at the current detection moment according to the association result between the detected lane line and the predicted lane line, the method further includes: After determining that the update of the fused lane line at the current detection time is completed for all predicted lane lines, determining whether there is an unused detection lane line; If so, the detected lane line is used as a new fused lane line at the current detection moment.

6. The method according to claim 1, characterized in that Determining whether there is an unused target detection lane line associated with the current predicted lane line includes: From all unused detection lane lines, obtain a current detection lane line in sequence; Obtain the lane line description parameters corresponding to the currently predicted lane line; Determine whether the parameter differences between various lane line description parameters of the currently detected lane line and various lane line description parameters of the currently predicted lane line are all less than or equal to a preset difference threshold; If yes, determine that the currently detected lane line is associated with the currently predicted lane line; If not, the process returns to executing the operation of obtaining one current detection lane line in turn among all unused detection lane lines until all unused detection lane lines are processed.

7. The method according to claim 1, characterized in that According to the Kalman filter algorithm and the target detected lane line, the fused lane line of the current predicted lane line at the current detection time is updated, including: The Kalman gain matrix of the currently predicted lane line is calculated based on the covariance matrix prediction value of the currently predicted lane line at the current detection moment, the preset observation matrix and the measurement noise covariance matrix; According to the current predicted lane line, the Kalman gain matrix, the observation matrix and the target detected lane line, the fused lane line of the current predicted lane line at the current detection moment is updated; According to the predicted value of the covariance matrix of the current predicted lane line at the current detection time, the Kalman gain matrix and the observation matrix, the final value of the covariance matrix of the currently updated fused lane line at the current detection time is calculated; The final value of the covariance matrix is ​​used to predict the predicted value of the covariance matrix of the currently updated fused lane line at the next detection moment.

8. The method according to claim 1, characterized in that According to the Kalman filter algorithm and the current predicted lane line, the fused lane line of the current predicted lane line at the current detection time is updated, including: The Kalman gain matrix of the currently predicted lane line is calculated based on the covariance matrix prediction value of the currently predicted lane line at the current detection moment, the preset observation matrix and the measurement noise covariance matrix; According to the current predicted lane line, the Kalman gain matrix and the observation matrix, a fused lane line of the current predicted lane line at the current detection moment is updated; According to the predicted value of the covariance matrix of the current predicted lane line at the current detection time, the Kalman gain matrix and the observation matrix, the final value of the covariance matrix of the currently updated fused lane line at the current detection time is calculated; The final value of the covariance matrix is ​​used to predict the predicted value of the covariance matrix of the currently updated fused lane line at the next detection moment.

9. The method according to any one of claims 1 to 8, characterized in that: After updating the fused lane line at the current detection moment according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line, the method further includes: According to the fused lane line at the current detection moment, the target lane line is screened and obtained as a reference for driving decision-making during the vehicle's automatic driving process.

10. The method according to claim 9, characterized in that According to the fused lane line at the current detection moment, the target lane line is screened and obtained, including: The credibility weight of the fused lane line updated by the detected lane line is increased; The credibility weight of the fused lane line updated by the unused detection lane line is reduced; The fused lane lines whose credibility weight is greater than or equal to the set weight threshold are selected as the target lane lines.

11. The method according to claim 10, characterized in that After the credibility weight of the fused lane line updated by the unused detection lane line is reduced, the following is further included: Delete the fused lane lines whose credibility weight is less than or equal to the set weight threshold.

12. A multi-source lane line fusion device, characterized in that: include: A detection lane line acquisition module is used to acquire the detection lane lines corresponding to each visual sensor in the vehicle in real time. A plurality of visual sensors are arranged in the vehicle. A predicted lane line acquisition module is used to calculate the predicted lane line of the fused lane line at the historical detection time at the current detection time if a lane line update is detected for the detection lane line corresponding to the target visual sensor; A fused lane line updating module, used to update the fused lane line at the current detection moment according to the detected lane line of the target visual sensor at the current detection moment and the predicted lane line; Among them, the fusion lane line update module is specifically used for: Obtain the current predicted lane line, and determine whether there is an unused target detection lane line associated with the current predicted lane line; If yes, then update the fused lane line of the current predicted lane line at the current detection time according to the Kalman filter algorithm and the target detection lane line, and mark the target detection lane line as being in use; If not, then based on the Kalman filter algorithm and the current predicted lane line, the fused lane line of the current predicted lane line at the current detection moment is updated.

13. A vehicle, characterized in that: The vehicle comprises: Multiple visual sensors are used to obtain real-time environmental feature information around the vehicle; one or more controllers; A storage device for storing one or more programs; When the one or more programs are executed by the one or more controllers, the one or more controllers implement the multi-source lane line fusion method as described in any one of claims 1-11.

14. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the multi-source lane line fusion method according to any one of claims 1 to 11.

Images