Road information generation method and apparatus, electronic device, and computer readable medium

Abstract

Embodiments of the present disclosure disclose a road information generation method and device, an electronic device and a computer readable medium. A specific implementation of the method comprises: determining a camera pose matrix corresponding to each lane line corner point coordinate group in a lane line corner point coordinate group sequence, to obtain a camera pose matrix sequence; generating a matching processing result group sequence; performing screening processing on each lane line corner point in each lane line corner point coordinate group in the lane line corner point coordinate group sequence, to generate a screened lane line corner point coordinate group sequence; determining a coordinate residual value of each screened lane line corner point coordinate in each screened lane line corner point coordinate group in the screened lane line corner point coordinate group sequence, to generate a coordinate residual value group sequence; and generating road information based on the coordinate residual value group sequence. The implementation can improve the accuracy of the generated road information.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to road information generation methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0002] Road information generation is an important technology in the field of autonomous driving. Currently, the common approach to road information generation is to treat lane corner points extracted from road images as static coordinates, determine the position of the corner point coordinates in the vehicle coordinate system or camera coordinate system through coordinate transformation, and then fit them into lane lines as road information.

[0003] However, the inventors discovered that when using the above method to generate road information, the following technical problems often arise:

[0004] First, if the lane lines are partially obscured by obstacles, the corner coordinates of the lane lines extracted from the road image may deviate as the obstacles move, resulting in insufficient accuracy of the extracted corner coordinates, which in turn reduces the accuracy of the generated road information.

[0005] Second, the error in the extracted lane line corner coordinates was not taken into account, which reduced the accuracy of the generated lane line equation.

[0006] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this disclosure provide methods, apparatuses, electronic devices, and computer-readable media for generating road information to address one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this disclosure provide a road information generation method, which includes: performing lane corner detection on each road image in a pre-acquired road image sequence to generate a set of lane corner coordinates and a corresponding set of corner coordinate covariance matrices, thereby obtaining a sequence of lane corner coordinate sets and a sequence of corner coordinate covariance matrices; determining a camera pose matrix corresponding to each lane corner coordinate set in the above-mentioned lane corner coordinate set sequence, thereby obtaining a sequence of camera pose matrices; and generating a sequence of matching processing results based on the above-mentioned camera pose matrix sequence and the above-mentioned lane corner coordinate set sequence. Based on the above matching result sequence, each lane corner point in each lane corner point coordinate group in the above lane corner point coordinate sequence is filtered to generate a filtered lane corner point coordinate sequence; based on the above corner point coordinate covariance matrix sequence and the above camera pose matrix sequence, the coordinate residual value of each filtered lane corner point coordinate in each filtered lane corner point coordinate group in the above filtered lane corner point coordinate sequence is determined to generate a coordinate residual value sequence; based on the above coordinate residual value sequence, road information is generated, wherein the above road information includes a set of lane line equations.

[0010] Secondly, some embodiments of this disclosure provide a road information generation apparatus, which includes: a detection unit configured to perform lane corner detection on each road image in a pre-acquired road image sequence to generate a set of lane corner coordinates and a corresponding set of corner coordinate covariance matrices, thereby obtaining a sequence of lane corner coordinate sets and a sequence of corner coordinate covariance matrices; a first determining unit configured to determine a camera pose matrix corresponding to each lane corner coordinate set in the above-mentioned lane corner coordinate set sequence, thereby obtaining a sequence of camera pose matrices; and a first generating unit configured to generate a sequence of matching processing results based on the above-mentioned camera pose matrix sequence and the above-mentioned lane corner coordinate set sequence; and a screening unit. The first processing unit is configured to filter each lane corner point in each lane corner point coordinate group in the above lane corner point coordinate group sequence based on the above matching processing result group sequence, so as to generate a filtered lane corner point coordinate group sequence; the second determining unit is configured to determine the coordinate residual value of each filtered lane corner point coordinate in each filtered lane corner point coordinate group in the above filtered lane corner point coordinate group sequence based on the above corner point coordinate covariance matrix group sequence and the above camera pose matrix sequence, so as to generate a coordinate residual value group sequence; the second generating unit is configured to generate road information based on the above coordinate residual value group sequence, wherein the above road information includes a lane line equation group.

[0011] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0012] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0013] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0014] The above-described embodiments of this disclosure have the following beneficial effects: the road information generation method of some embodiments of this disclosure can improve the accuracy of the generated road information. Specifically, the reason for the reduced accuracy of the generated road information is that if the lane lines are partially obscured by obstacles, the lane line corner coordinates extracted from the road image are prone to deviation as the obstacles move, resulting in insufficient accuracy of the extracted corner coordinates. Based on this, the road information generation method of some embodiments of this disclosure performs lane line corner detection on each road image in the pre-acquired road image sequence to generate a set of lane line corner coordinates and a corresponding set of corner coordinate covariance matrices, obtaining a sequence of lane line corner coordinate sets and a sequence of corner coordinate covariance matrices. First, through lane line corner detection, not only can the lane line corner coordinates be obtained, but also the corner coordinate covariance matrix corresponding to each lane line corner coordinate can be obtained. Because the corner coordinate covariance matrix is ​​introduced, it can be used to determine whether the lane line corner coordinates have deviated. Next, the camera pose matrix corresponding to each lane corner point coordinate group in the above lane corner point coordinate group sequence is determined, resulting in a camera pose matrix sequence. Then, based on the above camera pose matrix sequence and the above lane corner point coordinate group sequence, a matching processing result group sequence is generated. By generating the matching processing result group sequence, it can be used to further determine whether the lane corner point coordinates have deviated. Next, based on the above matching processing result group sequence, each lane corner point in each lane corner point coordinate group in the above lane corner point coordinate group sequence is filtered to generate a filtered lane corner point coordinate group sequence. The filtering process can be used to remove lane corner point coordinates with deviations. In this way, more accurate data support can be provided for the generation of subsequent road information. Afterwards, based on the above corner point coordinate covariance matrix group sequence and the above camera pose matrix sequence, the coordinate residual value of each filtered lane corner point coordinate in each filtered lane corner point coordinate group in the above filtered lane corner point coordinate group sequence is determined to generate a coordinate residual value group sequence. Here, by determining the coordinate residual values, it is possible to further determine whether the lane line corner point coordinates have deviated. This further improves the accuracy of the generated lane line corner point coordinates. Finally, based on the above sequence of coordinate residual values, road information is generated, which includes a set of lane line equations. Because the accuracy of the generated lane line corner point coordinates is improved, the accuracy of the generated road information is also improved. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0016] Figure 1 This is a flowchart of some embodiments of the road information generation method according to the present disclosure;

[0017] Figure 2 These are schematic diagrams illustrating the structure of some embodiments of the road information generation apparatus according to this disclosure;

[0018] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0021] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0022] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0023] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0024] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Figure 1 A flow 100 of some embodiments of a road information generation method according to the present disclosure is shown. The road information generation method includes the following steps:

[0026] Step 101: Perform lane line corner detection on each road image in the pre-acquired road image sequence to generate lane line corner coordinate set and corresponding corner coordinate covariance matrix set, thus obtaining lane line corner coordinate set sequence and corner coordinate covariance matrix set sequence.

[0027] In some embodiments, the execution entity of the road information generation method can perform lane corner detection on each road image in the pre-acquired road image sequence to generate a set of lane corner coordinates and a corresponding set of corner coordinate covariance matrices, resulting in a sequence of lane corner coordinate sets and a sequence of corner coordinate covariance matrices. The road image sequence can be a series of frames captured by the forward-facing camera of the current vehicle. A preset corner detection algorithm can be used to perform corner detection on the road images to generate a set of lane corner coordinates and a corner coordinate covariance matrix corresponding to each lane corner coordinate. Each set of lane corner coordinates can correspond to a road image, representing the corner of the lane line in that road image. The lane corner coordinates can be the coordinates of a lane corner in the road image. For example, a lane corner can be the coordinates of a corner of a dashed lane line, or the edge coordinates of a solid lane line. The corner coordinate covariance matrix can be a diagonal matrix used to represent the detection error of the corresponding lane corner.

[0028] Step 102: Determine the camera pose matrix corresponding to each lane corner point coordinate group in the lane corner point coordinate group sequence to obtain the camera pose matrix sequence.

[0029] In some embodiments, the execution entity can determine the camera pose matrix corresponding to each lane corner point coordinate group in the lane corner point coordinate group sequence, thereby obtaining a camera pose matrix sequence. Specifically, the camera pose matrix corresponding to each road image sequence in the road image sequence can be obtained from the vehicle's inertial unit via wired or wireless means, serving as the camera pose matrix corresponding to the lane corner point coordinate group.

[0030] In some optional implementations of certain embodiments, the execution entity determines the camera pose matrix corresponding to each lane corner point coordinate group in the above-mentioned lane corner point coordinate group sequence to obtain the camera pose matrix sequence, which may include the following steps:

[0031] The first step is to obtain the initial camera pose matrix sequence. Each initial camera pose matrix in this sequence can be the initial camera pose matrix for a consecutive frame time period corresponding to each road image in the road image sequence. Here, the acquisition frequency of the initial camera pose matrix differs from the acquisition frequency of the road images in the road image sequence.

[0032] The second step involves interpolating the initial camera pose matrix sequence to generate a new camera pose matrix sequence. Since the frequency of the acquired initial camera pose matrices differs from the acquisition frequency of the road images in the road image sequence, the timestamps corresponding to the initial camera pose matrices also differ from those corresponding to the road images. Therefore, a preset interpolation algorithm can be used to interpolate each initial camera pose matrix to generate a camera pose matrix at the same timestamp as the road images, resulting in a camera pose matrix sequence.

[0033] As an example, the above interpolation algorithms may include, but are not limited to, at least one of the following: piecewise linear interpolation, cubic spline interpolation, bilinear interpolation, nearest neighbor interpolation, etc.

[0034] Step 103: Generate a matching processing result sequence based on the camera pose matrix sequence and the lane line corner point coordinate sequence.

[0035] In some embodiments, the execution entity can generate a matching result sequence based on the camera pose matrix sequence and the lane corner coordinate set sequence. Specifically, the camera pose matrix sequence and the lane corner coordinate set sequence can be input to a preset lane corner matching algorithm interface to output the matching result sequence. Here, the lane corner matching algorithm interface can be a preset interface for lane corner matching. The matching result can be the result of matching a lane corner with corresponding lane corners in other road images.

[0036] In some optional implementations of certain embodiments, the execution entity generates a matching processing result sequence based on the camera pose matrix sequence and the lane line corner point coordinate sequence, which may include the following steps:

[0037] Based on the aforementioned camera pose matrix sequence, the lane corner coordinates in each lane corner coordinate group of the aforementioned lane corner coordinate group sequence are matched with the lane corner coordinates in other lane corner coordinate groups to generate a matching result group sequence. Each matching result in the aforementioned matching result group sequence may include a tracking matching success flag or a tracking matching failure flag. A preset lane corner tracking method can be used, utilizing the aforementioned camera pose matrix sequence, to match the lane corner coordinates in each lane corner coordinate group of the aforementioned lane corner coordinate group sequence with the lane corner coordinates in other lane corner coordinate groups to generate a matching result group sequence. Here, the tracking matching success flag can be used to indicate that there is a correspondence between the lane corner coordinates and the lane corner coordinates in other lane corner coordinate groups, that is, it indicates that the lane corner coordinates have been successfully tracked and matched. The tracking matching failure flag can be used to indicate that there is a correspondence between the lane line corner coordinates and the lane line corner coordinates in other lane line corner coordinate groups, that is, it indicates that the lane line corner coordinates are false detection coordinates.

[0038] As an example, the lane corner tracking methods mentioned above may include, but are not limited to, at least one of the following: SIFT (Scale-invariant Feature Transform) algorithm, Surf (SpeededUpRobust Features) algorithm, Harris corner detection, FAST corner detection, BRIEF (BinaryRobust Independent Elementary Features) algorithm, etc.

[0039] In addition, each matching result in the above-mentioned matching result sequence may also include a matching image identifier sequence. The matching image identifiers in the matching image identifier sequence may be image identifiers of road images containing other lane corner points that match the lane corner point coordinates.

[0040] Step 104: Based on the matching processing result group sequence, filter each lane corner point in each lane corner point coordinate group in the lane corner point coordinate group sequence to generate a filtered lane corner point coordinate group sequence.

[0041] In some embodiments, the execution entity may, based on the matching result group sequence, perform filtering on each lane corner point in each lane corner point coordinate group in the lane corner point coordinate group sequence to generate a filtered lane corner point coordinate group sequence.

[0042] In some optional implementations of certain embodiments, the execution entity, based on the matching result sequence, filters each lane corner point in each lane corner point coordinate group in the lane corner point coordinate group sequence to generate a filtered lane corner point coordinate group sequence, which may include the following steps:

[0043] The matching results in the corresponding matching result sequence within the above lane corner coordinate group sequence, including those with matching failure indicators, are removed. The resulting lane corner coordinate group sequence is then used as the filtered lane corner coordinate group sequence. Here, since the matching failure indicator signifies that the lane corner coordinates are false positives, these coordinates can be removed. This improves the accuracy of the filtered lane corner coordinates within the filtered lane corner coordinate group sequence.

[0044] Step 105: Based on the corner coordinate covariance matrix sequence and the camera pose matrix sequence, determine the coordinate residual value of each selected lane corner coordinate in each selected lane corner coordinate group in the selected lane corner coordinate sequence, so as to generate a coordinate residual value sequence.

[0045] In some embodiments, the execution entity may determine the coordinate residual value of each selected lane line corner coordinate in each selected lane line corner coordinate group in the selected lane line corner coordinate group sequence based on the selected corner coordinate covariance matrix group sequence and the camera pose matrix sequence, so as to generate a coordinate residual value group sequence.

[0046] In some optional implementations of certain embodiments, the execution entity determines the coordinate residual value of each selected lane line corner coordinate in each selected lane line corner coordinate group in the selected lane line corner coordinate group sequence based on the corner coordinate covariance matrix group sequence and the camera pose matrix sequence, in order to generate a coordinate residual value group sequence, which may include the following steps:

[0047] The first step involves transforming the coordinates of each selected lane line corner point in each selected lane line corner point coordinate group within the aforementioned sequence of corner point coordinate sets to generate transformed corner point coordinates, resulting in a transformed corner point coordinate set sequence. This coordinate transformation can involve converting the selected lane line coordinates from the image coordinate system to the camera coordinate system. For each selected lane line corner point coordinate in each selected lane line corner point coordinate group within the aforementioned sequence of corner point coordinate sets, firstly, the camera pose matrix corresponding to the selected lane line corner point coordinates can be selected from the aforementioned camera pose matrix sequence to obtain the target camera pose matrix set. Here, each target pose matrix in the target camera pose matrix set can be a camera pose matrix representing the road image at the same timestamp from the matching image identifier sequence included in the matching processing result corresponding to the selected lane line corner point coordinates. Then, the coordinates of each selected lane corner point in each selected lane corner point coordinate group in the above sequence of selected lane corner point coordinate groups can be transformed using the following formula to generate the transformed corner point coordinates:

[0048] Where P represents the transformed corner coordinates. Here, the transformed corner coordinates are three-dimensional coordinates in the camera coordinate system. i represents the sequence number. I represents the number of target pose matrices in the target pose matrix group. e represents the reprojection error of the filtered lane line corner coordinates projected onto the camera coordinate system corresponding to the target camera pose matrix in the target camera pose matrix group. T represents the transpose of the matrix. Here, the corresponding coordinate system can refer to the camera coordinate system at the time (i.e., the timestamp) where the target camera pose matrix is ​​located. ei represents the reprojection error of the filtered lane line corner coordinates projected onto the camera coordinate system corresponding to the i-th target camera pose matrix in the target camera pose matrix group. σ represents the corner coordinate covariance matrix corresponding to the filtered lane line corner coordinates. R represents the target camera pose matrix in the target camera pose matrix group. Ri represents the i-th target camera pose matrix in the target camera pose matrix group. R′ represents the target camera pose matrix corresponding to the filtered lane line corner coordinates in the target camera pose matrix group. p represents the filtered lane line corner coordinates. `arg min` represents the value of minimizing the objective function, i.e., the value of the corner point coordinates after transformation to the minimum. `Π(x)` represents the projection function, used to project the three-dimensional coordinates to a two-dimensional coordinate system (e.g., an image coordinate system). `x` represents the input parameter. Here, the input parameter `M` represents the camera intrinsic parameter matrix of the aforementioned forward-looking vehicle camera. `()1∶2` means taking the first and second elements of the vector within the parentheses. `()3` means taking the third element of the vector within the parentheses.

[0049] In practice, the above formula can be solved using the least squares method.

[0050] The second step involves performing the following generation steps for each transformed corner point coordinate in each transformed corner point coordinate group sequence to generate coordinate residual values:

[0051] The first sub-step involves identifying the road images in the aforementioned road image sequence that match the transformed corner coordinates, thus obtaining a set of matched road images. Here, "matching" can refer to: the matched road images that match the transformed corner coordinates can be: the road images represented by each matching image identifier in the matched image identifier sequence included in the matching processing result corresponding to the transformed corner coordinates.

[0052] The second sub-step involves projecting the transformed corner point coordinates onto each matching road image in the matching road image group to generate a projected corner point coordinate group. A perspective transformation algorithm can be used to project the transformed corner point coordinates onto each matching road image in the matching road image group to generate the projected corner point coordinate group.

[0053] The third sub-step involves determining the total projection error value of each projected corner point coordinate in the projected corner point coordinate set, based on the aforementioned sequence of corner point coordinate covariance matrices, and defining this total projection error value as the coordinate residual value. The total projection error value of each projected corner point coordinate in the projected corner point coordinate set can be determined using the following formula:

[0054] Where E represents the total projection error value. j represents the sequence number. j represents the number of projected corner point coordinates in the projected corner point coordinate group. f represents the projection error value of the projected corner point coordinates in the projected corner point coordinate group. fj represents the projection error value of the j-th projected corner point coordinate in the projected corner point coordinate group. K represents the target camera pose matrix corresponding to the projected corner point coordinates in the projected corner point coordinate group. Kj represents the target camera pose matrix corresponding to the j-th projected corner point coordinate in the projected corner point coordinate group. K′ represents the target camera pose matrix corresponding to the transformed corner point coordinates in the target camera pose matrix group.

[0055] Step 106: Generate road information based on the coordinate residual value sequence.

[0056] In some embodiments, the aforementioned execution entity may generate road information based on the aforementioned coordinate residual value sequence.

[0057] In some optional implementations of certain embodiments, the execution entity generates road information based on the aforementioned coordinate residual value sequence, which may include the following steps:

[0058] The first step involves removing the transformed corner point coordinates from the corresponding coordinate residual value sequences that do not meet a preset residual condition, thus generating a removed corner point coordinate sequence. The preset residual condition can be that the coordinate residual value is less than or equal to a preset residual threshold. Here, transformed corner point coordinates that do not meet the preset residual condition indicate that the transformed corner point coordinates are occluded by an obstacle and deviate from their position due to the movement of the obstacle. Therefore, such transformed corner point coordinates need to be removed.

[0059] The second step involves fitting the coordinates of the removed corner points corresponding to the same lane line in the aforementioned sequence of removed corner point coordinates to generate lane line equations, thus obtaining a set of lane line equations, which is then used as road information. Here, a polynomial fitting method can be used to fit the coordinates of the removed corner points corresponding to the same lane line in the aforementioned sequence of removed corner point coordinates to generate lane line equations, resulting in a set of lane line equations.

[0060] The above formulas and related contents, as an inventive point of this disclosure, solve the second technical problem mentioned in the background art: "The accuracy of the generated lane line equation is reduced because the error of the extracted lane line corner coordinates is not considered." Factors leading to reduced accuracy of the generated lane line equation often include the failure to consider the error of the extracted lane line corner coordinates. Solving these factors improves the accuracy of the generated lane line equation. To achieve this, firstly, interpolation processing can determine the camera pose matrix at the same time as each road image. This can improve the accuracy of the camera pose matrix involved in coordinate transformation. Secondly, by using a more accurate camera pose matrix to perform coordinate transformation on each filtered lane line corner coordinate, more accurate transformed corner coordinates can be obtained. Here, because the above formula for generating transformed corner coordinates is introduced, the camera pose matrix corresponding to each filtered lane line corner coordinate can be used to minimize the reprojection error between the filtered lane line corner coordinates and the corner coordinates at different times. This improves the accuracy of the generated transformed corner coordinates. Then, through coordinate projection, the coordinates of each transformed corner point can be projected onto the image coordinate system of the matching road image. This allows us to determine the transformation error between the transformed corner point coordinates and the corresponding corner point coordinates in the image coordinate systems of each matching road image—that is, the total projection error. This accounts for the error in the extracted lane line corner point coordinates, and the total projection error can be used to filter the transformed corner point coordinates. This improves the accuracy of the generated corner point coordinates after removal, and consequently, improves the accuracy of the generated lane line equations.

[0061] Optionally, the above residual threshold can be generated through the following steps:

[0062] The first step is to determine the corresponding target residual value for each preprocessed corner point coordinate in the preprocessed corner point coordinate group using the methods described in steps 103-105 above, thus obtaining the target residual value group. Here, each preprocessed corner point coordinate in the aforementioned preprocessed corner point coordinate group can be the lane line corner point coordinates detected from the road image after preprocessing, provided it is not obscured by obstacles.

[0063] The second step is to fit the distribution of each target residual value in the above target residual value group to a folded normal distribution. The location and scale parameters of this folded normal distribution can be determined using a maximum likelihood estimation algorithm.

[0064] The third step is to determine the residual threshold by adding the product of the location parameter, the scale parameter, and the coefficient.

[0065] As an example, the coefficient above can be 2 or 3. If the coefficient is 2, it indicates that the probability of the target residual value being less than the residual threshold is approximately 95.45%. If the coefficient is 3, it indicates that the probability of the target residual value being less than the residual threshold is approximately 99.73%.

[0066] Here, if the target residual value is greater than the residual threshold, it indicates that the processed corner coordinates corresponding to the target residual value are occluded. Therefore, the generated residual threshold can be used to filter the transformed corner coordinates, thereby improving the accuracy of the transformed corner coordinates.

[0067] Optionally, the aforementioned executing entity may also send the aforementioned road information to a display terminal for display.

[0068] Optionally, for each of the filtered lane line corner point coordinate groups in the above-mentioned filtered lane line corner point coordinate group sequence, its corresponding lane line information can be determined. The lane line information in the lane line information group can include the lane line type identifier. For example, the lane type identifier can be a solid line identifier or a dashed line identifier, used to indicate whether the lane line is a solid line or a dashed line. Here, a preset lane line detection algorithm can be used to determine the lane line information corresponding to each of the filtered lane line corner point coordinate groups in the above-mentioned filtered lane line corner point coordinate group sequence, thus obtaining the lane line information group. Then, for each converted lane line corner point coordinate group whose corresponding lane line information is a solid line identifier, if there are converted lane line corner point coordinates in the above-mentioned converted lane line corner point coordinate group whose coordinate residual value satisfies the above-mentioned preset residual condition, then the generated coordinate residual value can be determined again through the methods described in steps 103-106 above to determine whether it satisfies the above-mentioned preset residual condition. Finally, if there are still coordinate residual values ​​that satisfy the above-mentioned preset residual condition, then it can be determined that the detection of the lane line type identifier corresponding to the above-mentioned filtered lane line corner point coordinate group is incorrect. Therefore, information indicating incorrect lane line recognition can also be added to the aforementioned road information.

[0069] As an example, the lane detection algorithms mentioned above may include, but are not limited to, at least one of the following: UFLD (UltraFast Structure-aware Deep Lane Detection), LaneNet (multi-branch lane detection network) algorithm, LSD (Line Segment Detector) algorithm, Freeman line detection algorithm, etc.

[0070] In practice, the above steps can be used to identify situations where dashed lane lines are misdetected as solid lane lines due to obstruction by obstacles. This improves the accuracy of lane line detection, which in turn enhances the accuracy of the generated road information.

[0071] The above-described embodiments of this disclosure have the following beneficial effects: the road information generation method of some embodiments of this disclosure can improve the accuracy of the generated road information. Specifically, the reason for the reduced accuracy of the generated road information is that if the lane lines are partially obscured by obstacles, the lane line corner coordinates extracted from the road image are prone to deviation as the obstacles move, resulting in insufficient accuracy of the extracted corner coordinates. Based on this, the road information generation method of some embodiments of this disclosure performs lane line corner detection on each road image in the pre-acquired road image sequence to generate a set of lane line corner coordinates and a corresponding set of corner coordinate covariance matrices, obtaining a sequence of lane line corner coordinate sets and a sequence of corner coordinate covariance matrices. First, through lane line corner detection, not only can the lane line corner coordinates be obtained, but also the corner coordinate covariance matrix corresponding to each lane line corner coordinate can be obtained. Because the corner coordinate covariance matrix is ​​introduced, it can be used to determine whether the lane line corner coordinates have deviated. Next, the camera pose matrix corresponding to each lane corner point coordinate group in the above lane corner point coordinate group sequence is determined, resulting in a camera pose matrix sequence. Then, based on the above camera pose matrix sequence and the above lane corner point coordinate group sequence, a matching processing result group sequence is generated. By generating the matching processing result group sequence, it can be used to further determine whether the lane corner point coordinates have deviated. Next, based on the above matching processing result group sequence, each lane corner point in each lane corner point coordinate group in the above lane corner point coordinate group sequence is filtered to generate a filtered lane corner point coordinate group sequence. The filtering process can be used to remove lane corner point coordinates with deviations. In this way, more accurate data support can be provided for the generation of subsequent road information. Afterwards, based on the above corner point coordinate covariance matrix group sequence and the above camera pose matrix sequence, the coordinate residual value of each filtered lane corner point coordinate in each filtered lane corner point coordinate group in the above filtered lane corner point coordinate group sequence is determined to generate a coordinate residual value group sequence. Here, by determining the coordinate residual values, it is possible to further determine whether the lane line corner point coordinates have deviated. This further improves the accuracy of the generated lane line corner point coordinates. Finally, based on the above sequence of coordinate residual values, road information is generated, which includes a set of lane line equations. Because the accuracy of the generated lane line corner point coordinates is improved, the accuracy of the generated road information is also improved.

[0072] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a road information generation device, which are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0073] like Figure 2 As shown, a road information generation device 200 in some embodiments includes: a detection unit 201, a first determination unit 202, a first generation unit 203, a filtering processing unit 204, a second determination unit 205, and a second generation unit 206. The detection unit 201 is configured to perform lane corner detection on each road image in a pre-acquired road image sequence to generate a set of lane corner coordinates and a corresponding set of corner coordinate covariance matrices, obtaining a sequence of lane corner coordinate sets and a sequence of corner coordinate covariance matrices. The first determination unit 202 is configured to determine the camera pose matrix corresponding to each lane corner coordinate set in the above-mentioned lane corner coordinate set sequence, obtaining a sequence of camera pose matrices. The first generation unit 203 is configured to generate a sequence of matching processing results based on the above-mentioned camera pose matrix sequence and the above-mentioned lane corner coordinate set sequence. The filtering processing unit 204 is configured to... The matching processing result sequence is used to filter each lane corner point in each lane corner point coordinate group in the above lane corner point coordinate group sequence to generate a filtered lane corner point coordinate group sequence; the second determining unit 205 is configured to determine the coordinate residual value of each filtered lane corner point coordinate in each filtered lane corner point coordinate group in the above filtered lane corner point coordinate group sequence based on the above corner point coordinate covariance matrix group sequence and the above camera pose matrix sequence to generate a coordinate residual value group sequence; the second generating unit 206 is configured to generate road information based on the above coordinate residual value group sequence, wherein the above road information includes a lane line equation set.

[0074] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0075] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0076] like Figure 3As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0077] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0078] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0079] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0080] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0081] The aforementioned computer-readable medium may be included in the aforementioned device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: perform lane corner detection on each road image in the pre-acquired road image sequence to generate a set of lane corner coordinates and a corresponding set of corner coordinate covariance matrices, obtaining a sequence of lane corner coordinate sets and a sequence of corner coordinate covariance matrices; determine the camera pose matrix corresponding to each lane corner coordinate set in the aforementioned sequence of lane corner coordinate sets, obtaining a sequence of camera pose matrices; and generate a matching... The process involves processing a sequence of matching results; based on this sequence, filtering is performed on each lane corner point in each lane corner point coordinate group within the aforementioned lane corner point coordinate sequence to generate a filtered lane corner point coordinate sequence; based on the aforementioned corner point coordinate covariance matrix sequence and the aforementioned camera pose matrix sequence, the coordinate residual value of each filtered lane corner point coordinate in each filtered lane corner point coordinate group within the aforementioned filtered lane corner point coordinate sequence is determined to generate a coordinate residual value sequence; and based on the aforementioned coordinate residual value sequence, road information is generated, wherein the road information includes a set of lane line equations.

[0082] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0084] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a detection unit, a first determination unit, a first generation unit, a filtering processing unit, a second determination unit, and a second generation unit. The names of these units do not necessarily limit the specific unit itself. For example, the detection unit may also be described as "a unit that performs lane corner detection on each road image in a pre-acquired road image sequence to generate a set of lane corner coordinates and a corresponding set of corner coordinate covariance matrices, thereby obtaining a sequence of lane corner coordinate sets and a sequence of corner coordinate covariance matrices."

[0085] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0086] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating road information, comprising: For each road image in the pre-acquired road image sequence, lane line corner points are detected to generate a set of lane line corner point coordinates and a corresponding set of corner point coordinate covariance matrices, resulting in a sequence of lane line corner point coordinate sets and a sequence of corner point coordinate covariance matrices. Determine the camera pose matrix corresponding to each lane corner point coordinate group in the lane corner point coordinate group sequence to obtain the camera pose matrix sequence; Based on the camera pose matrix sequence and the lane line corner point coordinate sequence, a matching processing result sequence is generated. Based on the matching result group sequence, each lane corner point in each lane corner point coordinate group in the lane corner point coordinate group sequence is filtered to generate a filtered lane corner point coordinate group sequence. Based on the corner coordinate covariance matrix sequence and the camera pose matrix sequence, determine the coordinate residual value of each selected lane corner coordinate in each selected lane corner coordinate group in the selected lane corner coordinate sequence to generate a coordinate residual value sequence. Based on the coordinate residual value sequence, road information is generated, wherein the road information includes a set of lane line equations.

2. The method according to claim 1, wherein, The method further includes: The road information is sent to the display terminal for display.

3. The method according to claim 1, wherein, The step of determining the camera pose matrix corresponding to each lane corner point coordinate group in the lane corner point coordinate group sequence, to obtain the camera pose matrix sequence, includes: Obtain the initial camera pose matrix sequence; The initial camera pose matrix sequence is interpolated to generate a new camera pose matrix sequence.

4. The method according to claim 1, wherein, The step of generating a matching processing result sequence based on the camera pose matrix sequence and the lane line corner point coordinate sequence includes: Based on the camera pose matrix sequence, the lane corner coordinates in each lane corner coordinate group in the lane corner coordinate group sequence are matched with the lane corner coordinates in other lane corner coordinate groups to generate a matching result group, resulting in a matching result group sequence. Each matching result in the matching result group sequence includes a tracking matching success flag or a tracking matching failure flag.

5. The method according to claim 1, wherein, The step of filtering each lane corner point in each lane corner point coordinate group within the lane corner point coordinate group sequence based on the matching processing result group sequence to generate a filtered lane corner point coordinate group sequence includes: The matching results in the corresponding matching result group sequence in the lane corner point coordinate group sequence are removed, including lane corner point coordinates with matching failure indicators, and the lane corner point coordinate group sequence after removal is determined as the filtered lane corner point coordinate group sequence.

6. The method according to claim 1, wherein, The determination of the coordinate residual value of each selected lane line corner coordinate in each selected lane line corner coordinate group in the selected lane line corner coordinate group sequence based on the corner coordinate covariance matrix sequence and the camera pose matrix sequence includes: Based on the corner coordinate covariance matrix sequence and the camera pose matrix sequence, coordinate transformation is performed on each selected lane corner coordinate in each selected lane corner coordinate group in the selected lane corner coordinate sequence to generate transformed corner coordinates, thus obtaining the transformed corner coordinate sequence. For each transformed corner point coordinate in each transformed corner point coordinate group in the sequence of transformed corner point coordinate groups, the following generation steps are performed to generate coordinate residual values: Determine the road images in the road image sequence that match the transformed corner point coordinates to obtain a matching road image group; The transformed corner coordinates are projected onto each matching road image in the matching road image group to generate the projected corner coordinate group; Based on the sequence of the corner coordinate covariance matrix group, the total projection error value of each corner coordinate after projection in the projected corner coordinate group is determined, and the total projection error value is determined as the coordinate residual value.

7. The method according to claim 6, wherein, The process of generating road information based on the coordinate residual value sequence includes: Remove the transformed corner point coordinates whose coordinate residual values ​​in the corresponding coordinate residual value group sequence do not meet the preset residual conditions, so as to generate a removed corner point coordinate group sequence. The coordinates of the removed corner points corresponding to the same lane line in the sequence of removed corner point coordinates are fitted to generate lane line equations, resulting in a set of lane line equations, and the set of lane line equations is determined as road information.

8. A road information generation device, comprising: The detection unit is configured to perform lane line corner detection on each road image in the pre-acquired road image sequence to generate a set of lane line corner coordinates and a corresponding set of corner coordinate covariance matrices, thereby obtaining a sequence of lane line corner coordinate sets and a sequence of corner coordinate covariance matrices. The first determining unit is configured to determine the camera pose matrix corresponding to each lane corner point coordinate group in the lane corner point coordinate group sequence, thereby obtaining a camera pose matrix sequence. The first generation unit is configured to generate a matching processing result set sequence based on the camera pose matrix sequence and the lane line corner point coordinate set sequence; The filtering processing unit is configured to perform filtering processing on each lane corner point in each lane corner point coordinate group in the lane corner point coordinate group sequence based on the matching processing result group sequence, so as to generate a filtered lane corner point coordinate group sequence. The second determining unit is configured to determine the coordinate residual value of each selected lane corner coordinate in each selected lane corner coordinate group in the selected lane corner coordinate group sequence based on the corner coordinate covariance matrix group sequence and the camera pose matrix sequence, so as to generate a coordinate residual value group sequence. The second generation unit is configured to generate road information based on the coordinate residual value set sequence, wherein the road information includes a set of lane line equations.

9. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images