Roll angle calibration method, device, equipment and storage medium for vehicle-mounted camera

Abstract

The present application discloses a roll angle calibration method, apparatus, device and storage medium for a vehicle-mounted camera, and relates to the field of vehicle networking technology. The embodiments of the present application can be applied to maps, autonomous driving, vehicle-mounted or other scenarios. The method comprises: obtaining a sequence of image frames collected by a vehicle-mounted camera during vehicle driving; selecting a plurality of image groups from the image frame sequence, each image group comprising a plurality of continuous image frames in the image frame sequence; obtaining lane line images corresponding to the plurality of image groups at a target roll angle, and obtaining a lane line image set corresponding to the target roll angle; and determining the roll angle of the vehicle-mounted camera based on the lane line image sets corresponding to different roll angles. The present application reduces the error of the fitted lane line and improves the accuracy of the roll angle calibration.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of vehicle networking technology, and in particular to a roll angle calibration method, device, equipment, and storage medium for a vehicle-mounted camera. Background Art

[0002] At present, by taking pictures of the current road surface with an on-board camera and performing IPM (Inverse Perspective Mapping) projection on the pictures, a two-dimensional plane image of the lane lines on the current road surface can be fitted. This is widely used in mapping and navigation.

[0003] In related technologies, if you want to fit lane lines, you usually perform IPM projection on a single frame image and calibrate the roll angle based on the width of the fitted lane line. This will ensure that the fitted lane lines are as clear as possible.

[0004] However, errors are inevitable when fitting lane lines based on a single-frame image, so the calibrated roll angle is less accurate. Summary of the Invention

[0005] The present invention provides a roll angle calibration method, device, equipment, and storage medium for a vehicle-mounted camera. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a roll angle calibration method for a vehicle-mounted camera is provided, the method comprising:

[0007] Obtaining a sequence of image frames captured by the vehicle-mounted camera during vehicle travel, the sequence of image frames comprising n image frames obtained by the vehicle-mounted camera photographing the road at n time stamps, where n is an integer greater than 1;

[0008] Selecting a plurality of image groups from the image frame sequence, each of the image groups comprising a plurality of consecutive image frames in the image frame sequence;

[0009] Acquiring lane line images corresponding to the plurality of image groups at a target roll angle, respectively, to obtain a lane line image set corresponding to the target roll angle; wherein the lane line image corresponding to each of the image groups at the target roll angle includes: at least one lane line obtained by fitting lane line detection points extracted from a plurality of image frames included in the image group at the target roll angle;

[0010] The roll angle of the vehicle-mounted camera is determined according to lane line image sets corresponding to different roll angles.

[0011] According to one aspect of an embodiment of the present application, a roll angle calibration device for a vehicle-mounted camera is provided, the device comprising:

[0012] a frame sequence acquisition module, configured to acquire a sequence of image frames captured by the vehicle-mounted camera during vehicle travel, wherein the sequence of image frames includes n image frames obtained by the vehicle-mounted camera photographing the road at n time stamps, where n is an integer greater than 1;

[0013] An image group selection module, configured to select a plurality of image groups from the image frame sequence, each of the image groups comprising a plurality of consecutive image frames in the image frame sequence;

[0014] an image set acquisition module, configured to acquire lane line images corresponding to the plurality of image groups at a target roll angle, thereby obtaining a lane line image set corresponding to the target roll angle; wherein the lane line image corresponding to each of the image groups at the target roll angle includes at least one lane line obtained by fitting lane line detection points extracted from the plurality of image frames included in the image group at the target roll angle;

[0015] The roll angle determination module is used to determine the roll angle of the vehicle-mounted camera based on lane line image sets corresponding to different roll angles.

[0016] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned roll angle calibration method of the vehicle-mounted camera.

[0017] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned roll angle calibration method of the vehicle-mounted camera.

[0018] According to one aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned roll angle calibration method for a vehicle-mounted camera.

[0019] The technical solution provided by the embodiments of the present application may include the following beneficial effects: when determining the roll angle of a vehicle-mounted camera, multiple image frames captured by the vehicle-mounted camera are used as a reference basis, avoiding the errors existing in considering only a single-frame image, and helping to improve the accuracy of the roll angle finally determined; moreover, the present application adopts the idea of ​​big data statistics, by fitting a plurality of lane line image sets corresponding to different roll angles, and selecting the roll angle corresponding to the lane line image set with the best overall fitting result as the roll angle corresponding to the vehicle-mounted camera finally determined, thereby reducing the error of the fitted lane line and improving the accuracy of the roll angle calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of an IPM projection provided by an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of lane line detection points in an image frame provided by an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a lane line image fitted based on lane line detection results provided by an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of an implementation environment for a solution provided by an embodiment of the present application;

[0024] Figure 5 This is a flow chart of a roll angle calibration method for a vehicle-mounted camera provided by one embodiment of the present application;

[0025] Figure 6 is a schematic diagram of a sliding window provided by an embodiment of the present application;

[0026] Figure 7 is a schematic diagram of lane detection results provided by an embodiment of the present application;

[0027] Figure 8 This is a schematic diagram of the effect of fitting a first-order function to a lane detection result provided by an embodiment of the present application;

[0028] Figure 9 This is a schematic diagram of the effect of fitting a lane detection result using a second-order function, provided by one embodiment of the present application;

[0029] Figure 10 Schematic diagram of a lane line image after fitting the lane line detection results, provided in one embodiment of the present application;

[0030] Figure 11 is a schematic diagram of a lane line image after fitting the lane line detection result provided by another embodiment of the present application;

[0031] Figure 12 This is a schematic diagram of a lane line image corresponding to a determined roll angle provided by an embodiment of the present application;

[0032] Figure 13 This is a schematic diagram of a lane line image corresponding to a roll angle of 1.0 provided by an embodiment of the present application;

[0033] Figure 14 This is a schematic diagram of a lane line image corresponding to a roll angle of 0.0 provided by an embodiment of the present application;

[0034] Figure 15 This is a schematic diagram of the quantitative relationship between the target roll angle and the lane line image provided by one embodiment of the present application;

[0035] Figure 16 is a flow chart of a roll angle calibration method for a vehicle-mounted camera provided by another embodiment of the present application;

[0036] Figure 17 is a flow chart of a roll angle calibration method for a vehicle-mounted camera provided by another embodiment of the present application;

[0037] Figure 18 is a schematic diagram of a lane line image provided by an embodiment of the present application;

[0038] Figure 19 is a flow chart of a roll angle calibration method for a vehicle-mounted camera provided by another embodiment of the present application;

[0039] Figure 20 This is an application scenario of an embodiment of the present application;

[0040] Figure 21 This is a block diagram of a roll angle calibration device for a vehicle-mounted camera provided by one embodiment of the present application;

[0041] Figure 22 is a block diagram of a roll angle calibration device for a vehicle-mounted camera provided by another embodiment of the present application;

[0042] Figure 23 This is a structural block diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0044] Before introducing the embodiments of the present application, the relevant terms involved in the present application are first explained.

[0045] IPM projection: refers to the situation where the originally parallel road surface appears at a certain angle when the onboard camera installed on the vehicle collects lane line images (such as Figure 1 (a)), restore the lane lines with a certain angle in the image to parallel (as shown in Figure 1 The process shown in (b) is called IPM projection.

[0046] Lane line detection model: A model used to detect lane lines, which obtains lane line detection points as output by inputting image frames. In one possible embodiment, the lane line detection model based on a deep neural network (such as LaneNet) regards lane line detection as an instance segmentation problem and solves an end-to-end problem, where the input is an image and the output is the coordinates of the lane line detection points (or "lane line pixels") and lane line identification. After obtaining this information, the image is usually projected to a bird's-eye view to further improve the lane line fitting. The lane line detection points obtained by the lane line detection model, such as Figure 2 As shown, a series of points along the lane line 20 are lane line detection points 21. Figure 2 The coordinates of the lane line detection point 21 shown in FIG are projected onto the bird's-eye view, i.e., the vehicle body coordinate system, according to the IPM projection to obtain the lane line detection result. The lane line image 30 is fitted according to the lane line detection result, i.e., Figure 3 As shown, 31 is the fitted lane line, and the lane lines from a bird's-eye view tend to be parallel.

[0047] Pitch angle: The angle at which a vehicle-mounted camera rotates around the horizontal plane perpendicular to the vehicle's speed direction is called the pitch angle, which can be understood as the "nodding" angle of the vehicle-mounted camera.

[0048] Yaw angle: The angle at which the on-board camera rotates around the direction of gravity is called the yaw angle, which can be understood as the angle at which the on-board camera "shakes its head".

[0049] Roll angle: The angle at which the on-board camera rotates around the direction of vehicle speed is called the roll angle, which can be understood as the angle at which the on-board camera "rolls".

[0050] Vehicle body coordinate system: refers to the two-dimensional coordinate system perpendicular to the direction of gravity.

[0051] Please refer to Figure 4 , which shows a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application. The implementation environment 400 of the solution may include: a vehicle 40, a vehicle-mounted camera 41 and a computer device 42.

[0052] An onboard camera 41 is mounted on a vehicle 40 and is used to capture lane markings on a road 44. The camera 41 captures the lane markings on the road 44 and transmits the captured images to a computer 42. The computer 42 processes and analyzes the received images to derive a calibrated roll angle, and then proceeds to fit the lane markings based on the calibrated roll angle.

[0053] The computer device 42 can be any electronic device with data calculation and storage capabilities. For example, the computer device 42 can be a terminal device or a server. The terminal device can be a terminal device equipped by the vehicle 40 itself, such as an on-board terminal device, a car computer, etc., or it can be a terminal device that can communicate with the on-board camera 41, such as a smart phone, a tablet computer, a PC (Personal Computer), a smart wearable device, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The on-board camera 41 and the computer device 42 can be directly or indirectly connected by wire or wireless means, and this application does not limit this. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0054] above Figure 4 The implementation environment of the scheme given is only exemplary and explanatory. In addition to being applicable to the above-mentioned implementation environment, the technical solution of this application can also be applied to other implementation environments or scenarios with camera roll angle calibration or lane line detection requirements. This application does not limit this.

[0055] Please refer to Figure 5 , which shows a flow chart of a roll angle calibration method for a vehicle-mounted camera provided by an embodiment of the present application. The execution subject of each step of the method can be Figure 4 The computer device 42 in the implementation environment of the solution shown. In the following method embodiment, for ease of description, only the execution subject of each step is introduced as "computer device". The method can include at least one of the following steps (510-540).

[0056] Step 510 , obtaining an image frame sequence captured by the vehicle-mounted camera during vehicle driving, the image frame sequence including n image frames obtained by the vehicle-mounted camera shooting the road at n time stamps, where n is an integer greater than 1.

[0057] On-board cameras refer to cameras installed on vehicles. Based on their installation location, they can be categorized into four categories: front-view, side-view, rear-view, and internal. Based on their functional applications, they can be divided into three main categories: driving assistance, parking assistance, and in-vehicle recording. The on-board cameras in the embodiments of this application primarily refer to forward-view cameras, used to capture image frames while the vehicle is in motion.

[0058] Timestamp: Typically a sequence of characters that uniquely identifies a moment in time. The time interval between two adjacent timestamps can be fixed or variable. In some embodiments, the time interval between two adjacent timestamps is fixed at 0.01s. In some embodiments, the time interval between two adjacent timestamps is variable and randomly varies from 0.01s to 0.1s.

[0059] Image frame: This refers to an image captured by an onboard camera of the current lane conditions. The mth image frame is captured at the mth timestamp, where m is an integer greater than 1 and less than or equal to n. The image frame reflects the current road conditions at a specific timestamp.

[0060] Image frame sequence: refers to n image frames captured by the vehicle's camera at n timestamps, where n is an integer greater than 1. These n image frames, arranged according to timestamps, form an image frame sequence. In some embodiments, where n is 1000, the image frame sequence is 1000 image frames captured by the vehicle's camera at 1000 timestamps.

[0061] The computer device acquires a sequence of image frames captured by a vehicle-mounted camera while the vehicle is traveling. The acquisition method may be at least one of photographing or recording. In some embodiments, the computer device acquires n image frames captured by the vehicle-mounted camera at n timestamps while the vehicle is traveling. In some embodiments, the computer device acquires image frames corresponding to the n timestamps in the video recorded by the vehicle-mounted camera.

[0062] Step 520 : Select multiple image groups from the image frame sequence, each image group including multiple consecutive image frames in the image frame sequence.

[0063] Image group: refers to multiple consecutive image frames selected from an image frame sequence. In some embodiments, an image group is five consecutive image frames from an image frame sequence. The number of image frames included in an image group can be set based on actual needs and is not limited by this application. The number of image groups included in different image groups can be the same or different.

[0064] A computer device selects multiple image groups from a sequence of image frames, each image group comprising multiple consecutive image frames from the sequence of image frames. In some embodiments, the image groups are selected using a sliding window. In some embodiments, the computer device moves the sliding window to multiple different positions in the sequence of image frames to obtain multiple image groups; wherein the multiple consecutive image frames contained in the sliding window at each position correspond to one image group. In some embodiments, each movement of the sliding window is spaced apart by one image frame. In some embodiments, each movement of the sliding window is spaced apart by multiple image frames.

[0065] In some embodiments, a sliding window includes 5 image frames, and each movement of the sliding window is separated by 1 image frame. Figure 6 , which shows a sliding window provided by an embodiment of the present application, wherein the image frame sequence 60 and the sliding window 61. Figure 6 (a) shows an image group corresponding to five image frames selected by a sliding window at the first timestamp, which are the first to fifth image frames (i.e., the image frames numbered 1 to 5 in the figure); Figure 6 (b) shows an image group corresponding to 5 image frames selected by the second timestamp in a sliding window manner, namely the second to sixth image frames (ie, image frames numbered 2 to 6 in the figure).

[0066] Step 530: Acquire lane line images corresponding to the target roll angle for each of the multiple image groups to obtain a lane line image set corresponding to the target roll angle; wherein the lane line image corresponding to each image group at the target roll angle includes: at least one lane line obtained by fitting lane line detection points extracted from the multiple image frames contained in the image group at the target roll angle.

[0067] Target roll angle: refers to any roll angle from among multiple roll angles. For each of the multiple different roll angles, a similar processing method as the target roll angle can be adopted to obtain a lane line image set corresponding to the roll angle, thereby obtaining lane line image sets corresponding to multiple different roll angles. In some embodiments, the target roll angle includes, but is not limited to, at least one or more of -2.0, -1.5, -1.0, -0.5, 0.0, 0.5, 1.0, 1.5, and 2.0. In some embodiments, there are five roll angles, namely 0.0, 0.5, 1.0, 1.5, and 2.0. Based on these five roll angles, lane line image sets corresponding to the five roll angles are obtained.

[0068] Lane Mark Image: A lane mark image is obtained by projecting multiple image frames from an image group. In some embodiments, the lane mark image in the vehicle body coordinate system is obtained by performing IPM projection on the lane mark detection points in the image frames. In some embodiments, the lane mark image includes at least one lane mark fitted to the lane mark detection points extracted from the multiple image frames included in the image group at a target roll angle. In some embodiments, the target roll angle is 2.0, and the lane mark image includes two lane marks fitted to the lane mark detection points extracted from the multiple image frames included in the image group at a target roll angle of 2.0. Figure 7 The lane line detection result provided by an embodiment of the present application is shown, including a lane line detection point 71.

[0069] Lane line image set: Multiple lane line images corresponding to the same roll angle constitute the lane line image set corresponding to that roll angle. Each roll angle corresponds to a lane line image set.

[0070] The computer device obtains lane line images corresponding to the target roll angle for the plurality of image groups, and obtains a lane line image set corresponding to the target roll angle. In some embodiments, lane line detection points extracted from the plurality of image frames included in the image group are fitted to obtain a lane line image containing at least one lane line. In some embodiments, lane line detection points are extracted from the plurality of image frames using a lane line detection model. In some embodiments, the lane line detection model is a LaneNet lane line detection model. In some embodiments, the lane line detection model can also be a model based on other neural network structural components, which is not limited in this application.

[0071] In some embodiments, the lane detection points are fitted by first projecting them through IPM and then fitting them using a first-order function. In some embodiments, the lane detection points are fitted by first projecting them through IPM and then fitting them using a second-order function. In some embodiments, the target roll angle is 1.0, and the computer device uses the LaneNet lane detection model to extract lane detection points from the five image frames included in the image group. After obtaining the coordinates of the lane detection points, the computer device uses IPM projection and fits the lane detection results using a first-order function to obtain five lane line images containing three lane lines, thereby obtaining a lane line image set. Figure 8 The effect of fitting the lane detection result using a first-order function according to an embodiment of the present application is shown. Figure 8 As shown, it is not difficult to see that the result of fitting using the first-order function is that the lane line detection result is less away from the fitted straight line 81.

[0072] In some embodiments, the target roll angle is 2.0, and the computer device uses the LaneNet lane line detection model to extract lane line detection points from the five image frames contained in the image group. After obtaining the coordinates of the lane line detection points, the lane line detection results are obtained through IPM projection. The lane line detection results are fitted with a second-order function to obtain five lane line images containing four lane lines, thereby obtaining a lane line image set. Figure 9 The effect of fitting the lane detection result with a second-order function according to one embodiment of the present application is shown. Figure 9 As shown in the figure, it is not difficult to see that the lane line detection result of the second-order function fitting is significantly different from the fitted straight line 91. The computer device removes lane line images from the lane line image set that meet the third condition, thereby obtaining a filtered lane line image set. The third condition includes at least one of the following: the lane line fitting error is greater than or equal to a first threshold, and the number of lane lines included is less than or equal to a second threshold.

[0073] In some embodiments, the first threshold is A, which is a positive number. In some embodiments, A is 0.5, and the computer device removes lane line images having a lane line fitting error greater than or equal to 0.5 from the lane line image set to obtain a filtered lane line image set. Figure 10 The lane line image after fitting the lane line detection result provided by one embodiment of the present application is shown. It is not difficult to see that the lane line detection result is far away from the fitting line 101. After calculation, its fitting error is greater than A. Figure 10 The lane line images shown should be removed. In some embodiments, the lane line image set contains 200 lane line images, but 20 of them have fitting errors greater than 0.5, and 20 have fitting errors equal to 0.5. After removing these 40 lane line images, a filtered lane line image set consisting of 160 lane line images is obtained.

[0074] In some embodiments, the second threshold is B, which is a positive integer. In some embodiments, B is 2. The computer device removes lane line images containing lane lines less than or equal to 2 from the lane line image set to obtain a filtered lane line image set. Figure 11 The lane line image after fitting the lane line detection result provided by another embodiment of the present application is shown. It is not difficult to see that the number of fitted lane lines 111 is only 2, and the number of lane lines is equal to 2. Figure 11 The lane line images shown should be removed. In some embodiments, the lane line image set contains 200 lane line images, but 10 of the lane line images contain less than 2 lane lines, and 10 of the lane line images contain 2 lane lines. After removing these 30 lane line images, a filtered lane line image set consisting of 170 lane line images is obtained.

[0075] In some embodiments, the first threshold is A, which is a positive number, and the second threshold is B, which is a positive integer. In some embodiments, A is 0.5 and B is 2. The computer device removes lane line images from the lane line image set whose lane line fitting error is greater than or equal to 0.5 or whose number of lane lines is less than or equal to 2, thereby obtaining a filtered lane line image set. In some embodiments, the lane line image set contains 200 lane line images, 40 of which have a lane line fitting error greater than or equal to 0.5 and 30 of which have a number of lane line lines less than or equal to 2. Ten of these lane line images have a lane line fitting error greater than or equal to 0.5 and a number of lane lines less than or equal to 2. In other words, a total of 60 lane line images have a lane line fitting error greater than or equal to 0.5 or a number of lane lines less than or equal to 2. These 60 lane line images are removed, resulting in a filtered lane line image set containing 140 lane line images.

[0076] Step 540 : Determine the roll angle of the vehicle-mounted camera based on the lane line image sets corresponding to different roll angles.

[0077] If the elimination step in step 530 is not performed, the roll angle of the vehicle-mounted camera is determined based on the lane line image sets corresponding to different roll angles, where the lane line image sets are not filtered.

[0078] If the elimination step in step 530 is performed to obtain a filtered lane line image set, the roll angle of the vehicle camera is determined based on the filtered lane line image sets corresponding to different roll angles. In some embodiments, a roll angle of 2.0 corresponds to a filtered lane line image set of 30 lane line images; in some embodiments, a roll angle of 1.0 corresponds to a filtered lane line image set of 100 lane line images. By eliminating lane line images from the lane line image set that meet the third condition, clearly unqualified lane line images can be removed, reducing processing time and improving the accuracy of roll angle calibration.

[0079] The computer device counts the number of lane line images that meet a first condition in a lane line image set corresponding to each roll angle; wherein the first condition is set based on the spacing between adjacent lane lines in the lane line image; and the roll angle corresponding to the lane line image set whose number meets the second condition is determined as the roll angle of the vehicle-mounted camera.

[0080] In some embodiments, the first condition is that the difference between the maximum and minimum spacing between adjacent lane lines in the lane line image is less than or equal to a threshold value. The second condition is that the number is a maximum. In some embodiments, the threshold value is 30 centimeters. In some embodiments, the number of lane line images in the lane line image set corresponding to each roll angle that meet the requirement that the difference between the maximum and minimum spacing between adjacent lane lines in the lane line image is less than or equal to 30 centimeters is counted. The roll angle corresponding to the lane line image set with the largest number of lane lines is determined as the roll angle of the vehicle-mounted camera. In some embodiments, in the lane line image set corresponding to a roll angle of 1.2, the number of lane line images in the lane line image set that meet the requirement that the difference between the maximum and minimum spacing between adjacent lane lines in the lane line image is less than or equal to 30 centimeters is 120, which is the maximum value. In the lane line image sets corresponding to other roll angles, the number of lane line images in the lane line image set that meet the requirement that the difference between the maximum and minimum spacing between adjacent lane lines in the lane line image is less than or equal to 30 centimeters is less than 120. In this case, 1.2 is determined to be the roll angle of the vehicle-mounted camera.

[0081] refer to Figure 12 , which shows a lane line image corresponding to the roll angle determined according to the method of the present application provided in one embodiment of the present application. Figure 12 As shown, the solid line 121 represents the lane line consistent with the actual road, while the dashed line 122 is the lane line fitted when the determined roll angle is 1.2. It can be seen that there is basically no deviation between the dashed line 122 and the solid line 121.

[0082] refer to Figure 13 , which shows the lane line image corresponding to the roll angle of 1.0 provided by one embodiment of the present application. Figure 13 As shown, the solid line 131 represents the lane line consistent with the actual road, while the dotted line 132 is the lane line fitted when the roll angle is 1.0. It can be seen that the dotted line 132 deviates from the solid line 131, which is slightly larger than Figure 12 Deviation in .

[0083] refer to Figure 14 , which shows the lane line image corresponding to the roll angle of 0.0 provided by one embodiment of the present application. Figure 14 As shown, the solid line 141 represents the lane line consistent with the actual road, while the dashed line 142 is the lane line fitted when the roll angle is an initial roll angle of 0.0. It can be seen that the dashed line 142 deviates greatly from the solid line 141.

[0084] Refer to Table 1, which shows a comparison between the roll angle and the lane line fitting results. When the roll angle is 0.0, the difference between the maximum and minimum spacing between adjacent lane lines in the lane line image is large. When the roll angle is 1.0, the difference between the maximum and minimum spacing between adjacent lane lines in the lane line image is small. When the roll angle is 1.0, the difference between the maximum and minimum spacing between adjacent lane lines in the lane line image is large, indicating that the roll angle affects the difference between the maximum and minimum spacing between adjacent lane lines in the lane line image.

[0085] Table 1 Comparison of roll angle and fitting results

[0086]

[0087] refer to Figure 15 , which shows the relationship between the target roll angle and the number of lane line images provided by an embodiment of the present application. From the figure, the horizontal axis represents the roll angle, and the vertical axis represents the number of lane line images that meet the requirement that the difference between the maximum spacing and the minimum spacing between adjacent lane lines in the lane line image set is less than or equal to 30 cm. 150 is Gaussian distributed as a whole, and the number increases with the increase of the roll angle, and decreases with the increase of the roll angle. A maximum value appears in the middle, and the roll angle corresponding to the maximum value is the roll angle that needs to be determined in the embodiment of the present application, that is, the optimal roll angle. It can be seen that when the roll angle is 1.2, the number of lane line images is the largest, which is 116, so the determined roll angle is 1.2. In the embodiment of the present application, the determined roll angle can be one or multiple that meet the conditions.

[0088] In some embodiments, the roll angle of the onboard camera is applied to the navigation map to determine the lane the vehicle is in, and navigation prompts or autonomous driving control are performed based on the lane the vehicle is in. For example, based on the final roll angle of the onboard camera, the image frames captured by the onboard camera during vehicle driving are projected using IPM to obtain the corresponding lane line image. Then, based on the projected lane line image, the curvature of the corresponding road or lane is matched from the navigation map, and navigation prompts such as turning or autonomous driving control are performed based on this curvature. Exemplarily, the above-mentioned navigation map can be a high-precision map.

[0089] The technical solution provided in the embodiment of the present application uses multiple image frames captured by the vehicle-mounted camera as a reference when determining the roll angle of the vehicle-mounted camera, avoiding the errors existing in considering only a single-frame image, and helping to improve the accuracy of the roll angle finally determined; moreover, the present application adopts the idea of ​​big data statistics, by fitting a plurality of lane line image sets corresponding to different roll angles, and selecting the roll angle corresponding to the lane line image set with the best overall fitting result as the roll angle corresponding to the vehicle-mounted camera finally determined, thereby reducing the error of the fitted lane line and improving the accuracy of the roll angle calibration.

[0090] Furthermore, by removing lane line images that meet the third condition from the lane line image set, a filtered lane line image set is obtained, which speeds up subsequent processing and further improves the accuracy of roll angle calibration. By selecting multiple image groups from the image frame sequence using a sliding window, multiple image frames can be selected at once. The resulting IPM projection and fitting based on these multiple image frames results in even smaller errors.

[0091] Please refer to Figure 16 , which shows a flow chart of a roll angle calibration method for a vehicle-mounted camera provided by another embodiment of the present application. The execution subject of each step of the method can be Figure 4 The computer device 42 in the implementation environment of the solution shown. In the following method embodiment, for ease of description, only the execution subject of each step is introduced as "computer device". The method can include at least one of the following steps (1610-1670).

[0092] Step 1610 , obtaining an image frame sequence captured by the vehicle-mounted camera during vehicle driving, the image frame sequence including n image frames obtained by the vehicle-mounted camera shooting the road at n time stamps, where n is an integer greater than 1.

[0093] Step 1620 : Select multiple image groups from the image frame sequence, each image group including multiple consecutive image frames in the image frame sequence.

[0094] Step 1630: For a target image group among multiple image groups, obtain lane line detection results corresponding to K image frames contained in the target image group. The lane line detection results include lane line detection points obtained by detecting lane lines in the image frames, where K is an integer greater than 1.

[0095] In some embodiments, K is 5. In some embodiments, for a target image group among the multiple image groups, lane line detection results corresponding to the five image frames included in the target image group are obtained. Of course, K can also be other positive integers, which are not limited in this application.

[0096] Step 1640 : Integrate the lane line detection results corresponding to the K image frames included in the target image group to obtain the lane line detection results corresponding to the target image group at the target roll angle.

[0097] In some embodiments, integration is performed, i.e., extracting lane line detection points from K image frames and performing IPM projection to obtain the projected results. The projection results of the K image frames are superimposed together to obtain the lane line detection results corresponding to the target image group at the target roll angle.

[0098] In some embodiments, the computer device processes the i-th image frame among the K image frames included in the target image group using a lane detection model to obtain a lane detection result corresponding to the i-th image frame. The lane detection model is an AI (artificial intelligence) model for extracting lane detection points. In some embodiments, the lane detection model is a LaneNet lane detection model.

[0099] In some embodiments, K is 5, and the computer device uses the LaneNet lane line detection model to process the i-th image frame among the 5 image frames included in the target image group to obtain the lane line detection result corresponding to the i-th image frame, where i is a positive integer less than or equal to 5.

[0100] Step 1650 : Fitting the lane line detection results corresponding to the target image group at the target roll angle to obtain the lane line image corresponding to the target image group at the target roll angle.

[0101] In some embodiments, a first-order function is used to fit the lane line detection results corresponding to the target image group at the target roll angle to obtain a first fitting result, and the fitting error corresponding to the first-order function is determined; a second-order function is used to fit the lane line detection results corresponding to the target image group at the target roll angle to obtain a second fitting result, and the fitting error corresponding to the second-order function is determined; if the fitting error corresponding to the first-order function is smaller than the fitting error corresponding to the second-order function, the first fitting result is determined as the lane line image corresponding to the target image group at the target roll angle; if the fitting error corresponding to the first-order function is larger than the fitting error corresponding to the second-order function, the second fitting result is determined as the lane line image corresponding to the target image group at the target roll angle.

[0102] In some embodiments, as Figure 8 As shown, the first-order function is used to fit the lane line detection results. The first-order function is y i =a0+a1x i , the fitting error corresponding to the first-order function is B, such as Figure 9As shown, a second-order function is used to fit the lane line detection results. The second-order function is: Get the fitting error corresponding to the second-order function C, B is greater than C, and the first-order function is used to fit the lane line detection result, which results in a larger error. Therefore, the second-order function is selected to fit the lane line detection result.

[0103] Step 1660 : Obtain a lane line image set corresponding to the target roll angle based on the lane line images corresponding to the multiple image groups at the target roll angle.

[0104] Step 1670: Determine the roll angle of the vehicle-mounted camera based on the lane line image sets corresponding to different roll angles.

[0105] The technical solution provided in this embodiment uses the lane detection model to process the i-th image frame among the K image frames in the target image group. This allows for faster acquisition of lane detection points, facilitating subsequent IPM projection operations on the lane detection point coordinates and accelerating processing. Furthermore, the lane detection results are fitted using different fitting methods, resulting in smaller errors in the fitting results.

[0106] Please refer to Figure 17 , which shows a flow chart of a roll angle calibration method for a vehicle-mounted camera provided by another embodiment of the present application. The execution subject of each step of the method can be Figure 4 The computer device 42 in the implementation environment of the solution shown. In the following method embodiment, for ease of description, only the execution subject of each step is introduced as "computer device". The method can include at least one of the following steps (1710-1790).

[0107] Step 1710 , obtaining an image frame sequence captured by the vehicle-mounted camera during vehicle travel, the image frame sequence including n image frames captured by the vehicle-mounted camera at n time stamps of the road, where n is an integer greater than 1.

[0108] Step 1720 : Select multiple image groups from the image frame sequence, each image group including multiple consecutive image frames in the image frame sequence.

[0109] Step 1730: For a target image group among multiple image groups, obtain lane line detection results corresponding to K image frames contained in the target image group. The lane line detection results include lane line detection points obtained by detecting lane lines in the image frames, where K is an integer greater than 1.

[0110] Step 1740: For the i-th image frame among the K image frames included in the target image group, based on the pitch angle, yaw angle and target roll angle of the vehicle-mounted camera, the lane line detection point in the i-th image frame is converted from the camera coordinate system corresponding to the vehicle-mounted camera to the body coordinate system corresponding to the vehicle, and the lane line detection result of the i-th image frame in the body coordinate system is obtained, where i is a positive integer less than or equal to K.

[0111] In some embodiments, the computer device obtains the pitch angle and yaw angle of the vehicle-mounted camera; wherein the pitch angle and yaw angle are obtained using IPM projection based on the parallel relationship of the lane lines.

[0112] refer to Figure 18 , which shows a lane line image provided by an embodiment of the present application. Figure 18 Figure (a) is based on a lane line image based on the initial pitch and yaw angles, and is obtained based on lane line detection result 181 for lane line detection points in the image frame. It is not difficult to see that the fitted lane lines 182 are not parallel, as the initial pitch and yaw angles are both 0. By continuously adjusting the pitch and yaw angles of the onboard camera, IPM projection is performed on the lane line detection points in the image frame based on the adjusted pitch and yaw angles, resulting in parallel lane lines fitted from the projected lane line detection result 181. Figure 18 (b) shows a lane image obtained based on the adjusted pitch and yaw angles. When the lane lines 184 fitted based on lane detection result 183 are parallel, the pitch and yaw angles are recorded. The pitch and yaw angles used in this embodiment are those obtained when the fitted lane lines are parallel.

[0113] In some embodiments, the computer device performs IPM projection on the lane line detection point in the i-th image frame based on the pitch angle, yaw angle and target roll angle to obtain the lane line detection result of the i-th image frame in the vehicle body coordinate system.

[0114] In some embodiments, when the fitted lane lines are parallel, the pitch angle is a, the yaw angle is b, the target roll angle is set to c, K is 5, and the lane line detection point in the i-th image frame is projected with IPM based on the pitch angle, yaw angle and target roll angle to obtain the lane line detection result of the i-th image frame in the vehicle body coordinate system, where i is a positive integer less than or equal to 5.

[0115] Step 1750: Perform position correction on the lane line detection result of the i-th image frame in the vehicle body coordinate system based on the posture information of the vehicle at the timestamp corresponding to the i-th image frame to obtain the corrected lane line detection result corresponding to the i-th image frame.

[0116] In some embodiments, the computer device also obtains reference posture information of the vehicle at a reference timestamp, the reference posture information including a reference position and a reference posture of the vehicle at the reference timestamp; determines the position of the vehicle at the timestamp corresponding to the i-th image frame based on the reference position of the vehicle at the reference timestamp, the interval between the timestamp corresponding to the i-th image frame and the reference timestamp, and the speed of the vehicle; determines the posture of the vehicle at the timestamp corresponding to the i-th image frame based on the reference posture of the vehicle at the reference timestamp, the interval between the timestamp corresponding to the i-th image frame and the reference timestamp, and the angular velocity of the vehicle; wherein, the posture information of the vehicle at the timestamp corresponding to the i-th image frame includes: the position and posture of the vehicle at the timestamp corresponding to the i-th image frame.

[0117] In some embodiments, the reference timestamp is the i-1th timestamp. The computer device obtains the position and posture of the vehicle at the i-1th timestamp, and determines the position of the vehicle at the timestamp corresponding to the i-1 image frame based on the position of the vehicle at the i-1th timestamp, the interval between the timestamp corresponding to the i-1th image frame and the i-1th timestamp, and the speed of the vehicle; determines the posture of the vehicle at the timestamp corresponding to the i-1 image frame based on the posture of the vehicle at the i-1th timestamp, the interval between the timestamp corresponding to the i-1th image frame and the i-1th timestamp, and the angular velocity of the vehicle; wherein the posture information of the vehicle at the timestamp corresponding to the i-1 image frame includes: the position and posture of the vehicle at the timestamp corresponding to the i-1 image frame. The posture information of the first timestamp is 0. I is an integer greater than 1 and less than or equal to K.

[0118] In some embodiments, the vehicle's position information can be obtained through velocity and gyroscope pre-integration:

[0119]

[0120] v←v+(R(a m -a b )+g)Δt

[0121]

[0122] Among them, p represents position, v represents speed (from wheel speed), a m represents the measured acceleration, a b Represents the zero bias of the accelerometer, R represents the conversion from the accelerometer coordinate system to the vehicle body coordinate system, and g represents the acceleration of gravity, which is usually q represents the attitude, which is the quaternion array of rotation angles, ω m represents the measured angular velocity, ω brepresents the zero bias of the gyroscope, and Δt represents the time interval between adjacent image frames. Since the integration time is short, that is, the time within a sliding window, acceleration can be ignored; and the bias (zero bias) of the gyroscope is small and can be ignored in a short time, so the pre-integration formula can be rewritten as:

[0123] p←16p+vΔt

[0124]

[0125] Step 1760 : Integrate the corrected lane line detection results corresponding to the K image frames in the vehicle body coordinate system to obtain the lane line detection results corresponding to the target image group at the target roll angle.

[0126] Step 1770 , fitting the lane line detection results corresponding to the target image group at the target roll angle to obtain the lane line image corresponding to the target image group at the target roll angle.

[0127] Step 1780: Obtain a lane line image set corresponding to the target roll angle based on the lane line images corresponding to the multiple image groups at the target roll angle.

[0128] Step 1790: Determine the roll angle of the vehicle-mounted camera based on the lane line image sets corresponding to different roll angles.

[0129] The technical solution provided in this embodiment converts the lane line detection points in the i-th image frame of the K image frames included in the target image group from the camera coordinate system corresponding to the vehicle camera to the vehicle body coordinate system corresponding to the vehicle based on the pitch angle, yaw angle, and target roll angle of the vehicle camera, thereby obtaining the lane line detection result for the i-th image frame in the vehicle body coordinate system. This can accelerate the acquisition of the lane line detection result for the i-th image frame in the vehicle body coordinate system and improve the accuracy of the obtained lane line detection result, facilitating subsequent fitting operations. Simultaneously, the lane line detection result for the i-th image frame in the vehicle body coordinate system is positionally corrected based on the vehicle's pose information at the timestamp corresponding to the i-th image frame, obtaining a corrected lane line detection result for the i-th image frame. This can make the lane line detection result more accurate. Based on the more accurate lane line detection result, the accuracy of roll angle calibration can be further improved.

[0130] Please refer to Figure 19 , which shows a flow chart of a roll angle calibration method for a vehicle-mounted camera provided by another embodiment of the present application. The execution subject of each step of the method can be Figure 4The computer device 42 in the implementation environment of the solution shown. In the following method embodiment, for ease of description, only the execution subject of each step is introduced as "computer device". The method can include at least one of the following steps (1910-1950).

[0131] Step 1910 , obtaining an image frame sequence captured by the vehicle-mounted camera during vehicle travel, the image frame sequence including n image frames obtained by the vehicle-mounted camera photographing the road at n time stamps, where n is an integer greater than 1.

[0132] The computer device obtains, for a target image group from the plurality of image groups, lane line detection results corresponding to K image frames included in the target image group, where the lane line detection results include lane line detection points obtained by detecting lane lines in the image frames, where K is an integer greater than 1; integrates the lane line detection results corresponding to the K image frames included in the target image group to obtain lane line detection results corresponding to the target image group at an initial roll angle; fits the lane line detection results corresponding to the target image group at the initial roll angle to obtain lane line images corresponding to the target image group at the initial roll angle; and obtains a lane line image set corresponding to the initial roll angle based on the lane line images corresponding to the plurality of image groups at the initial roll angle. In some embodiments, the initial roll angle is 0.0.

[0133] Step 1920 : Select multiple image groups from the image frame sequence, each image group including multiple consecutive image frames in the image frame sequence.

[0134] Step 1930 : Acquire lane line images corresponding to the multiple image groups at the initial roll angle, and obtain a lane line image set corresponding to the initial roll angle.

[0135] In some embodiments, the initial roll angle is 0.0, and lane line images corresponding to multiple image groups when the initial roll angle is 0.0 are obtained to obtain a lane line image set corresponding to the initial roll angle of 0.0.

[0136] Step 1940: Adjust the lane line image set corresponding to the initial roll angle according to the target roll angle to obtain the lane line image set corresponding to the target roll angle.

[0137] In some embodiments, the positions of each lane line in the lane line image set corresponding to the initial roll angle are adjusted based on the target roll angle to obtain the lane line image set corresponding to the target roll angle. In some embodiments, the target roll angle is adjusted based on the number of lane line images in the lane line image set corresponding to the initial roll angle that have a difference between the maximum and minimum spacing between adjacent lane lines in the lane line image of less than or equal to 30 centimeters. When the number of lane line images in the lane line image set corresponding to the target roll angle that have a difference between the maximum and minimum spacing between adjacent lane lines in the lane line image of less than or equal to 30 centimeters is greater than the number of lane line images corresponding to the initial roll angle, the target roll angle is further increased. When the number of lane line images in the lane line image set corresponding to the target roll angle that have a difference between the maximum and minimum spacing between adjacent lane lines in the lane line image of less than or equal to 30 centimeters is the largest, the roll angle at that time is determined to be the roll angle of the vehicle-mounted camera.

[0138] Step 1950: Determine the roll angle of the vehicle-mounted camera based on the lane line image sets corresponding to different roll angles.

[0139] The technical solution provided in this embodiment obtains a lane line image set corresponding to the target roll angle by adjusting the lane line image set corresponding to the initial roll angle according to the target roll angle. The roll angle can be calibrated by adjusting the target roll angle, which reduces the data processing amount to a certain extent and saves data processing time.

[0140] Figure 20 An application scenario of an embodiment of the present application is exemplified. Figure 20 It is a high-precision navigation map 2000. The map 2000 shows vehicles 2001, roads 2002, lane lines 2003, etc. The lane lines 2003 are obtained by taking a picture of the current road 2002 and processing it according to the roll angle determined by the method of this application. Generally speaking, navigation maps provide directional guidance to drivers. The identification of signs, entrance complexities, pedestrians, etc. is completed by the driver. Through the method of this application, the roll angle of the vehicle-mounted camera is calibrated, and the lane lines can be accurately described, making the description of the entire road more accurate, clear and comprehensive, and the following can be obtained. Figure 20 High-precision navigation map shown.

[0141] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0142] Please refer to Figure 21, which shows a block diagram of a roll angle calibration device for a vehicle-mounted camera provided by an embodiment of the present application. The device has the function of implementing the above-mentioned method example, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be the computer device described above, or it can be set in a computer device. Figure 21 As shown, the apparatus 2100 may include: a frame sequence acquisition module 2110 , an image selection module 2120 , an image set acquisition module 2130 and a roll angle determination module 2140 .

[0143] The frame sequence acquisition module 2110 is used to obtain an image frame sequence captured by the vehicle-mounted camera during vehicle driving, wherein the image frame sequence includes n image frames obtained by the vehicle-mounted camera shooting the road at n time stamps, where n is an integer greater than 1.

[0144] The image group selection module 2120 is configured to select a plurality of image groups from the image frame sequence, each of the image groups including a plurality of continuous image frames in the image frame sequence.

[0145] The image set acquisition module 2130 is also used to obtain the lane line images corresponding to the multiple image groups at the target roll angle, and obtain the lane line image set corresponding to the target roll angle; wherein, the lane line image corresponding to each of the image groups at the target roll angle includes: at least one lane line obtained by fitting the lane line detection points extracted from the multiple image frames contained in the image group at the target roll angle.

[0146] The roll angle determination module 2140 is configured to determine the roll angle of the vehicle-mounted camera based on lane line image sets corresponding to different roll angles.

[0147] In some embodiments, as Figure 22 As shown, the image set acquisition module 2130 includes: a result acquisition submodule 2132 , a result integration submodule 2134 , a result fitting submodule 2136 and an image set acquisition submodule 2138 .

[0148] The result acquisition submodule 2132 is used to obtain, for a target image group among the multiple image groups, lane line detection results corresponding to the K image frames contained in the target image group, wherein the lane line detection results include lane line detection points obtained by detecting the lane lines in the image frames, and K is an integer greater than 1.

[0149] The result integration submodule 2134 is used to integrate the lane line detection results corresponding to the K image frames included in the target image group, and obtain the lane line detection results corresponding to the target image group at the target roll angle.

[0150] The result fitting submodule 2136 is used to fit the lane line detection results corresponding to the target image group at the target roll angle to obtain the lane line image corresponding to the target image group at the target roll angle.

[0151] The image set acquisition submodule 2138 is configured to obtain a lane line image set corresponding to the target roll angle based on the lane line images corresponding to the multiple image groups at the target roll angle.

[0152] In some embodiments, as Figure 22 As shown, the result integration submodule 2134 includes: a detection point conversion unit S1, a detection result correction unit S2 and a detection result integration unit S3.

[0153] A detection point conversion unit S1 is configured to convert, for an i-th image frame among the K image frames included in the target image group, the lane line detection point in the i-th image frame from the camera coordinate system corresponding to the vehicle-mounted camera to the vehicle body coordinate system corresponding to the vehicle based on the pitch angle, yaw angle, and roll angle of the vehicle-mounted camera, thereby obtaining a lane line detection result for the i-th image frame in the vehicle body coordinate system, where i is a positive integer less than or equal to K.

[0154] The detection result correction unit S2 is used to perform position correction on the lane line detection result of the i-th image frame in the vehicle body coordinate system according to the posture information of the vehicle at the timestamp corresponding to the i-th image frame, so as to obtain the corrected lane line detection result corresponding to the i-th image frame.

[0155] The detection result integration unit S3 is used to integrate the corrected lane line detection results corresponding to the K image frames in the vehicle body coordinate system to obtain the lane line detection result corresponding to the target image group at the target roll angle.

[0156] In some embodiments, as Figure 22 As shown, the detection point conversion unit S1 is used to obtain the pitch angle and yaw angle of the vehicle-mounted camera; wherein the pitch angle and yaw angle are obtained by using IPM projection based on the parallel relationship of the lane lines; based on the pitch angle and yaw angle and the target roll angle, the lane line detection point in the i-th image frame is IPM projected to obtain the lane line detection result of the i-th image frame in the vehicle body coordinate system.

[0157] In some embodiments, as Figure 22 As shown, the result integration submodule 2134 further includes a posture information acquisition unit S4, a position determination unit S5 and a posture determination unit S6.

[0158] The posture information acquisition unit S4 is used to acquire reference posture information of the vehicle at a reference time stamp, where the reference posture information includes a reference position and a reference posture of the vehicle at the reference time stamp.

[0159] The position determination unit S5 is used to determine the position of the vehicle at the timestamp corresponding to the i-th image frame based on the reference position of the vehicle at the reference timestamp, the interval between the timestamp corresponding to the i-th image frame and the reference timestamp, and the speed of the vehicle.

[0160] The posture determination unit S6 is used to determine the posture of the vehicle at the timestamp corresponding to the i-th image frame based on the reference posture of the vehicle at the reference timestamp, the interval between the timestamp corresponding to the i-th image frame and the reference timestamp, and the angular velocity of the vehicle.

[0161] In some embodiments, the result fitting submodule 2136 is used to: use a first-order function to fit the lane line detection results corresponding to the target image group at the target roll angle to obtain a first fitting result, and determine the fitting error corresponding to the first-order function; use a second-order function to fit the lane line detection results corresponding to the target image group at the target roll angle to obtain a second fitting result, and determine the fitting error corresponding to the second-order function; if the fitting error corresponding to the first-order function is smaller than the fitting error corresponding to the second-order function, then determine the first fitting result as the lane line image corresponding to the target image group at the target roll angle; if the fitting error corresponding to the first-order function is larger than the fitting error corresponding to the second-order function, then determine the second fitting result as the lane line image corresponding to the target image group at the target roll angle.

[0162] In some embodiments, the result acquisition submodule 2132 is configured to process the i-th image frame among the K image frames included in the target image group using a lane detection model to obtain a lane detection result corresponding to the i-th image frame. The lane detection model is an AI model for extracting lane detection points.

[0163] In some embodiments, as Figure 22 As shown, the image set acquisition module 2130 also includes an image set adjustment submodule 2139 .

[0164] The image set acquisition submodule 2138 is configured to acquire lane line images corresponding to the plurality of image groups at the initial roll angle, and obtain a lane line image set corresponding to the initial roll angle.

[0165] The image set adjustment submodule 2139 is configured to adjust the lane line image set corresponding to the initial roll angle according to the target roll angle to obtain the lane line image set corresponding to the target roll angle.

[0166] In some embodiments, the image set adjustment submodule 2139 is used to adjust the position of each lane line in the lane line image set corresponding to the initial roll angle according to the target roll angle to obtain the lane line image set corresponding to the target roll angle.

[0167] In some embodiments, the result acquisition submodule 2132 is used to obtain, for a target image group among the multiple image groups, lane line detection results corresponding to the K image frames contained in the target image group, wherein the lane line detection results include lane line detection points obtained by detecting the lane lines in the image frames, and K is an integer greater than 1.

[0168] The result integration submodule 2134 is used to integrate the lane line detection results corresponding to the K image frames included in the target image group, and obtain the lane line detection results corresponding to the target image group at the initial roll angle.

[0169] The result fitting submodule 2136 is used to fit the lane line detection results corresponding to the target image group at the initial roll angle to obtain the lane line image corresponding to the target image group at the initial roll angle.

[0170] The image set acquisition submodule 2138 is configured to obtain a lane line image set corresponding to the initial roll angle based on the lane line images corresponding to the multiple image groups at the initial roll angle.

[0171] In some embodiments, the roll angle determination module 2140 is used for: a counting submodule 2142, for counting the number of lane line images that meet the first condition in the lane line image set corresponding to each roll angle; wherein the first condition is set according to the spacing between adjacent lane lines in the lane line image; and the roll angle corresponding to the lane line image set whose number meets the second condition is determined as the roll angle of the vehicle-mounted camera.

[0172] In some embodiments, the first condition includes: a difference between a maximum spacing and a minimum spacing between adjacent lane lines in the lane line image is less than or equal to a threshold value.

[0173] In some embodiments, the second condition includes: the number is a maximum value.

[0174] In some embodiments, the image group selection module 2120 is used to move the sliding window to multiple different positions in the image frame sequence to obtain the multiple image groups; wherein, the multiple continuous image frames contained in the sliding window at each position correspond to an image group.

[0175] In some embodiments, as Figure 22 As shown, the device 2100 further includes an image rejection module 2150 .

[0176] The image removal module 2150 is used to remove lane line images that meet a third condition from the lane line image set to obtain a filtered lane line image set; wherein the third condition includes at least one of the following: the lane line fitting error is greater than or equal to a first threshold, and the number of lane lines included is less than or equal to a second threshold.

[0177] The roll angle determination module 2140 is further configured to determine the roll angle of the vehicle-mounted camera based on the filtered lane line image sets corresponding to the different roll angles.

[0178] In some embodiments, the apparatus 2100 further includes a post-processing module ( Figure 22 (not shown) for applying the roll angle of the vehicle-mounted camera to a navigation map to determine the lane in which the vehicle is located; and performing navigation prompts or automatic driving control according to the lane in which the vehicle is located.

[0179] The technical solution provided in the embodiment of the present application uses multiple image frames captured by the vehicle-mounted camera as a reference when determining the roll angle of the vehicle-mounted camera, avoiding the errors existing in considering only a single-frame image, and helping to improve the accuracy of the roll angle finally determined; moreover, the present application adopts the idea of ​​big data statistics, by fitting a plurality of lane line image sets corresponding to different roll angles, and selecting the roll angle corresponding to the lane line image set with the best overall fitting result as the roll angle corresponding to the vehicle-mounted camera finally determined, thereby reducing the error of the fitted lane line and improving the accuracy of the roll angle calibration.

[0180] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0181] Please refer to Figure 23, which shows a block diagram of the structure of a computer device 2300 provided in one embodiment of the present application. Specifically, the computer device 2300 includes a central processing unit (CPU) 2301, a system memory 2304 including a random access memory (RAM) 2302 and a read-only memory (ROM) 2303, and a system bus 2305 connecting the system memory 2304 and the CPU 2301. The computer device 2300 also includes a basic input / output system (I / O system) 2306 that helps transfer information between various components within the computer, and a mass storage device 2307 for storing an operating system 2313, application programs 2314, and other program modules 2315.

[0182] The basic input / output system 2306 includes a display 2308 for displaying information and input devices 2309, such as a mouse and keyboard, for user account input. Both the display 2308 and the input devices 2309 are connected to the central processing unit 2301 via an input / output controller 2310 connected to the system bus 2305. The basic input / output system 2306 may also include an input / output controller 2310 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 2310 also provides output to a display screen, printer, or other types of output devices.

[0183] The mass storage device 2307 is connected to the central processing unit 2301 via a mass storage controller (not shown) connected to the system bus 2305. The mass storage device 2307 and its associated computer-readable media provide non-volatile storage for the computer device 2300. In other words, the mass storage device 1007 may include computer-readable media (not shown) such as a hard disk or a CD-ROM drive.

[0184] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the aforementioned types. The above-mentioned system memory 2304 and mass storage device 2307 may be collectively referred to as memory.

[0185] According to various embodiments of the present application, the computer device 2300 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 2300 may be connected to the network 2312 via the network interface unit 2311 connected to the system bus 2305, or the network interface unit 2311 may be used to connect to other types of networks or remote computer systems (not shown).

[0186] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium. When the computer program is executed by a processor, the roll angle calibration method of the vehicle-mounted camera is implemented.

[0187] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0188] In an exemplary embodiment, a computer program product is also provided. The computer program product includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described roll angle calibration method for a vehicle-mounted camera.

[0189] It should be understood that the "K" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0190] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A roll angle calibration method for a vehicle-mounted camera, characterized in that: The method comprises: Obtaining a sequence of image frames captured by the vehicle-mounted camera during vehicle travel, the sequence of image frames comprising n image frames obtained by the vehicle-mounted camera photographing the road at n time stamps, where n is an integer greater than 1; Selecting a plurality of image groups from the image frame sequence, each of the image groups comprising a plurality of consecutive image frames in the image frame sequence; Acquiring lane line images corresponding to the plurality of image groups at a target roll angle, respectively, to obtain a lane line image set corresponding to the target roll angle; wherein the lane line image corresponding to each of the image groups at the target roll angle includes: at least one lane line obtained by fitting lane line detection points extracted from a plurality of image frames included in the image group at the target roll angle; The roll angle of the vehicle-mounted camera is determined according to lane line image sets corresponding to different roll angles.

2. The method according to claim 1, characterized in that The acquiring of lane line images corresponding to the plurality of image groups at the target roll angle to obtain a lane line image set corresponding to the target roll angle includes: For a target image group among the multiple image groups, obtaining lane line detection results corresponding to K image frames included in the target image group, wherein the lane line detection results include lane line detection points obtained by detecting lane lines in the image frames, where K is an integer greater than 1; Integrating lane line detection results corresponding to the K image frames included in the target image group to obtain a lane line detection result corresponding to the target image group at the target roll angle; Fitting the lane line detection results corresponding to the target image group at the target roll angle to obtain a lane line image corresponding to the target image group at the target roll angle; A lane line image set corresponding to the target roll angle is obtained based on the lane line images respectively corresponding to the multiple image groups at the target roll angle.

3. The method according to claim 2, characterized in that The integrating the lane line detection results corresponding to the K image frames included in the target image group to obtain the lane line detection result corresponding to the target image group at the target roll angle includes: For an i-th image frame among the K image frames included in the target image group, based on the pitch angle and yaw angle of the vehicle-mounted camera and the target roll angle, convert the lane line detection point in the i-th image frame from the camera coordinate system corresponding to the vehicle-mounted camera to the body coordinate system corresponding to the vehicle, thereby obtaining a lane line detection result of the i-th image frame in the body coordinate system, where i is a positive integer less than or equal to K; Performing position correction on the lane line detection result of the i-th image frame in the vehicle body coordinate system according to the posture information of the vehicle at the timestamp corresponding to the i-th image frame, to obtain a corrected lane line detection result corresponding to the i-th image frame; The corrected lane line detection results corresponding to the K image frames are integrated in the vehicle body coordinate system to obtain the lane line detection result corresponding to the target image group at the target roll angle.

4. The method according to claim 3, characterized in that The converting, based on the pitch angle, yaw angle, and roll angle of the vehicle-mounted camera, the lane line detection point in the i-th image frame from the camera coordinate system corresponding to the vehicle-mounted camera to the vehicle body coordinate system corresponding to the vehicle, and obtaining the lane line detection result of the i-th image frame in the vehicle body coordinate system, includes: Obtaining the pitch angle and yaw angle of the vehicle-mounted camera; wherein the pitch angle and yaw angle are obtained by using inverse perspective transformation (IPM) projection according to the parallel relationship of the lane lines; According to the pitch angle, yaw angle and target roll angle, IPM projection is performed on the lane line detection point in the i-th image frame to obtain the lane line detection result of the i-th image frame in the vehicle body coordinate system.

5. The method according to claim 3, characterized in that The method further comprises: Acquire reference posture information of the vehicle at a reference time stamp, the reference posture information including a reference position and a reference posture of the vehicle at the reference time stamp; Determining the position of the vehicle at the timestamp corresponding to the i-th image frame according to the reference position of the vehicle at the reference timestamp, the interval between the timestamp corresponding to the i-th image frame and the reference timestamp, and the speed of the vehicle; Determining a posture of the vehicle at the timestamp corresponding to the i-th image frame based on a reference posture of the vehicle at the reference timestamp, an interval between the timestamp corresponding to the i-th image frame and the reference timestamp, and an angular velocity of the vehicle; The position and posture information of the vehicle at the timestamp corresponding to the i-th image frame includes: the position and posture of the vehicle at the timestamp corresponding to the i-th image frame.

6. The method according to claim 2, characterized in that The fitting of the lane line detection results corresponding to the target image group at the target roll angle to obtain the lane line images corresponding to the target image group at the target roll angle includes: Fitting the lane line detection results corresponding to the target image group at the target roll angle using a first-order function to obtain a first fitting result, and determining a fitting error corresponding to the first-order function; Fitting the lane line detection results corresponding to the target image group at the target roll angle using a second-order function to obtain a second fitting result, and determining a fitting error corresponding to the second-order function; If the fitting error corresponding to the first-order function is smaller than the fitting error corresponding to the second-order function, determining the first fitting result as the lane line image corresponding to the target image group at the target roll angle; If the fitting error corresponding to the first-order function is greater than the fitting error corresponding to the second-order function, the second fitting result is determined as the lane line image corresponding to the target image group at the target roll angle.

7. The method according to claim 2, characterized in that The obtaining lane line detection results corresponding to the K image frames included in the target image group includes: For an i-th image frame among the K image frames included in the target image group, the lane line detection model is used to process the i-th image frame to obtain a lane line detection result corresponding to the i-th image frame; The lane line detection model is an artificial intelligence (AI) model used to extract lane line detection points.

8. The method according to claim 1, characterized in that The acquiring of lane line images corresponding to the plurality of image groups at the target roll angle to obtain a lane line image set corresponding to the target roll angle includes: Acquire lane line images corresponding to the plurality of image groups at the initial roll angle, to obtain a lane line image set corresponding to the initial roll angle; The lane line image set corresponding to the initial roll angle is adjusted according to the target roll angle to obtain the lane line image set corresponding to the target roll angle.

9. The method according to claim 8, characterized in that The adjusting the lane line image set corresponding to the initial roll angle according to the target roll angle to obtain the lane line image set corresponding to the target roll angle includes: The position of each lane line in the lane line image set corresponding to the initial roll angle is adjusted according to the target roll angle to obtain the lane line image set corresponding to the target roll angle.

10. The method according to claim 8, characterized in that The acquiring of lane line images corresponding to the plurality of image groups at the initial roll angles to obtain a lane line image set corresponding to the initial roll angles includes: For a target image group among the multiple image groups, obtaining lane line detection results corresponding to K image frames included in the target image group, wherein the lane line detection results include lane line detection points obtained by detecting lane lines in the image frames, where K is an integer greater than 1; Integrating lane line detection results corresponding to the K image frames included in the target image group to obtain a lane line detection result corresponding to the target image group at the initial roll angle; Fitting the lane line detection results corresponding to the target image group at the initial roll angle to obtain a lane line image corresponding to the target image group at the initial roll angle; A lane line image set corresponding to the initial roll angle is obtained based on the lane line images corresponding to the multiple image groups at the initial roll angle.

11. The method according to claim 1, wherein The determining the roll angle of the vehicle-mounted camera according to lane line image sets corresponding to different roll angles includes: Counting the number of lane line images that meet a first condition in the lane line image set corresponding to each roll angle; wherein the first condition is set according to the distance between adjacent lane lines in the lane line image; The roll angles corresponding to the lane line image sets that meet the second condition are determined as the roll angles of the vehicle-mounted camera.

12. The method according to claim 11, characterized in that The first condition includes: a difference between a maximum distance and a minimum distance between adjacent lane lines in the lane line image is less than or equal to a threshold value.

13. The method according to claim 11, characterized in that The second condition includes: the number is a maximum value.

14. The method according to claim 1, wherein The selecting a plurality of image groups from the image frame sequence comprises: The sliding window is moved to a plurality of different positions in the image frame sequence to obtain the plurality of image groups; wherein the plurality of continuous image frames contained in the sliding window at each position corresponds to one image group.

15. The method according to claim 1, wherein After acquiring lane line images corresponding to the plurality of image groups at the target roll angle, and obtaining a lane line image set corresponding to the target roll angle, the method further includes: Eliminating lane line images that meet a third condition from the lane line image set to obtain a filtered lane line image set; wherein the third condition includes at least one of the following: a lane line fitting error is greater than or equal to a first threshold, and the number of lane lines included is less than or equal to a second threshold; The determining the roll angle of the vehicle-mounted camera according to lane line image sets corresponding to different roll angles includes: The roll angle of the vehicle-mounted camera is determined according to the filtered lane line image sets corresponding to the different roll angles.

16. The method according to any one of claims 1 to 15, characterized in that The determining the roll angle of the vehicle-mounted camera according to lane line image sets corresponding to different roll angles further includes: Applying the roll angle of the vehicle-mounted camera to a navigation map to determine the lane in which the vehicle is located; Navigation prompts or automatic driving control are provided according to the lane in which the vehicle is located.

17. A roll angle calibration device for a vehicle-mounted camera, characterized in that: The device comprises: a frame sequence acquisition module, configured to acquire a sequence of image frames captured by the vehicle-mounted camera during vehicle travel, wherein the sequence of image frames includes n image frames obtained by the vehicle-mounted camera photographing the road at n time stamps, where n is an integer greater than 1; An image group selection module, configured to select a plurality of image groups from the image frame sequence, each of the image groups comprising a plurality of consecutive image frames in the image frame sequence; an image set acquisition module, configured to acquire lane line images corresponding to the plurality of image groups at a target roll angle, thereby obtaining a lane line image set corresponding to the target roll angle; wherein the lane line image corresponding to each of the image groups at the target roll angle includes at least one lane line obtained by fitting lane line detection points extracted from the plurality of image frames included in the image group at the target roll angle; The roll angle determination module is used to determine the roll angle of the vehicle-mounted camera based on lane line image sets corresponding to different roll angles.

18. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 16.

20. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the method according to any one of claims 1 to 16.

Images