A calibration method, apparatus, electronic device and storage medium

Abstract

This application discloses a calibration method, apparatus, electronic device, and storage medium. Based on the extrinsic parameters of a visual sensor at the current moment, this application acquires a current frame image using the visual sensor; establishes a coordinate system set corresponding to the current frame image and determines the transformation relationships between the coordinate systems in the set; updates the extrinsic parameters of the visual sensor at the current moment according to the transformation relationships; obtains an extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current moment and historical moments; determines calibration values ​​based on the extrinsic parameter dataset, where the calibration values ​​are the most frequently occurring difference values; the difference values ​​are the different values ​​corresponding to the extrinsic parameters of the visual sensor; and assigns the calibration values ​​to the extrinsic parameters of the visual sensor to complete the extrinsic parameter calibration of the visual sensor. In this application, the extrinsic parameters of the visual sensor can be calibrated accurately and quickly. Therefore, this solution improves the accuracy and robustness of visual sensor extrinsic parameter calibration.

Description

Technical Field

[0001] This application relates to the field of computer vision, specifically to a calibration method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, intelligent driver assistance systems have been increasingly used to improve traffic information safety. Obtaining information about the three-dimensional real world through two-dimensional image information captured by vehicle vision sensors has become the foundation for the realization of automotive driving safety assistance systems. Visual sensor parameter calibration is one of the key technologies in computer vision systems. That is, when a car equipped with a vehicle vision sensor is driving, the external parameters of the vision sensor may change due to reasons such as turning deviation, excessive braking, road bumps, and vehicle collisions, and then recalibration is required.

[0003] However, in practical applications, existing calibration methods generally process images taken while driving in a straight line to calibrate the extrinsic parameters of the vision sensor. This results in low calibration accuracy and poor robustness. Therefore, there is an urgent need for a calibration method that can solve the problems of low calibration accuracy and poor robustness of existing calibration methods. Summary of the Invention

[0004] This application provides a calibration method, apparatus, electronic device, and storage medium that can improve the accuracy and robustness of visual sensor extrinsic parameter calibration.

[0005] This application provides a calibration method, including:

[0006] Based on the extrinsic parameters of the visual sensor at the current moment, the current frame image is acquired through the visual sensor.

[0007] Establish a set of coordinate systems corresponding to the current frame image, and determine the transformation relationships between the coordinate systems in the set;

[0008] Update the extrinsic parameters of the vision sensor at the current moment based on the transformation relationship;

[0009] Obtain the extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current time and historical time. The historical time includes the time before the current time.

[0010] The calibration values ​​are determined based on the extrinsic parameter dataset. The calibration values ​​are the most frequently occurring difference values; the difference values ​​are the different values ​​corresponding to the extrinsic parameters of the vision sensor.

[0011] The calibration values ​​are assigned to the extrinsic parameters of the vision sensor to complete the extrinsic parameter calibration of the vision sensor.

[0012] This application also provides a calibration device, including:

[0013] An image acquisition unit is used to acquire the current frame image through the visual sensor based on the extrinsic parameters of the visual sensor at the current moment.

[0014] A coordinate system establishment unit is used to establish a set of coordinate systems corresponding to the current frame image and determine the transformation relationship between the coordinate systems in the set of coordinate systems;

[0015] The extrinsic parameter update unit is used to update the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship;

[0016] A dataset acquisition unit is used to acquire an extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current time and at historical times, and the historical times include times before the current time.

[0017] The calibration value determination unit is used to determine calibration values ​​based on the extrinsic parameter dataset, wherein the calibration values ​​are the most frequently occurring difference values; and the difference values ​​are the different values ​​corresponding to the extrinsic parameters of the visual sensor.

[0018] The extrinsic parameter calibration completion unit is used to assign the calibration value to the extrinsic parameter of the vision sensor to complete the extrinsic parameter calibration of the vision sensor.

[0019] In some embodiments, the extrinsic parameter update unit includes an extrinsic parameter update subunit, which is used for...

[0020] The lane lines in the current frame image are identified by feature points to obtain a labeled image of lane line feature points, which includes lane line feature points.

[0021] Obtain the transformation relationship between the pixel coordinate system and the world coordinate system in the coordinate system set;

[0022] Based on the transformation relationship, the marked image in the pixel coordinate system is transformed to the first preset plane in the world coordinate system to obtain the lane line feature point transformation map, which includes the transformed lane line feature points.

[0023] The transformed lane line feature points in the lane line feature point transformation map are filtered to obtain the first feature point filtering map, which includes the filtered feature points.

[0024] The lane line corresponding to the current driving lane is obtained by fitting the selected feature points.

[0025] When the lane lines corresponding to the current driving lane are parallel to each other, calculate the lane line angle between the lane line corresponding to the current driving lane and the first coordinate axis of the first feature point screening map.

[0026] The lane line angle is used as the yaw angle of the visual sensor at the current moment.

[0027] In some embodiments, the extrinsic parameter update subunit includes a lane line fitting unit, which is used for:

[0028] Determine the optical axis coordinates of the visual sensor corresponding to the first feature point screening map, where the optical axis coordinates include the second coordinates;

[0029] The first feature point screening map is divided into regions based on the second coordinates;

[0030] Determine the first and second feature points corresponding to each lane line in the region;

[0031] Obtain the length between the first feature point and the second feature point;

[0032] Connect the lane lines of the first feature point and the second feature point into a straight line;

[0033] Determine the slope of the straight line in the first feature point selection graph;

[0034] When the length between the first feature point and the second feature point is greater than a preset length, and the slope of the straight line in the first feature point screening map is greater than a preset slope, the lane line corresponding to the region is obtained; the lane line corresponding to the region includes the first feature point and the second feature point;

[0035] Use the lane lines corresponding to the area as the lane lines corresponding to the current driving lane.

[0036] In some embodiments, the extrinsic parameter update subunit further includes a pitch angle update unit, which is used for:

[0037] When the lane lines corresponding to the current driving lane are not parallel to each other, obtain the first intersection point between the lane lines corresponding to the current driving lane.

[0038] Determine the origin of the first feature point selection map;

[0039] Determine the first distance between the first intersection point and the origin of the first feature point screening map;

[0040] Set the pitch angle step size based on the first distance;

[0041] Adjust the pitch angle in the extrinsic parameters of the vision sensor at the current moment according to the pitch angle step size to obtain the adjusted pitch angle of the vision sensor;

[0042] The adjusted visual sensor pitch angle is used as the visual sensor pitch angle calibrated at the current moment.

[0043] In some embodiments, the pitch angle update unit further includes a pitch angle update subunit, which is used for:

[0044] Based on the adjusted pitch angle of the visual sensor, a new first image is acquired at the next moment through the visual sensor;

[0045] Determine the second feature point selection map corresponding to the first image;

[0046] Determine the origin of the second feature point selection map;

[0047] Obtain the lane line corresponding to the current driving lane in the second feature point filtering image;

[0048] When the lane lines corresponding to the current driving lane in the second feature point filtering image are not parallel to each other, determine the second intersection point between the lane lines corresponding to the current driving lane in the second feature point filtering image.

[0049] Obtain the second distance between the second intersection point and the origin of the second feature point filtering map;

[0050] When the sign of the second distance is opposite to the sign of the first distance, the absolute values ​​of the first and second distances are compared to obtain the maximum distance; the signs of the first and second distances correspond to the directions of the first coordinate axis, respectively.

[0051] The pitch angle of the visual sensor corresponding to the maximum distance is used as the pitch angle of the visual sensor calibrated at the next moment.

[0052] In some embodiments, the extrinsic parameter update unit further includes a height determination unit, the height determination unit being configured to:

[0053] The yaw angle of the visual sensor is corrected based on the yaw angle of the visual sensor calibrated at the current time, and the corrected yaw angle of the visual sensor is obtained.

[0054] A new second image is acquired using the vision sensor based on the corrected yaw angle.

[0055] Determine the third feature point selection map corresponding to the second image;

[0056] Obtain the lane line corresponding to the current driving lane in the third feature point filtering image;

[0057] The lane lines corresponding to the current driving lane in the third feature point filtering image are respectively denoted as the first lane line and the second lane line.

[0058] In the third feature point filtering map, determine the first intercept between the first lane line and the first coordinate axis, and the second intercept between the second lane line and the first coordinate axis;

[0059] The height of the vision sensor is determined based on the first intercept and the second intercept.

[0060] In some embodiments, the height determining unit includes a height determining subunit:

[0061] Calculate the difference between the first intercept and the second intercept, and use the difference as the lane width measurement value;

[0062] Obtain the preset value of lane line width, and use the ratio between the measured value of lane line width and the preset value of lane line width as the height weight of the visual sensor.

[0063] Obtain the preset height value of the vision sensor;

[0064] The visual sensor height is obtained by multiplying the visual sensor height weight by the preset visual sensor height value.

[0065] This application also provides an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute steps in any of the calibration methods provided in this application.

[0066] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the calibration methods provided in this application.

[0067] This application embodiment can acquire the current frame image by the visual sensor based on the extrinsic parameters of the visual sensor at the current moment; establish a coordinate system set corresponding to the current frame image and determine the transformation relationship between the coordinate systems in the coordinate system set; update the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship; obtain the extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current moment and historical moments, and the historical moments include moments before the current moment; determine the calibration value according to the extrinsic parameter dataset, the calibration value is the difference value that appears most frequently; the difference value is the different value corresponding to the extrinsic parameters of the visual sensor; and assign the calibration value to the extrinsic parameters of the visual sensor to complete the extrinsic parameter calibration of the visual sensor.

[0068] In this application, a set of coordinate systems can be established, and the extrinsic parameters of the visual sensor corresponding to the current frame image can be calibrated according to the transformation relationships between the coordinate systems in the set. Simultaneously, the height of the visual sensor can be calculated. This application calibrates the extrinsic parameters of the visual sensor for each frame image, then compiles the calibrated extrinsic parameters of all frames into a visual sensor extrinsic parameter dataset, and statistically analyzes the frequency proportion of different values ​​corresponding to each type of visual sensor extrinsic parameter, thereby determining the final calibrated visual sensor extrinsic parameters. This improves the accuracy and robustness of the visual sensor extrinsic parameter calibration. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1a This is a schematic diagram of a scenario for the calibration method provided in the embodiments of this application;

[0071] Figure 1b This is a schematic flowchart of the calibration method provided in the embodiments of this application;

[0072] Figure 1c This is a schematic diagram of the process for updating the extrinsic parameters of a vision sensor provided in an embodiment of this application;

[0073] Figure 2a This is a schematic diagram illustrating the application of the calibration method provided in this embodiment to a server scenario;

[0074] Figure 2b This is a schematic diagram of the structure of the first feature point screening map provided in the embodiments of this application;

[0075] Figure 2c This is a schematic diagram of the region division structure provided in an embodiment of this application;

[0076] Figure 2d This is a schematic diagram of the lane line structure corresponding to the area provided in the embodiments of this application;

[0077] Figure 2e This is a schematic diagram of the parallel structure of lane lines provided in an embodiment of this application;

[0078] Figure 2f This is a schematic diagram of the lane line structure after yaw angle correction provided in the embodiments of this application;

[0079] Figure 3 This is a schematic diagram of the first structure of the calibration device provided in the embodiments of this application;

[0080] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0081] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0082] This application provides a calibration method, apparatus, electronic device, and storage medium.

[0083] Specifically, the calibration device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0084] In some embodiments, the calibration device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the calibration method of this application.

[0085] In some embodiments, the server may also be implemented as a terminal.

[0086] For example, refer to Figure 1a The electronic device may include a server 10, a storage terminal 11, and a vision sensor 12, etc. The storage terminal 11 stores images, and the server 10, the storage terminal 11, and the vision sensor 12 communicate with each other, which will not be described in detail here.

[0087] Server 10 may include a processor and memory. Server 10 can acquire the current frame image through the visual sensor based on the extrinsic parameters of the visual sensor at the current moment; establish a coordinate system set corresponding to the current frame image and determine the transformation relationship between the coordinate systems in the coordinate system set; update the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship; obtain an extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current moment and historical moments, with historical moments including moments before the current moment; determine calibration values ​​based on the extrinsic parameter dataset, where the calibration values ​​are the most frequently occurring difference values; the difference values ​​are the different values ​​corresponding to the extrinsic parameters of the visual sensor; and assign the calibration values ​​to the extrinsic parameters of the visual sensor to complete the extrinsic parameter calibration of the visual sensor.

[0088] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0089] Artificial intelligence (AI) is a technology that uses digital computers to simulate human perception of the environment, acquisition of knowledge, and use of that knowledge. This technology can enable machines to possess functions similar to human perception, reasoning, and decision-making. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0090] Computer vision (CV) is a technology that uses computers to perform operations such as recognition, measurement, and further processing of target images, replacing the human eye. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition. For example, image processing techniques include image colorization and image outline extraction.

[0091] Key technologies in speech technology include automatic speech recognition, speech synthesis, and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech being one of the most promising methods.

[0092] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0093] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0094] Autonomous driving technology typically includes high-precision mapping, environmental perception, behavior decision-making, path planning, and motion control. Autonomous driving technology has broad application prospects.

[0095] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0096] In this embodiment, a computer vision-based calibration method involving artificial intelligence is provided, such as... Figure 1b As shown, the specific process of this calibration method can be as follows:

[0097] 100. Based on the extrinsic parameters of the visual sensor at the current moment, acquire the current frame image through the visual sensor.

[0098] A visual sensor is an instrument that uses optical elements and imaging devices to acquire image information of the external environment. For example, a visual sensor can be a lens, a camera, an infrared sensor, etc.

[0099] The current moment refers to the moment when the image is captured in real time during the process of using a visual sensor to capture an image. The corresponding concepts are historical moment or next moment. Historical moment refers to all moments before the current moment; next moment refers to moments before the current moment.

[0100] Extrinsics are the parameters of a vision sensor in the world coordinate system, such as the sensor's position and rotation direction. The corresponding concept is intrinsics, which are parameters related to the characteristics of the vision sensor itself, such as the sensor's focal length and pixel size.

[0101] In this embodiment, the extrinsic parameters of the visual sensor include the pitch angle and yaw angle. This embodiment first acquires a captured image based on the extrinsic parameters of the visual sensor at the current moment, and then uses the captured image as the current frame image.

[0102] The current frame image is the image captured at the current moment during the image capture process using a visual sensor.

[0103] In this scheme, the current frame image can be acquired by the visual sensor based on the extrinsic parameters of the visual sensor at the current moment. For example, how can the image be acquired based on the extrinsic parameters?

[0104] 110. Establish the set of coordinate systems corresponding to the current frame image, and determine the transformation relationship between the coordinate systems in the set.

[0105] The coordinate system set can include a variety of different coordinate systems. The current frame image can correspond to a variety of different coordinate systems, i.e., the coordinate system set. In the embodiments of this application, the coordinate system set corresponding to the current frame image can include the world coordinate system, the visual sensor coordinate system, the pixel coordinate system, and the imaging plane coordinate system.

[0106] The world coordinate system is the absolute coordinate system of the objective three-dimensional world, used to describe the absolute position of objects in space. In this embodiment, it is used to describe the reference position of the vision sensor. The vision sensor is mounted on a vehicle, and the X-axis of the world coordinate system... w O w Y w The plane lies on a horizontal ground surface, with the origin O of the world coordinate system. w Located on the central axis of the vision sensor, perpendicular to X w O w Y w The location of the intersection of the planes, X in the world coordinate system. w The axis represents the direction of travel for a vehicle.

[0107] The pixel coordinate system is a two-dimensional coordinate system established using the image captured by the vision sensor as the plane. The image acquired by the vision sensor can be stored as an array in the computer, and the value of each element (pixel) in the array is the brightness (grayscale) of the image point. A Cartesian coordinate system uv is defined on the image, and the coordinates (u, v) of each pixel are the column number and row number of that pixel in the array, respectively. Therefore, (u, v) are pixel coordinates in units of pixels, and the origin of the pixel coordinate system is defined at the top left corner of the image.

[0108] Since the pixel coordinate system represents the column and row number of a pixel in the captured image, but does not express the physical position of the pixel in the image using physical units, it is necessary to establish an imaging plane coordinate system xy, which expresses the position in physical units (e.g., centimeters). (x, y) represents the coordinates of the imaging plane coordinate system measured in physical units. In the xy coordinate system, i.e., the imaging plane coordinate system, the origin is defined at the intersection of the optical axis of the vision sensor and the image plane.

[0109] O, the origin of the visual sensor coordinate system c X is the optical center of the visual sensor. c axis and Y c The Z axis is parallel to the x-axis and y-axis of the imaging plane coordinate system. c The axis is the optical axis of the vision sensor, Z. c The optical axis is perpendicular to the image plane. The intersection of the optical axis and the image plane is the principal point O′ of the image, which is located from the origin O. c With X c Axis, Y c Axis and Z c The three-dimensional coordinate system composed of axes is called the vision sensor coordinate system. c O′ represents the focal length f of the vision sensor.

[0110] The transformation relationship between coordinate systems refers to the transformation relationship between one coordinate system and another. For example, transforming from the visual sensor coordinate system to the world coordinate system involves rotation and translation processes.

[0111] In this embodiment of the application, determining the transformation relationship between coordinate systems in the coordinate system set includes:

[0112] Get the pixel coordinates (u, v) of the pixel in the current frame image;

[0113] Set the initial height h of the vision sensor and the Z-axis of the pixels in the current frame image in the vision sensor coordinate system. c The distance d along the axis;

[0114] The initial height translation vector is obtained based on the initial height value h from the visual sensor.

[0115] In this embodiment, the transformation from the visual sensor coordinate system to the world coordinate system involves a translation process. Since the origin O of the world coordinate system... w Located on the central axis of the vision sensor, perpendicular to X w O w Y w Given the location of the intersection point of the planes, the initial translation vector of the height from the visual sensor coordinate system to the world coordinate system can be determined based on the initial height h of the visual sensor.

[0116] Obtain the intrinsic parameter K of the vision sensor. f x The ratio of the focal length f of the vision sensor to the physical size dx of each pixel along the x-axis is represented by f. y c is the ratio of the focal length f of the vision sensor to the physical size dy of each pixel in the y-axis direction. x c y These are the coordinates of the optical center of the visual sensor about the u-axis and v-axis in the pixel coordinate system, respectively.

[0117] Obtain the rotation matrix R from the visual sensor coordinate system to the world coordinate system, where the rotation matrix R = R x R y , where R x For the visual sensor to orbit X c The first rotation matrix obtained by rotating the axis by the yaw angle is R. y For the visual sensor to orbit around Y c The second rotation matrix is ​​obtained by rotating the axis by the pitch angle.

[0118] In this embodiment, the transformation from the visual sensor coordinate system to the world coordinate system involves a rotation process, rotating around different coordinate axes by different angles to obtain corresponding rotation matrices. The first rotation matrix... Second rotation matrix

[0119]

[0120] By calculating the equation Get the coordinates (X,Y) of the pixel in the current frame image in the world coordinate system.

[0121] 120. Update the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship.

[0122] In one embodiment, such as Figure 1c As shown, the extrinsic parameters of the visual sensor at the current moment are updated according to the transformation relationship, including:

[0123] 1201. Perform feature point recognition on the lane lines in the current frame image to obtain a labeled image of the lane line feature points. The labeled image of the lane line feature points includes the lane line feature points.

[0124] Lane lines, also known as lane dividing lines, are traffic markings set on a level surface to separate traffic flows traveling in the same direction.

[0125] In this embodiment, the feature points corresponding to the lane lines in the current frame image are first identified, and then a marked image with lane line feature points can be obtained.

[0126] 1202. Obtain the transformation relationship between the pixel coordinate system and the world coordinate system in the coordinate system set.

[0127] The coordinates in the coordinate system set include pixel coordinate system, imaging plane coordinate system, vision sensor coordinate system and world coordinate system.

[0128] In the embodiments of this application, the transformation from the pixel coordinate system to the world coordinate system involves four transformation processes: from the pixel coordinate system to the imaging plane coordinate system, from the imaging plane coordinate system to the visual sensor coordinate system, and from the visual sensor coordinate system to the world coordinate system.

[0129] 1203. Based on the transformation relationship, transform the marked image in the pixel coordinate system to the first preset plane in the world coordinate system to obtain the lane line feature point transformation map, which includes the transformed lane line feature points.

[0130] The first preset plane is the X-axis of the world coordinate system. w O w Y w flat.

[0131] In this embodiment of the application, the marked image in the pixel coordinate system can be transformed to the X coordinate system in the world coordinate system through the transformation relationship. w O w Y w The plane can also map the coordinates of lane line feature pixels (i.e., lane line feature points) in the marked image in the pixel coordinate system to the X coordinate system in the world coordinate system. w O w Y w Coordinates of a plane.

[0132] 1204. The transformed lane line feature points in the lane line feature point transformation map are filtered to obtain the first feature point filtering map, which includes the filtered feature points.

[0133] In addition to the converted lane line feature points, the lane line feature point conversion map may also include other non-lane line feature points.

[0134] In this embodiment of the application, other non-lane line feature points are removed from the lane line feature point conversion map, and a first feature point filtering map with the filtered feature points can be obtained.

[0135] 1205. Fit the selected feature points to obtain the lane line corresponding to the current driving lane.

[0136] In this embodiment, the coordinates of the filtered feature points have been determined. To obtain the lane line corresponding to the current driving lane based on the coordinates of the filtered feature points, a fitting operation is still required.

[0137] The embodiments of this application can use the least squares method to fit the selected feature points into lane lines, or other methods such as gradient descent can be used to fit lane lines.

[0138] In one embodiment, the lane line corresponding to the current driving lane is obtained by fitting the selected feature points, including:

[0139] Determine the optical axis coordinates of the visual sensor corresponding to the first feature point screening map. The optical axis coordinates include the second coordinates.

[0140] In this embodiment of the application, an image is captured by a vision sensor, and the optical axis of the vision sensor corresponds to an optical axis point (u) on the captured image. c ,v c Based on the transformation relationship, the image in the pixel coordinate system is transformed to the X coordinate system in the world coordinate system. w O w Y w In a plane, the optical axis point (u) c ,v c There will be a corresponding point in the first feature point selection map, and the optical axis point (u) will be selected. c ,v c The coordinates of the corresponding points in the first feature point selection map are used as the optical axis coordinates.

[0141] The optical axis coordinates include a second coordinate and a first coordinate. The origin O1 is the upper left corner of the first feature point screening map. The first coordinate corresponds to the x1 coordinate axis of the first feature point screening map, and the direction of the x1 coordinate axis is vertically downward. The second coordinate corresponds to the y1 coordinate axis of the first feature point screening map, and the direction of the y1 coordinate axis is horizontally to the right.

[0142] The first feature point screening map is divided into regions based on the second coordinate.

[0143] In this embodiment of the application, the second coordinate is the y1 coordinate value corresponding to the optical axis of the visual sensor in the first feature point screening map.

[0144] In this embodiment, the first feature point screening map can be divided into a first region and a second region according to the y1 coordinate value corresponding to the optical axis of the vision sensor in the first feature point screening map. The first region corresponds to the left region of the first feature point screening map, and the second region corresponds to the right region of the first feature point screening map.

[0145] Determine the first and second feature points corresponding to each lane line in the region.

[0146] In this embodiment of the application, when the vehicle travels in a straight line, each region of the first feature point screening map may have several lane lines, and each lane line contains several feature points, including the first feature point and the second feature point.

[0147] In this embodiment, the feature point farthest from the origin and the feature point closest to the origin in each lane line can be obtained according to the distance of each feature point in each lane line to the origin of the first feature point screening map. The feature point farthest from the origin is taken as the first feature point of the lane line, and the feature point closest to the origin is taken as the second feature point of the lane line.

[0148] Obtain the length between the first feature point and the second feature point.

[0149] In this embodiment of the application, the coordinates of the first feature point and the second feature point in the first feature point screening map can be obtained, and the length between the first feature point and the second feature point can be obtained based on the coordinates of the first feature point and the second feature point in the first feature point screening map.

[0150] Connect the first feature point and the lane line of the second feature point to form a straight line.

[0151] Determine the slope of the straight line in the first feature point screening graph.

[0152] When the length between the first feature point and the second feature point is greater than a preset length and the slope of the straight line in the first feature point screening map is greater than a preset slope, the lane line corresponding to the region is obtained; the lane line corresponding to the region includes the first feature point and the second feature point.

[0153] Use the lane lines corresponding to the area as the lane lines corresponding to the current driving lane.

[0154] In this embodiment, lane lines corresponding to a region can only be obtained through the first feature point and the second feature point when the length between the first feature point and the second feature point is greater than a preset length, and the slope of the straight line connecting the lane lines of the first feature point and the second feature point in the first feature point screening map is greater than a preset slope. This can improve the calibration efficiency of the visual sensor extrinsic parameters and the accuracy of obtaining lane lines corresponding to a region.

[0155] 1206. When the lane lines corresponding to the current driving lane are parallel to each other, calculate the lane line angle between the lane line corresponding to the current driving lane and the first coordinate axis of the first feature point screening map.

[0156] 1207. Use the lane line angle as the yaw angle of the visual sensor calibrated at the current moment.

[0157] In this embodiment, the current driving lane has two lane lines, which may be parallel or non-parallel. If the two lane lines in the current driving lane are parallel, it indicates that the initial visual sensor pitch angle is accurate, and there is no need to update the current visual sensor pitch angle.

[0158] In the embodiments of this application, when calibrating the yaw angle of the visual sensor, it is required that the lane lines corresponding to the current driving lane are parallel to each other as a prerequisite. When the two lane lines corresponding to the current driving lane are parallel to each other, the angle between any one of the two lane lines corresponding to the current driving lane and the x1 coordinate axis of the first feature point screening map is obtained. This angle is recorded as the lane line angle and used as the yaw angle of the visual sensor calibrated at the current time, thus completing the calibration of the yaw angle of the visual sensor at the current time.

[0159] In one embodiment, updating the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship further includes:

[0160] When the lane lines corresponding to the current driving lane are not parallel to each other, obtain the first intersection point between the lane lines corresponding to the current driving lane.

[0161] Determine the origin of the first feature point selection map;

[0162] Determine the first distance between the first intersection point and the origin of the first feature point screening map;

[0163] Set the pitch angle step size based on the first distance;

[0164] Adjust the pitch angle in the extrinsic parameters of the vision sensor at the current moment according to the pitch angle step size to obtain the adjusted pitch angle of the vision sensor;

[0165] The adjusted visual sensor pitch angle is used as the visual sensor pitch angle calibrated at the current moment.

[0166] In this embodiment, the origin of the first feature point filtering map is located at the upper left corner of the first feature point filtering map. The first distance is the distance between the first intersection point and the origin of the first feature point filtering map. The pitch angle step size is the step size when updating the pitch angle in the extrinsic parameters of the vision sensor at the current moment. For example, the pitch angle step size can be set to 0.1 degrees, 0.2 degrees, 0.4 degrees, etc.

[0167] If the lane lines corresponding to the current driving lane are not parallel to each other, the pitch angle of the visual sensor at the current moment needs to be updated. When setting the pitch angle step size based on the first distance, several distance range intervals can be set sequentially according to the distance between the first intersection point and the origin of the first feature point screening map. A different step size can be set for each distance range interval. The larger the distance between the first intersection point and the origin of the first feature point screening map, the smaller the pitch angle step size can be set. For example, several distance range intervals can be set as three distance range intervals: the first interval [0,100), the second interval [100,300), and the third interval [300,+∞). When the distance between the first intersection point and the origin of the first feature point screening map is in the first interval, the pitch angle step size is set to 0.4 degrees; when the distance between the first intersection point and the origin of the first feature point screening map is in the second interval, the pitch angle step size is set to 0.2 degrees; and when the distance between the first intersection point and the origin of the first feature point screening map is in the third interval, the pitch angle step size is set to 0.1 degrees. In this embodiment, the longer the distance between the first intersection point and the origin of the first feature point screening map, the closer the lane lines corresponding to the current driving lane are to parallelism. When the distance is relatively large, it indicates that the angle difference is small, and a small step size is used to approximate it (for example, when the distance is greater than 300 meters, the pitch angle step size is set to 0.1 degrees); when the distance is small, it indicates that the angle difference is large, and a large step size is used to approximate it (for example, when the distance is less than 100 meters, the pitch angle step size is set to 0.4 degrees). This can reduce the number of iterations and improve the efficiency of visual sensor extrinsic parameter calibration.

[0168] In one embodiment, after updating the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship, the method further includes:

[0169] Based on the adjusted pitch angle of the visual sensor, a new first image is acquired at the next moment through the visual sensor;

[0170] Determine the second feature point selection map corresponding to the first image;

[0171] Determine the origin of the second feature point selection map;

[0172] Obtain the lane line corresponding to the current driving lane in the second feature point filtering image;

[0173] When the lane lines corresponding to the current driving lane in the second feature point filtering image are not parallel to each other, determine the second intersection point between the lane lines corresponding to the current driving lane in the second feature point filtering image.

[0174] Obtain the second distance between the second intersection point and the origin of the second feature point filtering map;

[0175] When the sign of the second distance is opposite to the sign of the first distance, the absolute values ​​of the first and second distances are compared to obtain the maximum distance; the signs of the first and second distances correspond to the directions of the first coordinate axis, respectively.

[0176] The pitch angle of the visual sensor corresponding to the maximum distance is used as the pitch angle of the visual sensor calibrated at the next moment.

[0177] In this embodiment, based on the adjusted pitch angle of the visual sensor, a new image is acquired at the next moment using the visual sensor, and this image is recorded as the first image. The origin of the second feature point filtering map is the same as the origin of the first feature point filtering map, both located at the upper left corner of the image. The direction of the coordinate axis of the second feature point filtering map is consistent with the direction of the coordinate axis of the first feature point filtering map, and the first coordinate axis is the x1 coordinate axis of the first feature point filtering map. The second distance and the first distance can represent not only the magnitude of the distance but also the direction. For example, when the sign of the second distance is negative, it indicates that the second intersection point is located in the negative region of the x1 coordinate axis; when the sign of the second distance is positive, it indicates that the second intersection point is located in the positive region of the x1 coordinate axis. When the lane lines corresponding to the current driving lane in the second feature point filtering map are not parallel to each other, the intersection point between the lane lines corresponding to the current driving lane in the second feature point filtering map can be determined, and this intersection point is recorded as the second intersection point. Then, the distance between the second intersection point and the origin of the second feature point filtering map is obtained, and this distance is recorded as the second distance.

[0178] In this embodiment of the application, when determining the second feature point filtering map corresponding to the first image, the first image in the pixel coordinate system is transformed to the first preset plane in the world coordinate system according to the transformation relationship to obtain the first lane line feature point transformation map. Then, the transformed lane line feature points in the first lane line feature point transformation map are filtered to obtain the second feature point filtering map.

[0179] In this embodiment, when the sign of the second distance is opposite to the sign of the first distance, it indicates that the intersection point of the lane line corresponding to the current driving lane in the feature point filtering image corresponding to the image captured at the next moment (i.e., the second intersection point) and the intersection point of the lane line corresponding to the current driving lane in the feature point filtering image corresponding to the image captured at the current moment (i.e., the first intersection point) have been reversed about the x1 coordinate axis. At this time, the pitch angle of the visual sensor that needs to be calibrated can be solved. The pitch angle corresponding to the maximum absolute value of the distance before and after the reversal is taken as the solution result, that is, the pitch angle of the visual sensor corresponding to the maximum distance is taken as the pitch angle of the visual sensor calibrated at the next moment.

[0180] In one embodiment, after using the lane line angle as the visual sensor yaw angle calibrated at the current moment, the method further includes:

[0181] The yaw angle of the visual sensor is corrected based on the yaw angle of the visual sensor calibrated at the current time, and the corrected yaw angle of the visual sensor is obtained.

[0182] A new second image is acquired using the vision sensor based on the corrected yaw angle.

[0183] Determine the third feature point selection map corresponding to the second image;

[0184] Obtain the lane line corresponding to the current driving lane in the third feature point filtering image;

[0185] The lane lines corresponding to the current driving lane in the third feature point filtering image are respectively denoted as the first lane line and the second lane line.

[0186] In the third feature point filtering map, determine the first intercept between the first lane line and the first coordinate axis, and the second intercept between the second lane line and the first coordinate axis;

[0187] The height of the vision sensor is determined based on the first intercept and the second intercept.

[0188] The first coordinate axis is the x1 coordinate axis of the third feature point selection map. The x1 coordinate axis of the third feature point selection map corresponds to the x1 coordinate axis of the second feature point selection map and the x1 coordinate axis of the first feature point selection map, respectively.

[0189] In this embodiment, after determining that the lane lines corresponding to the current driving lane are parallel to each other, the yaw angle of the visual sensor at the current moment can be calibrated. Then, the yaw angle of the visual sensor is corrected according to the calibrated yaw angle of the visual sensor at the current moment, so that the lane line corresponding to the current driving lane can be parallel to the x1 coordinate axis of the third feature point screening map after correction.

[0190] This application embodiment first obtains the intercepts between the two lane lines corresponding to the current driving lane and the x1 coordinate axis of the third feature point screening map, that is, the perpendicular distances between the two lane lines and the x1 coordinate axis of the third feature point screening map. Then, the height of the vision sensor is determined based on the first intercept and the second intercept. This application can not only calibrate the extrinsic parameters of the vision sensor, but also obtain the height information of the vision sensor.

[0191] In one embodiment, determining the height of the vision sensor based on a first intercept and a second intercept includes:

[0192] Calculate the difference between the first intercept and the second intercept, and use the difference as the lane width measurement value;

[0193] Obtain the preset value of lane line width, and use the ratio between the measured value of lane line width and the preset value of lane line width as the height weight of the visual sensor.

[0194] Obtain the preset height value of the vision sensor;

[0195] The visual sensor height is obtained by multiplying the visual sensor height weight by the preset visual sensor height value.

[0196] In this embodiment, the height of the vision sensor can be determined using lane width measurements. (Vision sensor height) H′ is the preset value for camera height, W is the measured value for lane width, and W′ is the preset value for lane width. For example, the preset value for lane width on an urban arterial road is W′ = 3.60.

[0197] 130. Obtain the extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current time and historical time.

[0198] In the embodiments of this application, historical moments include moments before the current moment.

[0199] The embodiments of this application can calibrate different external parameters of the visual sensor at different times. For example, at the previous time, the pitch angle of the visual sensor was calibrated to -2 degrees; at the current time, the pitch angle of the visual sensor was calibrated to -3 degrees and the yaw angle to 0 degrees; at the next time, the pitch angle of the visual sensor was calibrated to -4 degrees and the yaw angle to -2 degrees.

[0200] 140. Determine the calibration values ​​based on the external parameter dataset.

[0201] In this embodiment, the calibration value is the most frequently occurring difference value, and the difference value is the different value corresponding to the extrinsic parameter of the visual sensor.

[0202] This application embodiment captures a large number of frame images during a straight-line journey of the vehicle. Each frame is analyzed, and the extrinsic parameters of the visual sensor are calibrated. The frequency of occurrence of each angle in the visual sensor's extrinsic parameters is statistically analyzed. For example, when statistically analyzing the pitch angle of the visual sensor, -3 degrees appeared 600 times, -4 degrees appeared 10 times, and -2 degrees appeared 30 times. By comparison, the value with the highest frequency is selected, i.e., the pitch angle is calibrated as -3 degrees, and -3 degrees is the corresponding calibration value for the pitch angle. Similarly, when statistically analyzing the yaw angle of the visual sensor, 0 degrees appeared 340 times, -1 degree appeared 60 times, and 2 degrees appeared 70 times. By comparison, the value with the highest frequency is selected, i.e., the yaw angle is calibrated as 0 degrees, and 0 degrees is the corresponding calibration value for the yaw angle.

[0203] 150. Assign the calibration values ​​to the extrinsic parameters of the vision sensor to complete the extrinsic parameter calibration of the vision sensor.

[0204] This application first counts the frequency of occurrence of each angle in the extrinsic parameters of the vision sensor, and then assigns the extrinsic parameter value of the vision sensor with the highest frequency of occurrence to the extrinsic parameter of the final calibrated vision sensor, thereby completing the extrinsic parameter calibration of the vision sensor.

[0205] The calibration scheme provided in this application can be applied to various visual sensor extrinsic parameter calibration scenarios in vehicles. For example, taking a car as an example, it can be used to calibrate the camera on the car. The scheme provided in this application can calibrate the camera's extrinsic parameters more accurately and quickly, further reducing the camera extrinsic parameter calibration error, thereby improving the accuracy and robustness of the camera extrinsic parameter calibration.

[0206] The method provided in this application embodiment can acquire the current frame image by the visual sensor based on the extrinsic parameters of the visual sensor at the current moment; establish a coordinate system set corresponding to the current frame image and determine the transformation relationship between the coordinate systems in the coordinate system set; update the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship; obtain an extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current moment and historical moments; determine calibration values ​​based on the extrinsic parameter dataset, where the calibration values ​​are the most frequently occurring difference values; the difference values ​​are the different values ​​corresponding to the extrinsic parameters of the visual sensor; and assign the calibration values ​​to the extrinsic parameters of the visual sensor to complete the extrinsic parameter calibration of the visual sensor. For example, this application embodiment can calibrate the extrinsic parameters of a camera installed on a car.

[0207] As can be seen from the above, the embodiments of this application can establish a coordinate system set corresponding to the current frame image, and calibrate the extrinsic parameters of the visual sensor corresponding to the current frame image according to the transformation relationship between the coordinate systems in the coordinate system set. Simultaneously, this application can also calculate the visual sensor height corresponding to the current frame image. This application calibrates the visual sensor extrinsic parameters for each frame image, then combines the calibrated visual sensor extrinsic parameters of all frames into a visual sensor extrinsic parameter dataset, and statistically analyzes the frequency proportion of different values ​​corresponding to each visual sensor extrinsic parameter, thereby determining the final calibrated visual sensor extrinsic parameters. Therefore, this solution can accurately and quickly calibrate the camera's extrinsic parameters, reducing camera extrinsic parameter calibration errors, thereby improving the accuracy and robustness of visual sensor extrinsic parameter calibration.

[0208] The method described in the above embodiments will be further described in detail below.

[0209] In this embodiment, the method of this application embodiment will be described in detail by taking the calibration of the extrinsic parameters of a camera installed on a car as an example.

[0210] like Figure 2a As shown, the specific process of one calibration method is as follows:

[0211] 200. Based on the camera's extrinsic parameters at the current moment, acquire the current frame image through the camera.

[0212] The extrinsic parameters of a camera are the parameters of the camera in the world coordinate system, such as the camera's position and rotation direction. The intrinsic parameters of a camera are the parameters related to the camera's own characteristics, such as the camera's focal length and pixel size.

[0213] In this embodiment, the camera's extrinsic parameters include the camera's pitch angle and yaw angle. This embodiment first acquires the captured image based on the camera's extrinsic parameters at the current moment, and then uses the captured image as the current frame image.

[0214] 210. Establish the set of coordinate systems corresponding to the current frame image, and determine the transformation relationship between the coordinate systems in the set.

[0215] In this embodiment of the application, the set of coordinate systems corresponding to the current frame image includes the world coordinate system, the camera coordinate system, the pixel coordinate system, and the imaging plane coordinate system.

[0216] The world coordinate system is an absolute coordinate system in the objective three-dimensional world, used to describe the absolute position of objects in space. In this embodiment, it is used to describe the reference position of the camera. The camera is mounted on a vehicle, and the X-axis of the world coordinate system... w O w Y w The plane is located on a horizontal ground surface, with the origin O of the world coordinate system. w Located on the central axis of the camera, perpendicular to X w O w Y w The location of the intersection of the planes, X in the world coordinate system. w The axis represents the direction of travel for a vehicle.

[0217] The pixel coordinate system is a two-dimensional coordinate system established using the captured image from the camera as the plane. The image captured by the camera can be stored as an array in the computer, where the value of each element (pixel) is the brightness (grayscale) of the image point. A Cartesian coordinate system uv is defined on the image, and the coordinates (u, v) of each pixel are the column and row numbers of that pixel in the array, respectively. Therefore, (u, v) are pixel coordinates in units of pixels, and the origin of the pixel coordinate system is defined at the top left corner of the image.

[0218] Since the pixel coordinate system represents the column and row number of a pixel in the captured image, but does not express the physical position of the pixel in the image using physical units, it is necessary to establish an imaging plane coordinate system xy, which expresses the position in physical units (e.g., centimeters). (x, y) represents the coordinates of the imaging plane coordinate system measured in physical units. In the xy coordinate system, i.e., the imaging plane coordinate system, the origin is defined at the intersection of the camera's optical axis and the image plane.

[0219] O of the camera coordinate system c For the camera's optical center, X c axis and Y c The Z axis is parallel to the x-axis and y-axis of the imaging plane coordinate system. c The axis is the optical axis of the camera, Z. c The optical axis is perpendicular to the image plane. The intersection of the optical axis and the image plane is the principal point O′ of the image, which is located from the origin O. c With X c Axis, Y c Axis and Z c The three-dimensional coordinate system composed of axes is called the camera coordinate system. c O′ represents the camera's focal length f.

[0220] In this embodiment of the application, determining the transformation relationship between coordinate systems in the coordinate system set includes:

[0221] Get the pixel coordinates (u, v) of the pixel in the current frame image;

[0222] Set the initial camera height value h and the Z-axis of the pixels in the current frame image in the camera coordinate system. c The distance d along the axis.

[0223] The initial height translation vector is obtained based on the initial camera height value h.

[0224] In this embodiment, the transformation from the camera coordinate system to the world coordinate system involves a translation process. Since the origin O of the world coordinate system... w Located on the central axis of the camera, perpendicular to X w O w Y w Given the intersection point of the planes, the initial translation vector of the height from the camera coordinate system to the world coordinate system can be determined based on the initial value h of the camera height.

[0225] Obtain camera intrinsic parameter K, camera intrinsic parameter f x This represents the ratio of the camera focal length f to the physical size dx of each pixel along the x-axis. yc is the ratio of the camera focal length f to the physical size dy of each pixel along the y-axis. x c y These are the coordinates of the camera's optical center in the pixel coordinate system with respect to the u-axis and v-axis, respectively.

[0226] Obtain the rotation matrix R from the camera coordinate system to the world coordinate system, where the rotation matrix R = R x R y , where R x For the camera to circle around X c The first rotation matrix obtained by rotating the axis by the yaw angle is R. y For the camera to circle around Y c The second rotation matrix is ​​obtained by rotating the axis by the pitch angle.

[0227] In this embodiment, the transformation from camera coordinate system to world coordinate system involves a rotation process, rotating around different coordinate axes by different angles to obtain corresponding rotation matrices. The first rotation matrix... Second rotation matrix

[0228] By calculating the equation Get the coordinates (X,Y) of the pixel in the current frame image in the world coordinate system.

[0229] 220. Update the camera's extrinsic parameters at the current moment based on the transformation relationship.

[0230] In one embodiment, updating the extrinsic parameters of the camera at the current moment according to the transformation relationship includes:

[0231] The lane lines in the current frame image are identified by feature point recognition to obtain a labeled image of the lane line feature points. The labeled image of the lane line feature points includes the lane line feature points.

[0232] Lane lines, also known as lane dividing lines, are traffic markings set on a level surface to separate traffic flows traveling in the same direction.

[0233] In this embodiment, the feature points corresponding to the lane lines in the current frame image are first identified, and then a marked image with lane line feature points can be obtained.

[0234] Obtain the transformation relationship between the pixel coordinate system and the world coordinate system in the coordinate system set.

[0235] The coordinates in the coordinate system set include pixel coordinates, imaging plane coordinates, camera coordinates, and world coordinates.

[0236] In the embodiments of this application, the transformation from the pixel coordinate system to the world coordinate system involves four transformation processes: from the pixel coordinate system to the imaging plane coordinate system, from the imaging plane coordinate system to the camera coordinate system, and from the camera coordinate system to the world coordinate system.

[0237] Based on the transformation relationship, the marked image in the pixel coordinate system is transformed to the first preset plane in the world coordinate system to obtain the lane line feature point transformation map, which includes the transformed lane line feature points.

[0238] The first preset plane is the X-axis of the world coordinate system. w O w Y w flat.

[0239] In this embodiment of the application, the marked image in the pixel coordinate system can be transformed to the X coordinate system in the world coordinate system through the transformation relationship. w O w Y w The plane can also map the coordinates of lane line feature pixels (i.e., lane line feature points) in the marked image in the pixel coordinate system to the X coordinate system in the world coordinate system. w O w Y w Coordinates of a plane.

[0240] The transformed lane line feature points in the lane line feature point transformation map are filtered to obtain the first feature point filtering map, such as... Figure 2b As shown, the first feature point filtering map includes the filtered feature points. The origin O1 of the first feature point filtering map is located at the upper left corner. The x1 coordinate axis of the first feature point filtering map is vertically downward, and the y1 coordinate axis of the first feature point filtering map is horizontally to the right.

[0241] In addition to the converted lane line feature points, the lane line feature point conversion map may also include other non-lane line feature points.

[0242] In this embodiment of the application, other non-lane line feature points are removed from the lane line feature point conversion map, and a first feature point filtering map with the filtered feature points can be obtained.

[0243] By fitting the selected feature points, the lane line corresponding to the current driving lane is obtained.

[0244] In this embodiment, the coordinates of the filtered feature points have been determined. To obtain the lane line corresponding to the current driving lane based on the coordinates of the filtered feature points, a fitting operation is still required.

[0245] The embodiments of this application can use the least squares method to fit the selected feature points into lane lines, or other methods such as gradient descent can be used to fit lane lines.

[0246] In one embodiment, the lane line corresponding to the current driving lane is obtained by fitting the selected feature points, including:

[0247] Determine the optical axis coordinates of the camera corresponding to the first feature point screening map. The optical axis coordinates include the second coordinates.

[0248] In this embodiment of the application, when an image is captured by a camera, the camera's optical axis will have a corresponding optical axis point (u) on the captured image. c ,v c Based on the transformation relationship, the image in the pixel coordinate system is transformed to the X coordinate system in the world coordinate system. w O w Y w In a plane, the optical axis point (u) c ,v c There will be a corresponding point in the first feature point selection map, and the optical axis point (u) will be selected. c ,v c The coordinates of the corresponding points in the first feature point selection map are used as the optical axis coordinates.

[0249] The optical axis coordinates include a second coordinate and a first coordinate. The origin O1 is the upper left corner of the first feature point screening map. The first coordinate corresponds to the x1 coordinate axis of the first feature point screening map, and the direction of the x1 coordinate axis is vertically downward. The second coordinate corresponds to the y1 coordinate axis of the first feature point screening map, and the direction of the y1 coordinate axis is horizontally to the right.

[0250] The first feature point screening map is divided into regions based on the second coordinate.

[0251] In this embodiment of the application, the second coordinate is the y1 coordinate value corresponding to the optical axis of the camera in the first feature point screening map.

[0252] In this embodiment, the first feature point filtering map can be divided into a first region and a second region based on the y1 coordinate value corresponding to the optical axis of the camera in the first feature point filtering map. The y1 coordinate corresponding to the optical axis in the first feature point filtering map is located at the intersection of the straight line in the first feature point filtering map and the lower edge of the image. Figure 2c As shown, in this embodiment, a straight line l can be determined by the y1 coordinate value corresponding to the optical axis in the first feature point screening map, such that the absolute value of the slope of the straight line l is not greater than the absolute value of the slope of the straight line connecting all lane lines in the first feature point screening map. Thus, the first feature point screening map is divided into a first region and a second region by this straight line l. The first region corresponds to the left region of the first feature point screening map, and the second region corresponds to the right region of the first feature point screening map.

[0253] Determine the first and second feature points corresponding to each lane line in the region.

[0254] In this embodiment of the application, when the vehicle travels in a straight line, each region of the first feature point screening map may have several lane lines, and each lane line contains several feature points, including the first feature point and the second feature point.

[0255] In this embodiment, the feature point farthest from the origin and the feature point closest to the origin in each lane line can be obtained according to the distance of each feature point in each lane line to the origin of the first feature point screening map. The feature point farthest from the origin is taken as the first feature point of the lane line, and the feature point closest to the origin is taken as the second feature point of the lane line.

[0256] Obtain the length between the first feature point and the second feature point.

[0257] In this embodiment of the application, the coordinates of the first feature point and the second feature point in the first feature point screening map can be obtained, and the length between the first feature point and the second feature point can be obtained based on the coordinates of the first feature point and the second feature point in the first feature point screening map.

[0258] Connect the first feature point and the lane line of the second feature point to form a straight line.

[0259] Determine the slope of the straight line in the first feature point screening graph.

[0260] When the length between the first feature point and the second feature point is greater than a preset length and the slope of the straight line in the first feature point selection map is greater than a preset slope, the lane line corresponding to the region is obtained; for example... Figure 2d As shown, the lane lines corresponding to the region include left lane lines and right lane lines. The left lane line is located in the first region, and the right lane line is located in the second region. Both the left lane line and the right lane line have a first feature point and a second feature point.

[0261] Use the lane lines corresponding to the area as the lane lines corresponding to the current driving lane.

[0262] In this embodiment of the application, the lane line corresponding to the region can only be obtained through the first feature point and the second feature point when the length between the first feature point and the second feature point is greater than a preset length, and the slope of the straight line connecting the lane lines of the first feature point and the second feature point in the first feature point screening map is greater than a preset slope. This can improve the calibration efficiency of camera extrinsic parameters and the accuracy of obtaining the lane line corresponding to the region.

[0263] When the lane lines corresponding to the current driving lane are parallel to each other, calculate the lane line angle between the lane line corresponding to the current driving lane and the first coordinate axis of the first feature point screening map.

[0264] The lane line angle is used as the camera yaw angle calibrated at the current moment.

[0265] In this embodiment of the application, the lane markings corresponding to the current driving lane have two lane markings, and these two lane markings may be parallel or non-parallel to each other. For example... Figure 2e As shown, if the two lane lines in the lane corresponding to the current driving lane are parallel to each other, it indicates that the initial camera pitch angle at the current moment is the accurate camera pitch angle, and there is no need to update the camera pitch angle at the current moment.

[0266] In this embodiment, calibrating the camera yaw angle requires that the lane lines corresponding to the current driving lane are parallel to each other. When two lane lines corresponding to the current driving lane are parallel to each other, the angle between any one of the two lane lines corresponding to the current driving lane and the x1 coordinate axis of the first feature point screening map is obtained. This angle is recorded as the lane line angle and used as the camera yaw angle calibrated at the current time. Figure 2e The included angle p is used as the camera yaw angle calibrated at the current time. That is, the calibration of the camera yaw angle at the current time is completed.

[0267] In one embodiment, updating the extrinsic parameters of the camera at the current moment according to the transformation relationship further includes:

[0268] When the lane lines corresponding to the current driving lane are not parallel to each other, obtain the first intersection point between the lane lines corresponding to the current driving lane.

[0269] Determine the origin of the first feature point selection map;

[0270] Determine the first distance between the first intersection point and the origin of the first feature point screening map;

[0271] Set the pitch angle step size based on the first distance;

[0272] Adjust the pitch angle in the camera's extrinsic parameters at the current moment according to the pitch angle step size to obtain the adjusted camera pitch angle;

[0273] The adjusted camera pitch angle is used as the camera pitch angle calibrated at the current moment.

[0274] In this embodiment, the origin of the first feature point filtering map is located at the upper left corner of the first feature point filtering map. The first distance is the distance between the first intersection point and the origin of the first feature point filtering map. The pitch angle step size is the step size when updating the pitch angle in the extrinsic parameters of the camera at the current moment. For example, the pitch angle step size can be set to 0.1 degrees, 0.2 degrees, 0.4 degrees, etc.

[0275] If the lane lines corresponding to the current driving lane are not parallel to each other, the camera pitch angle at the current moment needs to be updated. When setting the pitch angle step size based on the first distance, several distance range intervals can be set sequentially according to the distance between the first intersection point and the origin of the first feature point screening map. A different step size can be set for each distance range interval. The larger the distance between the first intersection point and the origin of the first feature point screening map, the smaller the pitch angle step size can be set. For example, several distance range intervals can be set as three distance range intervals: the first interval [0,100), the second interval [100,300), and the third interval [300,+∞). When the distance between the first intersection point and the origin of the first feature point screening map is in the first interval, the pitch angle step size is set to 0.4 degrees; when the distance between the first intersection point and the origin of the first feature point screening map is in the second interval, the pitch angle step size is set to 0.2 degrees; and when the distance between the first intersection point and the origin of the first feature point screening map is in the third interval, the pitch angle step size is set to 0.1 degrees. In this embodiment, the longer the distance between the first intersection point and the origin of the first feature point screening map, the closer the lane lines corresponding to the current driving lane are to parallelism. When the distance is relatively large, it indicates that the angle difference is small, and a small step size is used to approximate it (for example, when the distance is greater than 300 meters, the pitch angle step size is set to 0.1 degrees); when the distance is small, it indicates that the angle difference is large, and a large step size is used to approximate it (for example, when the distance is less than 100 meters, the pitch angle step size is set to 0.4 degrees). This can reduce the number of iterations and improve the efficiency of camera extrinsic parameter calibration.

[0276] In one embodiment, after updating the extrinsic parameters of the camera at the current moment according to the transformation relationship, the method further includes:

[0277] Based on the adjusted camera pitch angle, a new first image is captured by the camera at the next moment;

[0278] Determine the second feature point selection map corresponding to the first image;

[0279] Determine the origin of the second feature point selection map;

[0280] Obtain the lane line corresponding to the current driving lane in the second feature point filtering image;

[0281] When the lane lines corresponding to the current driving lane in the second feature point filtering image are not parallel to each other, determine the second intersection point between the lane lines corresponding to the current driving lane in the second feature point filtering image.

[0282] Obtain the second distance between the second intersection point and the origin of the second feature point filtering map;

[0283] When the sign of the second distance is opposite to the sign of the first distance, the absolute values ​​of the first and second distances are compared to obtain the maximum distance; the signs of the first and second distances correspond to the directions of the first coordinate axis, respectively.

[0284] The camera pitch angle corresponding to the maximum distance is used as the camera pitch angle calibrated at the next moment.

[0285] In this embodiment, based on the adjusted camera pitch angle, a new image is acquired by the camera at the next moment, and this image is recorded as the first image. The origin of the second feature point filtering map is the same as the origin of the first feature point filtering map, both located at the upper left corner of the image. The direction of the coordinate axis of the second feature point filtering map is consistent with the direction of the coordinate axis of the first feature point filtering map, and the first coordinate axis is the x1 coordinate axis of the first feature point filtering map. The second distance and the first distance can represent not only the magnitude of the distance but also the direction. For example, when the sign of the second distance is negative, it indicates that the second intersection point is located in the negative region of the x1 coordinate axis; when the sign of the second distance is positive, it indicates that the second intersection point is located in the positive region of the x1 coordinate axis. When the lane lines corresponding to the current driving lane in the second feature point filtering map are not parallel to each other, the intersection point between the lane lines corresponding to the current driving lane in the second feature point filtering map can be determined, and this intersection point is recorded as the second intersection point. Then, the distance between the second intersection point and the origin of the second feature point filtering map is obtained, and this distance is recorded as the second distance.

[0286] In this embodiment of the application, when determining the second feature point filtering map corresponding to the first image, the first image in the pixel coordinate system is transformed to the first preset plane in the world coordinate system according to the transformation relationship to obtain the first lane line feature point transformation map. Then, the transformed lane line feature points in the first lane line feature point transformation map are filtered to obtain the second feature point filtering map.

[0287] In this embodiment of the application, when the sign of the second distance is opposite to the sign of the first distance, it indicates that the intersection point of the lane line corresponding to the current driving lane in the feature point filtering map corresponding to the image captured at the next moment (i.e., the second intersection point) and the intersection point of the lane line corresponding to the current driving lane in the feature point filtering map corresponding to the image captured at the current moment (i.e., the first intersection point) have been reversed about the x1 coordinate axis. At this time, the camera pitch angle that needs to be calibrated can be solved. The pitch angle corresponding to the maximum absolute value of the distance before and after the reversal is taken as the solution result, that is, the camera pitch angle corresponding to the maximum distance is taken as the camera pitch angle calibrated at the next moment.

[0288] In one embodiment, after using the lane line angle as the camera yaw angle calibrated at the current time, the method further includes:

[0289] The camera's yaw angle is corrected based on the yaw angle calibrated at the current time to obtain the corrected camera yaw angle;

[0290] A new second image is acquired by the camera based on the corrected camera yaw angle;

[0291] Determine the third feature point selection map corresponding to the second image;

[0292] Obtain the lane line corresponding to the current driving lane in the third feature point filtering image;

[0293] The lane lines corresponding to the current driving lane in the third feature point filtering image are respectively recorded as the first lane line and the second lane line.

[0294] In the third feature point filtering map, determine the first intercept between the first lane line and the first coordinate axis, and the second intercept between the second lane line and the first coordinate axis;

[0295] The camera height is determined based on the first and second intercepts.

[0296] The first coordinate axis is the x1 coordinate axis of the third feature point selection map. The x1 coordinate axis of the third feature point selection map corresponds to the x1 coordinate axis of the second feature point selection map and the x1 coordinate axis of the first feature point selection map, respectively.

[0297] In this embodiment, after determining that the lane lines corresponding to the current driving lane are parallel to each other, the camera yaw angle at the current moment can be calibrated. Then, the camera yaw angle is corrected based on the calibrated camera yaw angle at the current moment, such as... Figure 2f As shown, this ensures that the lane line corresponding to the current driving lane after correction is parallel to the x1 coordinate axis of the third feature point screening map.

[0298] This application embodiment can first obtain the intercepts between the two lane lines corresponding to the current driving lane and the x1 coordinate axis of the third feature point screening map, that is, the perpendicular distances between the two lane lines and the x1 coordinate axis of the third feature point screening map, and then determine the camera height based on the first intercept and the second intercept. This application can not only calibrate the camera's extrinsic parameters, but also obtain the camera height information.

[0299] In one embodiment, determining the camera height based on a first intercept and a second intercept includes:

[0300] Calculate the difference between the first intercept and the second intercept, and use the difference as the lane width measurement value;

[0301] Obtain the preset value of lane line width, and use the ratio between the measured value of lane line width and the preset value of lane line width as the camera height weight;

[0302] Obtain the camera height preset value;

[0303] The camera height is obtained by multiplying the camera height weight by the preset camera height value.

[0304] In this embodiment, the camera height can be determined using lane width measurements. (Camera height) H′ is the preset value for camera height, W is the measured value for lane width, and W′ is the preset value for lane width. For example, the preset value for lane width on an urban arterial road is W′ = 3.60.

[0305] 230. Obtain the extrinsic parameter dataset, which includes the camera's extrinsic parameters at the current time and historical time points.

[0306] In the embodiments of this application, historical moments include moments before the current moment.

[0307] The embodiments of this application can calibrate different camera extrinsic parameters at different times. For example, at the previous time, the camera's pitch angle was calibrated to -2 degrees; at the current time, the camera's pitch angle was calibrated to -3 degrees and yaw angle to 0 degrees; at the next time, the camera's pitch angle was calibrated to -4 degrees and yaw angle to -2 degrees.

[0308] 240. Determine the calibration values ​​based on the external parameter dataset.

[0309] In this embodiment, the calibration value is the difference value that appears most frequently, and the difference value is the different value corresponding to the camera's external parameters.

[0310] This application embodiment captures a large number of frame images during a straight-line journey of the vehicle. Each frame is analyzed and the camera's extrinsic parameters are calibrated. The frequency of occurrence of each angle in the camera's extrinsic parameters is statistically analyzed. For example, when statistically analyzing the camera's pitch angle, -3 degrees appeared 600 times, -4 degrees appeared 10 times, and -2 degrees appeared 30 times. By comparison, the value with the highest frequency is selected, i.e., the pitch angle is calibrated as -3 degrees, and -3 degrees is the corresponding calibration value for the pitch angle. Similarly, when statistically analyzing the camera's yaw angle, 0 degrees appeared 340 times, -1 degree appeared 60 times, and 2 degrees appeared 70 times. By comparison, the value with the highest frequency is selected, i.e., the yaw angle is calibrated as 0 degrees, and 0 degrees is the corresponding calibration value for the yaw angle.

[0311] 250. Assign the calibration values ​​to the camera's external parameters to complete the camera's external parameter calibration.

[0312] This application first counts the frequency of occurrence of each angle in the camera's extrinsic parameters, and then assigns the extrinsic parameter value of the camera with the highest frequency of occurrence to the extrinsic parameters of the final calibrated camera, thereby completing the extrinsic parameter calibration of the camera.

[0313] As can be seen from the above, the embodiments of this application can establish a coordinate system set corresponding to the current frame image, and calibrate the visual sensor extrinsic parameters corresponding to the current frame image according to the transformation relationship between the coordinate systems in the coordinate system set. Simultaneously, this application can also calculate the camera height corresponding to the current frame image. This application calibrates the camera extrinsic parameters for each frame image, then combines the calibrated camera extrinsic parameters of all frames into a camera extrinsic parameter dataset, and statistically analyzes the frequency proportion of different values ​​corresponding to each type of camera extrinsic parameter, thereby determining the final calibrated camera extrinsic parameters. Therefore, this solution can accurately and quickly calibrate the camera extrinsic parameters, reducing camera extrinsic parameter calibration errors, thereby improving the accuracy and robustness of visual sensor extrinsic parameter calibration.

[0314] To better implement the above methods, this application also provides a calibration device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0315] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the calibration device specifically integrated into an electronic device as an example.

[0316] For example, such as Figure 3 As shown, the calibration device may include an image acquisition unit 13, a coordinate system establishment unit 14, an external parameter update unit 15, a dataset acquisition unit 16, a calibration value determination unit 17, and an external parameter calibration completion unit 18, as follows:

[0317] (I) Image Acquisition Unit 13

[0318] The image acquisition unit 13 is used to acquire the current frame image through the visual sensor based on the extrinsic parameters of the visual sensor at the current moment.

[0319] (II) Coordinate System Establishment Unit 14

[0320] The coordinate system establishment unit 14 is used to establish the coordinate system set corresponding to the current frame image and determine the transformation relationship between the coordinate systems in the coordinate system set.

[0321] (III) External Parameter Update Unit 15

[0322] The extrinsic parameter update unit 15 is used to update the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship.

[0323] In some embodiments, the extrinsic parameter update unit 15 includes an extrinsic parameter update subunit, which is used for...

[0324] The lane lines in the current frame image are identified by feature points to obtain a labeled image of lane line feature points, which includes lane line feature points.

[0325] Obtain the transformation relationship between the pixel coordinate system and the world coordinate system in the coordinate system set;

[0326] Based on the transformation relationship, the marked image in the pixel coordinate system is transformed to the first preset plane in the world coordinate system to obtain the lane line feature point transformation map, which includes the transformed lane line feature points.

[0327] The transformed lane line feature points in the lane line feature point transformation map are filtered to obtain the first feature point filtering map, which includes the filtered feature points.

[0328] The lane line corresponding to the current driving lane is obtained by fitting the selected feature points.

[0329] When the lane lines corresponding to the current driving lane are parallel to each other, calculate the lane line angle between the lane line corresponding to the current driving lane and the first coordinate axis of the first feature point screening map.

[0330] The lane line angle is used as the yaw angle of the visual sensor at the current moment.

[0331] In some embodiments, the extrinsic parameter update subunit includes a lane line fitting unit, which is used for:

[0332] Determine the optical axis coordinates of the visual sensor corresponding to the first feature point screening map, where the optical axis coordinates include the second coordinates;

[0333] The first feature point screening map is divided into regions based on the second coordinates;

[0334] Determine the first and second feature points corresponding to each lane line in the region;

[0335] Obtain the length between the first feature point and the second feature point;

[0336] Connect the lane lines of the first feature point and the second feature point into a straight line;

[0337] Determine the slope of the straight line in the first feature point selection graph;

[0338] When the length between the first feature point and the second feature point is greater than a preset length, and the slope of the straight line in the first feature point screening map is greater than a preset slope, the lane line corresponding to the region is obtained; the lane line corresponding to the region includes the first feature point and the second feature point;

[0339] Use the lane lines corresponding to the area as the lane lines corresponding to the current driving lane.

[0340] In some embodiments, the extrinsic parameter update subunit further includes a pitch angle update unit, which is used for:

[0341] When the lane lines corresponding to the current driving lane are not parallel to each other, obtain the first intersection point between the lane lines corresponding to the current driving lane.

[0342] Determine the origin of the first feature point selection map;

[0343] Determine the first distance between the first intersection point and the origin of the first feature point screening map;

[0344] Set the pitch angle step size based on the first distance;

[0345] Adjust the pitch angle in the extrinsic parameters of the vision sensor at the current moment according to the pitch angle step size to obtain the adjusted pitch angle of the vision sensor;

[0346] The adjusted visual sensor pitch angle is used as the visual sensor pitch angle calibrated at the current moment.

[0347] In some embodiments, the pitch angle update unit further includes a pitch angle update subunit, which is used for:

[0348] Based on the adjusted pitch angle of the visual sensor, a new first image is acquired at the next moment through the visual sensor;

[0349] Determine the second feature point selection map corresponding to the first image;

[0350] Determine the origin of the second feature point selection map;

[0351] Obtain the lane line corresponding to the current driving lane in the second feature point filtering image;

[0352] When the lane lines corresponding to the current driving lane in the second feature point filtering image are not parallel to each other, determine the second intersection point between the lane lines corresponding to the current driving lane in the second feature point filtering image.

[0353] Obtain the second distance between the second intersection point and the origin of the second feature point filtering map;

[0354] When the sign of the second distance is opposite to the sign of the first distance, the absolute values ​​of the first and second distances are compared to obtain the maximum distance; the signs of the first and second distances correspond to the directions of the first coordinate axis, respectively.

[0355] The pitch angle of the visual sensor corresponding to the maximum distance is used as the pitch angle of the visual sensor calibrated at the next moment.

[0356] In some embodiments, the extrinsic parameter update unit further includes a height determination unit, the height determination unit being configured to:

[0357] The yaw angle of the visual sensor is corrected based on the yaw angle of the visual sensor calibrated at the current time, and the corrected yaw angle of the visual sensor is obtained.

[0358] A new second image is acquired using the vision sensor based on the corrected yaw angle.

[0359] Determine the third feature point selection map corresponding to the second image;

[0360] Obtain the lane line corresponding to the current driving lane in the third feature point filtering image;

[0361] The lane lines corresponding to the current driving lane in the third feature point filtering image are respectively denoted as the first lane line and the second lane line.

[0362] In the third feature point filtering map, determine the first intercept between the first lane line and the first coordinate axis, and the second intercept between the second lane line and the first coordinate axis;

[0363] The height of the vision sensor is determined based on the first intercept and the second intercept.

[0364] In some embodiments, the height determining unit includes a height determining subunit:

[0365] Calculate the difference between the first intercept and the second intercept, and use the difference as the lane width measurement value;

[0366] Obtain the preset value of lane line width, and use the ratio between the measured value of lane line width and the preset value of lane line width as the height weight of the visual sensor.

[0367] Obtain the preset height value of the vision sensor;

[0368] The visual sensor height is obtained by multiplying the visual sensor height weight by the preset visual sensor height value.

[0369] (IV) Dataset Acquisition Unit 16

[0370] Dataset acquisition unit 16 is used to acquire extrinsic parameter datasets, which include the extrinsic parameters of the visual sensor at the current time and historical time points. Historical time points include time points prior to the current time.

[0371] (V) Calibration Value Determination Unit 17

[0372] The calibration value determination unit 17 is used to determine the calibration value based on the external parameter dataset. The calibration value is the difference value that appears most frequently. The difference value is the different value corresponding to the external parameter of the vision sensor.

[0373] (VI) External parameter calibration completion unit 18

[0374] The extrinsic parameter calibration completion unit 18 is used to assign calibration values ​​to the extrinsic parameters of the vision sensor in order to complete the extrinsic parameter calibration of the vision sensor.

[0375] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0376] As can be seen from the above, the calibration device of this embodiment consists of an image acquisition unit 13, a coordinate system establishment unit 14, an extrinsic parameter update unit 15, a dataset acquisition unit 16, a calibration value determination unit 17, and an extrinsic parameter calibration completion unit 18. It can establish a coordinate system set corresponding to the current frame image and calibrate the visual sensor extrinsic parameters corresponding to the current frame image according to the transformation relationship between the coordinate systems in the coordinate system set. At the same time, this application can also calculate the camera height corresponding to the current frame image. This application calibrates the camera extrinsic parameters for each frame image, and then combines the camera extrinsic parameters calibrated for all frames into a camera extrinsic parameter dataset, and counts the frequency ratio of different values ​​corresponding to each type of camera extrinsic parameter, thereby determining the final calibrated camera extrinsic parameters.

[0377] Therefore, the embodiments of this application can improve the accuracy and speed of camera extrinsic parameter calibration, reduce camera extrinsic parameter calibration errors, and thus improve the accuracy and robustness of visual sensor extrinsic parameter calibration.

[0378] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0379] In some embodiments, the calibration device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the calibration method of this application.

[0380] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 4 As shown, it illustrates a schematic diagram of the server structure involved in an embodiment of this application. Specifically:

[0381] The server may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 4 The server architecture shown does not constitute a limitation on the server and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Wherein:

[0382] The processor 401 is the control center of the server, connecting various parts of the server through various interfaces and lines. It performs various server functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, thereby performing overall server monitoring. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 401.

[0383] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0384] The server also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0385] The server may also include an input module 404, which can be used to receive input numeric or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0386] The server may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, through which the server can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0387] Although not shown, the server may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the server loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0388] Based on the extrinsic parameters of the visual sensor at the current moment, the current frame image is acquired through the visual sensor.

[0389] Establish a set of coordinate systems corresponding to the current frame image, and determine the transformation relationships between the coordinate systems in the set;

[0390] Update the extrinsic parameters of the vision sensor at the current moment based on the transformation relationship;

[0391] Obtain the extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current time and historical time. The historical time includes the time before the current time.

[0392] The calibration values ​​are determined based on the extrinsic parameter dataset. The calibration values ​​are the most frequently occurring difference values. The difference values ​​are the different values ​​corresponding to the extrinsic parameters of the visual sensor.

[0393] The calibration values ​​are assigned to the extrinsic parameters of the vision sensor to complete the extrinsic parameter calibration of the vision sensor.

[0394] In one embodiment, when updating the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship, the processor 401 is specifically configured to perform the following steps:

[0395] The lane lines in the current frame image are identified by feature points to obtain a labeled image of lane line feature points, wherein the labeled image of lane line feature points includes lane line feature points.

[0396] Obtain the transformation relationship between the pixel coordinate system and the world coordinate system in the coordinate system set;

[0397] According to the transformation relationship, the marked image in the pixel coordinate system is transformed to the first preset plane in the world coordinate system to obtain a lane line feature point transformation map, which includes the transformed lane line feature points.

[0398] The transformed lane line feature points in the lane line feature point transformation map are filtered to obtain a first feature point filtering map, which includes the filtered feature points.

[0399] The lane line corresponding to the current driving lane is obtained by fitting the filtered feature points.

[0400] When the lane lines corresponding to the current driving lane are parallel to each other, calculate the lane line angle between the lane line corresponding to the current driving lane and the first coordinate axis of the first feature point screening map.

[0401] The included angle of the lane lines is used as the yaw angle of the visual sensor calibrated at the current moment.

[0402] In one embodiment, when fitting the selected feature points to obtain the lane line corresponding to the current driving lane, the processor 401 specifically performs the following steps:

[0403] Determine the optical axis coordinates of the visual sensor in the first feature point screening map, wherein the optical axis coordinates include a second coordinate.

[0404] The first feature point filtering map is divided into regions based on the second coordinates;

[0405] Determine the first feature point and the second feature point corresponding to each lane line in the region;

[0406] Obtain the length between the first feature point and the second feature point;

[0407] Connect the first feature point and the lane line of the second feature point into a straight line;

[0408] Determine the slope of the straight line in the first feature point screening map;

[0409] When the length between the first feature point and the second feature point is greater than a preset length, and the slope of the straight line in the first feature point filtering map is greater than a preset slope, the lane line corresponding to the region is obtained; the lane line corresponding to the region includes the first feature point and the second feature point;

[0410] The lane lines corresponding to the area are used as the lane lines corresponding to the current driving lane.

[0411] In one embodiment, when updating the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship, the processor 401 is further configured to perform the following steps:

[0412] When the lane lines corresponding to the current driving lane are not parallel to each other, obtain the first intersection point between the lane lines corresponding to the current driving lane.

[0413] Determine the origin of the first feature point screening map;

[0414] Determine the first distance between the first intersection point and the origin of the first feature point screening map;

[0415] Set the pitch angle step size based on the first distance;

[0416] Adjust the pitch angle in the extrinsic parameters of the visual sensor at the current moment according to the pitch angle step size to obtain the adjusted pitch angle of the visual sensor;

[0417] The adjusted visual sensor pitch angle is used as the visual sensor pitch angle calibrated at the current moment.

[0418] In one embodiment, after updating the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship, the processor 401 is further configured to perform the following steps:

[0419] Based on the adjusted pitch angle of the visual sensor, a new first image is acquired at the next moment through the visual sensor;

[0420] Determine the second feature point filtering map corresponding to the first image;

[0421] Determine the origin of the second feature point filtering map;

[0422] Obtain the lane line corresponding to the current driving lane in the second feature point filtering image;

[0423] When the lane lines corresponding to the current driving lane in the second feature point filtering image are not parallel to each other, determine the second intersection point between the lane lines corresponding to the current driving lane in the second feature point filtering image.

[0424] Obtain the second distance between the second intersection point and the origin of the second feature point filtering map;

[0425] When the sign of the second distance is opposite to the sign of the first distance, the absolute values ​​of the first distance and the second distance are compared to obtain the maximum distance; the signs of the first distance and the second distance correspond to the directions of the first coordinate axis, respectively.

[0426] The pitch angle of the visual sensor corresponding to the maximum distance is used as the pitch angle of the visual sensor calibrated at the next moment.

[0427] In one embodiment, after using the lane line angle as the visual sensor yaw angle calibrated at the current moment, the processor 401 is further configured to perform the following steps:

[0428] The yaw angle of the visual sensor is corrected based on the yaw angle of the visual sensor calibrated at the current time to obtain the corrected yaw angle of the visual sensor.

[0429] Based on the corrected yaw angle of the visual sensor, a new second image is acquired through the visual sensor;

[0430] Determine the third feature point selection map corresponding to the second image;

[0431] Obtain the lane line corresponding to the current driving lane in the third feature point filtering image;

[0432] The lane lines corresponding to the current driving lane in the third feature point filtering image are respectively recorded as the first lane line and the second lane line.

[0433] In the third feature point filtering map, a first intercept between the first lane line and the first coordinate axis, and a second intercept between the second lane line and the first coordinate axis are determined;

[0434] The height of the vision sensor is determined based on the first intercept and the second intercept.

[0435] In one embodiment, when determining the visual sensor height based on the first intercept and the second intercept, the processor 401 is further configured to perform the following steps:

[0436] Calculate the difference between the first intercept and the second intercept, and use the difference as the lane width measurement value;

[0437] Obtain a preset value for lane line width, and use the ratio between the measured value of lane line width and the preset value of lane line width as the height weight of the visual sensor.

[0438] Obtain the preset height value of the vision sensor;

[0439] The visual sensor height is obtained by multiplying the visual sensor height weight by the preset visual sensor height value.

[0440] The server in this embodiment can establish a coordinate system set corresponding to the current frame image, and calibrate the visual sensor extrinsic parameters corresponding to the current frame image according to the transformation relationship between the coordinate systems in the coordinate system set. Simultaneously, this application can also calculate the camera height corresponding to the current frame image. This application calibrates the camera extrinsic parameters for each frame image, then combines the calibrated camera extrinsic parameters of all frames into a camera extrinsic parameter dataset, and counts the frequency proportion of different values ​​corresponding to each type of camera extrinsic parameter, thereby determining the final calibrated camera extrinsic parameters. Therefore, this solution can accurately and quickly calibrate the camera extrinsic parameters, reducing camera extrinsic parameter calibration errors, thereby improving the accuracy and robustness of visual sensor extrinsic parameter calibration.

[0441] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the above calibration methods.

[0442] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0443] As can be seen from the above, the embodiments of this application can establish a coordinate system set corresponding to the current frame image, and calibrate the visual sensor extrinsic parameters corresponding to the current frame image according to the transformation relationship between the coordinate systems in the coordinate system set. Simultaneously, this application can also calculate the camera height corresponding to the current frame image. This application calibrates the camera extrinsic parameters for each frame image, then combines the calibrated camera extrinsic parameters of all frames into a camera extrinsic parameter dataset, and statistically analyzes the frequency proportion of different values ​​corresponding to each type of camera extrinsic parameter, thereby determining the final calibrated camera extrinsic parameters. Therefore, this solution can accurately and quickly calibrate the camera extrinsic parameters, reduce camera extrinsic parameter calibration errors, and thus improve the accuracy and robustness of visual sensor extrinsic parameter calibration.

[0444] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0445] To this end, embodiments of this application provide a computer-readable storage medium storing multiple instructions that can be loaded by a processor to execute the steps in any of the calibration methods provided in this application. For example, the instructions can execute the following steps: based on the extrinsic parameters of the visual sensor at the current moment, acquire the current frame image through the visual sensor; establish a coordinate system set corresponding to the current frame image and determine the transformation relationship between the coordinate systems in the coordinate system set; update the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship; obtain an extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current moment and historical moments; determine calibration values ​​based on the extrinsic parameter dataset, where the calibration values ​​are the most frequently occurring difference values; the difference values ​​are the different values ​​corresponding to the extrinsic parameters of the visual sensor; and assign the calibration values ​​to the extrinsic parameters of the visual sensor to complete the extrinsic parameter calibration of the visual sensor.

[0446] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0447] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the calibration aspect provided in the above embodiments or the methods provided in various optional implementations of the calibration aspect.

[0448] Since the instructions stored in the storage medium can execute the steps in any of the calibration methods provided in the embodiments of this application, the beneficial effects that any of the calibration methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0449] The above provides a detailed description of a calibration method, apparatus, server, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A calibration method, characterized in that, include: Based on the extrinsic parameters of the visual sensor at the current moment, the current frame image is acquired through the visual sensor. Establish a set of coordinate systems corresponding to the current frame image, and determine the transformation relationships between the coordinate systems in the set of coordinate systems; Update the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship to obtain the yaw angle of the visual sensor calibrated at the current moment; The yaw angle of the visual sensor is corrected based on the yaw angle of the visual sensor calibrated at the current time to obtain the corrected yaw angle of the visual sensor. Based on the corrected yaw angle of the visual sensor, a new second image is acquired through the visual sensor; Determine the third feature point filter map corresponding to the second image; obtain the lane line corresponding to the current driving lane in the third feature point filter map; The lane lines corresponding to the current driving lane in the third feature point filtering image are respectively recorded as the first lane line and the second lane line. In the third feature point filtering map, a first intercept between the first lane line and the first coordinate axis, and a second intercept between the second lane line and the first coordinate axis are determined; the height of the visual sensor is determined based on the first intercept and the second intercept; wherein, the third feature point filtering map is obtained by: transforming the second image in the pixel coordinate system to a first preset plane in the world coordinate system according to the transformation relationship to obtain a first lane line feature point transformation map; and filtering the transformed lane line feature points in the first lane line feature point transformation map to obtain the third feature point filtering map; the first coordinate axis is the horizontal coordinate axis of the third feature point filtering map; Obtain the extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current time and at historical times, where the historical times include times before the current time. The calibration values ​​are determined based on the extrinsic parameter dataset, and the calibration values ​​are the most frequently occurring difference values; the difference values ​​are the different values ​​corresponding to the extrinsic parameters of the visual sensor. The calibration value is assigned to the extrinsic parameters of the vision sensor to complete the extrinsic parameter calibration of the vision sensor.

2. The calibration method as described in claim 1, characterized in that, The step of updating the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship includes: The lane lines in the current frame image are identified by feature points to obtain a labeled image of lane line feature points, wherein the labeled image of lane line feature points includes lane line feature points. Obtain the transformation relationship between the pixel coordinate system and the world coordinate system in the coordinate system set; According to the transformation relationship, the marked image in the pixel coordinate system is transformed to the first preset plane in the world coordinate system to obtain a lane line feature point transformation map, which includes the transformed lane line feature points. The transformed lane line feature points in the lane line feature point transformation map are filtered to obtain a first feature point filtering map, which includes the filtered feature points. The lane line corresponding to the current driving lane is obtained by fitting the filtered feature points. When the lane lines corresponding to the current driving lane are parallel to each other, calculate the lane line angle between the lane line corresponding to the current driving lane and the first coordinate axis of the first feature point screening map. The included angle of the lane lines is used as the yaw angle of the visual sensor calibrated at the current moment.

3. The calibration method as described in claim 2, characterized in that, The step of fitting the selected feature points to obtain the lane line corresponding to the current driving lane includes: Determine the optical axis coordinates of the visual sensor in the first feature point screening map, wherein the optical axis coordinates include a second coordinate. The first feature point filtering map is divided into regions based on the second coordinates; Determine the first feature point and the second feature point corresponding to each lane line in the region; Obtain the length between the first feature point and the second feature point; Connect the first feature point and the lane line of the second feature point into a straight line; Determine the slope of the straight line in the first feature point screening map; When the length between the first feature point and the second feature point is greater than a preset length, and the slope of the straight line in the first feature point filtering map is greater than a preset slope, the lane line corresponding to the region is obtained; the lane line corresponding to the region includes the first feature point and the second feature point; The lane lines corresponding to the area are used as the lane lines corresponding to the current driving lane.

4. The calibration method as described in claim 2, characterized in that, The step of updating the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship further includes: When the lane lines corresponding to the current driving lane are not parallel to each other, obtain the first intersection point between the lane lines corresponding to the current driving lane. Determine the origin of the first feature point screening map; Determine the first distance between the first intersection point and the origin of the first feature point screening map; Set the pitch angle step size based on the first distance; Adjust the pitch angle in the extrinsic parameters of the visual sensor at the current moment according to the pitch angle step size to obtain the adjusted pitch angle of the visual sensor; The adjusted visual sensor pitch angle is used as the visual sensor pitch angle calibrated at the current moment.

5. The calibration method as described in claim 4, characterized in that, After updating the extrinsic parameters of the visual sensor at the current moment according to the transformation relationship, the method further includes: Based on the adjusted pitch angle of the visual sensor, a new first image is acquired at the next moment through the visual sensor; Determine the second feature point filtering map corresponding to the first image; Determine the origin of the second feature point filtering map; Obtain the lane line corresponding to the current driving lane in the second feature point filtering image; When the lane lines corresponding to the current driving lane in the second feature point filtering image are not parallel to each other, determine the second intersection point between the lane lines corresponding to the current driving lane in the second feature point filtering image. Obtain the second distance between the second intersection point and the origin of the second feature point filtering map; When the sign of the second distance is opposite to the sign of the first distance, the absolute values ​​of the first distance and the second distance are compared to obtain the maximum distance; the signs of the first distance and the second distance correspond to the directions of the first coordinate axis, respectively. The pitch angle of the visual sensor corresponding to the maximum distance is used as the pitch angle of the visual sensor calibrated at the next moment.

6. The calibration method as described in claim 1, characterized in that, Determining the visual sensor height based on the first intercept and the second intercept includes: Calculate the difference between the first intercept and the second intercept, and use the difference as the lane width measurement value; Obtain a preset value for lane line width, and use the ratio between the measured value of lane line width and the preset value of lane line width as the height weight of the visual sensor. Obtain the preset height value of the vision sensor; The visual sensor height is obtained by multiplying the visual sensor height weight by the preset visual sensor height value.

7. A calibration device, characterized in that, include: An image acquisition unit is used to acquire the current frame image through the visual sensor based on the extrinsic parameters of the visual sensor at the current moment. A coordinate system establishment unit is used to establish a set of coordinate systems corresponding to the current frame image and determine the transformation relationship between the coordinate systems in the set of coordinate systems; The extrinsic parameter update unit is used to update the extrinsic parameters of the vision sensor at the current time according to the transformation relationship, so as to obtain the yaw angle of the vision sensor calibrated at the current time. The yaw angle of the visual sensor is corrected based on the yaw angle of the visual sensor calibrated at the current time to obtain the corrected yaw angle of the visual sensor. Based on the corrected yaw angle of the visual sensor, a new second image is acquired through the visual sensor; Determine the third feature point filter map corresponding to the second image; obtain the lane line corresponding to the current driving lane in the third feature point filter map; The lane lines corresponding to the current driving lane in the third feature point filtering image are respectively recorded as the first lane line and the second lane line. In the third feature point filtering map, a first intercept between the first lane line and the first coordinate axis, and a second intercept between the second lane line and the first coordinate axis are determined; the height of the visual sensor is determined based on the first intercept and the second intercept; wherein, the third feature point filtering map is obtained by: transforming the second image in the pixel coordinate system to a first preset plane in the world coordinate system according to the transformation relationship to obtain a first lane line feature point transformation map; and filtering the transformed lane line feature points in the first lane line feature point transformation map to obtain the third feature point filtering map; the first coordinate axis is the horizontal coordinate axis of the third feature point filtering map; A dataset acquisition unit is used to acquire an extrinsic parameter dataset, which includes the extrinsic parameters of the visual sensor at the current time and at historical times, and the historical times include times before the current time. The calibration value determination unit is used to determine calibration values ​​based on the extrinsic parameter dataset, wherein the calibration values ​​are the most frequently occurring difference values; and the difference values ​​are the different values ​​corresponding to the extrinsic parameters of the visual sensor. The extrinsic parameter calibration completion unit is used to assign the calibration value to the extrinsic parameter of the vision sensor to complete the extrinsic parameter calibration of the vision sensor.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the calibration method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the calibration method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the calibration method according to any one of claims 1 to 6.

Images