A camera extrinsic parameter detection and adjustment method and system, an electronic device, and a storage medium

Abstract

The application provides a camera external parameter detection adjustment method and system, electronic equipment and a storage medium. The application obtains an original picture collected by a side-view camera, determines a value of camera external parameter change through frame subtraction based on the original picture, triggers an external parameter alarm when the value of the camera external parameter change is greater than a preset adjustment threshold, and adjusts the heading angle of the side-view camera through lane line adjustment and virtual camera adjustment for the camera with abnormal external parameters, thereby solving the problem that the prior art cannot automatically detect and adjust the external parameters with the help of a mark line and a vanishing point, and the overlapping viewing angles between the side-view cameras are small, and it is difficult to realize multi-camera calibration with the help of multi-view stereovision.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving, and in particular to a method, system, electronic device, and storage medium for detecting and adjusting camera extrinsic parameters. Background Technology

[0002] Autonomous driving, as a key technology for realizing vehicle intelligence, has received increasing attention and research from the government and traditional automakers. Statistics show that approximately 90% of traffic accidents in my country each year are caused by driver error. Autonomous driving systems acquire various sensory information and then intervene or control vehicle operation, reducing or replacing driver intervention and thus reducing traffic accidents to some extent. With the advancement of mass production of autonomous driving, multi-camera side-view perception systems have become crucial for mass production technology. Since the extrinsic parameters (rotation and translation parameters) of the camera relative to the vehicle body change with vehicle use (long-term ground vibrations), accurately obtaining the extrinsic parameters of the camera relative to the vehicle coordinate system is the most critical technology in multi-camera BEV perception.

[0003] Existing technologies can be broadly categorized into two types. One is the extrinsic parameter estimation method based on vanishing point calibration, which can be used for front / rear camera calibration. This method requires a clear lane line in the environment and ensures that the vehicle is centered and parallel to the lane line. The camera's extrinsic parameters are estimated by observing the intersection points (vanishing points) of parallel lines in the image. By analyzing the vanishing point positions and changes in the image, camera parameters can be updated and adjusted in real time, achieving real-time extrinsic parameter calibration. The other type is calibration based on multi-view stereo vision, which calculates extrinsic parameters using the same object points in the shared field of view of multiple cameras, such as... Figure 1 As shown in Figure 2, during camera movement, the pixel coordinates of the same object point captured by the binocular camera on both images are used to calculate its corresponding 3D coordinates. Some fixed scene feature points are selected as common viewpoints, and their pixel coordinates are found on both images, with their corresponding 3D coordinates calculated. By comparing the changes in the 3D coordinates of the common viewpoints captured at adjacent time points, the camera pose change is estimated. Commonly used algorithms include the Iterative Closest Point (ICP) algorithm and nonlinear optimization algorithms. Finally, using the estimated camera pose, the coordinates of the currently captured object point are transformed to obtain its corresponding pixel coordinates on the other camera image, thus obtaining the camera's extrinsic parameter matrix. However, existing technologies struggle to automatically detect and adjust extrinsic parameters using datum lines and vanishing points, while also having a small overlap between side-view cameras (adjacent cameras overlap by 10°), making it difficult to achieve multi-camera calibration using multi-view stereo vision. Summary of the Invention

[0004] Therefore, the application provides a camera external parameter detection and adjustment method, which is used to solve the problems that the prior art is difficult to automatically detect and adjust external parameters by using a reticle and a vanishing point, and it is difficult to realize multi-camera calibration by using multi-view stereo vision.

[0005] To solve the above problems, the application provides a camera external parameter detection and adjustment method, which comprises the following steps:

[0006] obtaining an original picture collected by a side-view camera;

[0007] determining a value of camera external parameter change by frame subtraction based on the original picture;

[0008] triggering an external parameter alarm when the value of the camera external parameter change is greater than a preset adjustment threshold, and adjusting a heading angle of the side-view camera through lane line adjustment and virtual camera adjustment.

[0009] In some possible implementation manners, the lane line adjustment comprises the following steps:

[0010] performing lane line detection based on the collected original picture to obtain a lane line detection result;

[0011] performing IPM projection based on the lane line detection result to obtain a point line of the lane line in a vehicle body coordinate system;

[0012] adjusting a yaw angle according to a rule that a vehicle is parallel to a lane line and a slope of the fitted straight line to complete the heading angle adjustment.

[0013] In some possible implementation manners, the step of determining the value of the camera external parameter change by frame subtraction based on the original picture comprises the following steps:

[0014] calibrating a vehicle body boundary of a side-view camera out-field calibration and an actual vehicle body boundary from the original picture;

[0015] determining a difference between areas of the vehicle body boundary of the out-field calibration and the actual vehicle body boundary as the value of the camera external parameter change based on the vehicle body boundary of the out-field calibration and the actual vehicle body boundary.

[0016] In some possible implementation manners, the step of performing lane line detection based on the collected original picture to obtain a lane line detection result comprises the following steps:

[0017] performing lane line detection on the original picture in an image space to obtain the lane line detection result.

[0018] In some possible implementation manners, the step of performing IPM projection based on the lane line detection result to obtain a point line of the lane line in a vehicle body coordinate system comprises the following steps:

[0019] obtaining lane line key points in an image space based on the lane line detection result;

[0020] projecting the lane line key points into a bird's eye view space to obtain a point line of the lane line in a body coordinate system.

[0021] In some possible implementation manners, the virtual camera adjustment comprises:

[0022] When the external parameter alarm is triggered, the virtual camera adjusts the heading angle according to a preset algorithm.

[0023] In some possible implementation manners, the side-view camera is installed according to a preset installation condition.

[0024] In another aspect, the present application also provides a camera external parameter detection and adjustment system, comprising:

[0025] a picture acquisition module configured to acquire an original picture collected by the side-view camera;

[0026] an external parameter value determination module configured to determine a value of a change in the camera external parameter based on the original picture through frame subtraction;

[0027] an external parameter value anomaly detection and adjustment module configured to trigger an external parameter alarm when the value of the change in the camera external parameter is greater than a preset adjustment threshold, and to adjust the heading angle of the side-view camera through lane line adjustment and virtual camera adjustment.

[0028] In another aspect, the present application also provides an electronic device, comprising a memory and a processor, wherein,

[0029] the memory is configured to store a program;

[0030] the processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps of the camera external parameter detection and adjustment method.

[0031] In another aspect, the present application also provides a computer readable storage medium for storing computer readable programs or instructions, which, when executed by a processor, can implement the steps of the camera external parameter detection and adjustment method.

[0032] Compared with the prior art, the present application has the beneficial effects including: the present application acquires the original pictures collected by the side-view cameras, then determines the value of the camera extrinsic parameter change based on the original pictures through frame subtraction, triggers the extrinsic parameter alarm when the value of the camera extrinsic parameter change is greater than the preset adjustment threshold, and adjusts the heading angle of the side-view camera through lane line adjustment and virtual camera adjustment for the camera with abnormal extrinsic parameters, thereby solving the problem that the prior art is difficult to automatically detect and adjust the extrinsic parameters with the help of the marking lines and vanishing points, and the overlapping angle between the side-view cameras is small, and it is difficult to realize multi-camera calibration with the help of multi-view stereovision. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A flowchart of an embodiment of the camera extrinsic parameter detection and adjustment method provided by the present application;

[0034] Figure 2 An embodiment of the vehicle body boundary calibration provided by the present application is shown in the schematic diagram;

[0035] Figure 3 An embodiment of the camera extrinsic parameter detection and adjustment system provided by the present application is shown in the structural schematic diagram;

[0036] Figure 4 An embodiment of the electronic device provided by the present application is shown in the structural schematic diagram. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative work fall within the scope of protection of the present application.

[0038] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented in no particular order, and the steps without logical context relationship can be reversed in order or implemented simultaneously. In addition, the person skilled in the art can add one or more other operations to the flowcharts or remove one or more operations from the flowcharts under the guidance of the content of the present application.

[0039] Some block diagrams shown in the drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0040] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all referring to a particular embodiment that is independent of all other embodiments. It is explicitly contemplated that embodiments described herein can be combined with each other in their individual aspects.

[0041] The embodiment of the application provides a camera extrinsic parameter detection adjustment method, which is described below.

[0042] Figure 1 An embodiment of the camera extrinsic parameter detection adjustment method provided by the application is shown in a flowchart, which comprises the following steps:

[0043] S101, acquiring an original picture collected by a side-view camera;

[0044] S102, determining a camera extrinsic parameter change value by frame subtraction based on the original picture;

[0045] S103, triggering an extrinsic parameter alarm when the camera extrinsic parameter change value is greater than a preset adjustment threshold, and adjusting a heading angle of the side-view camera through lane line adjustment and virtual camera adjustment.

[0046] Compared with the prior art, the application acquires an original picture collected by a side-view camera, then determines a camera extrinsic parameter change value by frame subtraction based on the original picture, triggers an extrinsic parameter alarm when the camera extrinsic parameter change value is greater than a preset adjustment threshold, and adjusts a heading angle of the side-view camera through lane line adjustment and virtual camera adjustment for a camera with an abnormal extrinsic parameter, thereby solving the problem that it is difficult to automatically detect and adjust an extrinsic parameter by means of a mark line and a vanishing point in the prior art, and it is difficult to realize multi-camera calibration by means of multi-view stereovision due to small overlapping angles between side-view cameras.

[0047] In a preferred embodiment of the application, the lane line adjustment comprises: performing lane line detection based on the acquired original picture to obtain a lane line detection result.

[0048] Performing IPM projection based on the lane line detection result to obtain a point column of the lane line in a vehicle body coordinate system.

[0049] Performing straight line fitting on the lane line based on the point column of the lane line in the vehicle body coordinate system to obtain a slope of a fitted straight line.

[0050] Adjusting a yaw angle according to a rule that a vehicle is parallel to a lane line and the slope of the fitted straight line to complete heading angle adjustment.

[0051] In specific embodiments, IPM projection is performed on the input lane line to obtain a corresponding lane line point series in the vehicle body coordinate system; a straight line is fitted to the lane line, and the slope k of the fitted straight line can obtain the error of the corresponding yaw angle θ = arctan(k); the distribution average of the heading angle error is used as the adjustment error value to obtain the adjustment matrix

[0052] The adjusted extrinsic rotation matrix is R3 = R1 R2, and the translation matrix remains unchanged.

[0053] It should be noted that the heading angle yaw of the marking line in the vehicle coordinate system has an error distribution, and under normal circumstances, this distribution is close to 0. When the camera extrinsic changes, the yaw angle distribution will change. This change Δyaw is the yaw angle that needs to be compensated and adjusted.

[0054] It should be further noted that the yaw angle is a yaw angle.

[0055] In the preferred embodiments of the present application, please refer to Figure 2 , Figure 2 An embodiment of the vehicle body boundary calibration provided by the present application is shown in the figure, and the value of the camera extrinsic change is determined based on the original picture through frame subtraction, including:

[0056] The original picture is calibrated to the side-view camera out-of-factory calibrated vehicle body boundary and the actual vehicle body boundary;

[0057] The difference between the area of the out-of-factory calibrated vehicle body boundary and the actual vehicle body boundary is determined based on the side-view camera out-of-factory calibrated vehicle body boundary and the actual vehicle body boundary, that is, the value of the camera extrinsic change.

[0058] In specific embodiments, the first curved boundary is the out-of-factory calibrated vehicle body boundary (image space), and when the camera angle is abnormal, the actual vehicle body boundary is the second boundary. When the absolute value of the area difference between the two exceeds a certain threshold, the camera installation angle is triggered.

[0059] Δs = |Sout-of-factory - current| (2-2-1-1)

[0060] When ΔS is greater than Sthreshold, it means that the camera extrinsic has changed greatly, and an alarm is issued; otherwise, it is normal. The solution of ΔS can save an image at the time of factory shipment, and the area of the vehicle body perception is subtracted to obtain the solution.

[0061] In the preferred embodiments of the present application, lane line detection is performed based on the collected original picture to obtain a lane line detection result, including:

[0062] The original picture is subjected to lane line detection in image space to obtain a lane line detection result.

[0063] In a preferred embodiment of the present application, the IPM projection based on the lane line detection result to obtain a point line of the lane line in the vehicle body coordinate system comprises:

[0064] Based on the lane line detection result, a lane line key point in the image space is obtained.

[0065] The lane line key point is projected into the bird's eye view space to obtain a point line of the lane line in the vehicle body coordinate system.

[0066] In a preferred embodiment of the present application, the virtual camera adjustment comprises:

[0067] When the external parameter alarm is triggered, the virtual camera adjusts the heading angle according to a preset algorithm.

[0068] In a preferred embodiment of the present application, the side-view camera is installed according to a preset installation condition.

[0069] In a specific embodiment, the present application makes a strict provision for the camera mounting bracket, which is as follows:

[0070]

[0071] In order to better implement the side-view camera external parameter abnormality detection method in the embodiment of the present application, on the basis of the side-view camera external parameter abnormality detection method, the present embodiment also provides a multi-task joint detection system. The multi-data source includes a first data source and at least one second data source, such as Figure 3 As shown in FIG. 3, a multi-task joint detection system 300 includes:

[0072] A picture acquisition module 301 acquires an original picture collected by a side-view camera.

[0073] An external parameter value determination module 302 determines a value of camera external parameter change based on the original picture through frame subtraction.

[0074] An external parameter value abnormality detection and adjustment module 303 triggers an external parameter alarm when the value of camera external parameter change is greater than a preset adjustment threshold, and adjusts the heading angle of the side-view camera through lane line adjustment and virtual camera adjustment.

[0075] The multi-task joint detection system 300 provided by the above embodiment can implement the technical solutions described in the side-view camera external parameter abnormality detection method embodiment. The principles of the implementation of the above modules or units can be referred to the corresponding content in the camera external parameter detection and adjustment method embodiment, which will not be described here.

[0076] AsFigure 4 The present application also provides an electronic device 400, as shown. The electronic device 400 includes a processor 401, a memory 402, and a display 403. Figure 4 Only some components of the electronic device 400 are shown, but it should be understood that all the shown components are not required, and more or less components can be implemented instead.

[0077] The processor 401 can be a central processing unit (CPU), a microprocessor, or other data processing chip in some embodiments, for running program codes or processing data stored in the memory 402, such as a camera extrinsic parameter detection and adjustment method in the present application.

[0078] In some embodiments, the processor 401 can be a single server or a group of servers. The group of servers can be centralized or distributed. In some embodiments, the processor 401 can be local or remote. In some embodiments, the processor 401 can be implemented on a cloud platform. In an embodiment, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-organizational cloud, a multi-cloud, or any combination thereof.

[0079] The memory 402 can be an internal storage unit of the electronic device 400 in some embodiments, such as a hard disk or a memory of the electronic device 400. The memory 402 can also be an external storage device of the electronic device 400 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 400.

[0080] Further, the memory 402 can include both the internal storage unit and the external storage device of the electronic device 400. The memory 402 is used to store application software and various data installed on the electronic device 400.

[0081] The display 403 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 403 is used to display information and visual user interfaces on the electronic device 400. The components 401-403 of the electronic device 400 communicate with each other through a system bus.

[0082] In some embodiments of the present application, when the processor 401 executes a camera extrinsic parameter detection and adjustment method in the memory 402, the following steps can be implemented:

[0083] obtain an original picture captured by the side-view camera;

[0084] determine a value of camera extrinsic parameter change by frame subtraction based on the original picture;

[0085] trigger an extrinsic parameter alarm when the value of the camera extrinsic parameter change is greater than a preset adjustment threshold, and adjust the heading angle of the side-view camera through lane line adjustment and virtual camera adjustment.

[0086] It should be understood that, in addition to the above functions, the processor 401 can also implement other functions when executing the camera extrinsic parameter detection and adjustment program in the memory 402. Details can be referred to the description of the corresponding method embodiments.

[0087] Further, the type of the electronic device 400 is not specifically limited, and the electronic device 400 can be a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop, or the like. Exemplary embodiments of the portable electronic device include, but are not limited to, a portable electronic device running an IOS, android, microsoft, or other operating system. The portable electronic device can also be other portable electronic devices, such as a laptop having a touch-sensitive surface (e.g., a touch panel). It should also be understood that, in some other embodiments of the present application, the electronic device 400 can also be a desktop computer having a touch-sensitive surface (e.g., a touch panel).

[0088] Correspondingly, the embodiments of the present application also provide a computer readable storage medium for storing computer readable programs or instructions, which are executed by a processor to implement the steps or functions in the above-mentioned side-view camera extrinsic parameter anomaly detection method.

[0089] Those skilled in the art can understand that all or part of the above-mentioned embodiment methods can be completed by a computer program to instruct related hardware (such as a processor, a controller, etc.) to complete. The computer program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0090] The above has carried out the detailed introduction to the camera external parameter detection adjustment method, the principle and the implementation mode of the application are described in this paper by applying specific examples, the above example is only used for helping understanding the method and the core idea of the application; Meanwhile, for the technical personnel in the neighborhood, according to the idea of the application, there will be changes in specific implementation mode and application range, the above is described, only for the preferred specific implementation mode of the application, but the protection scope of the application is not limited to this, any skilled person in the art can easily think of the changes or replacements within the technical range disclosed by the application, which should be covered in the protection scope of the application.

Claims

1. A camera extrinsic parameter detection adjustment method, characterized in that, The method comprises: acquiring an original picture collected by a side-view camera; determining a value of camera extrinsic parameter change through frame subtraction based on the original picture; triggering an extrinsic parameter alarm when the value of the camera extrinsic parameter change is greater than a preset adjustment threshold, and adjusting a heading angle of the side-view camera through lane line adjustment and virtual camera adjustment; the lane line adjustment comprises: detecting a lane line based on the acquired original picture to obtain a lane line detection result; projecting the lane line detection result to obtain a point column of the lane line in a vehicle body coordinate system based on IPM; fitting a straight line to the lane line based on the point column of the lane line in the vehicle body coordinate system to obtain a slope of the fitted straight line; adjusting a yaw angle according to a rule that a vehicle is parallel to a lane line and the slope of the fitted straight line to complete the heading angle adjustment; the determination of the value of the camera extrinsic parameter change through frame subtraction based on the original picture comprises: calibrating a side-view camera out-field calibration vehicle body boundary and an actual vehicle body boundary from the original picture; determining a difference between the area of the out-field calibration vehicle body boundary and the area of the actual vehicle body boundary as the value of the camera extrinsic parameter change based on the out-field calibration vehicle body boundary and the actual vehicle body boundary.

2. The camera extrinsic parameter detection and adjustment method of claim 1, wherein, the lane line detection based on the acquired original picture to obtain a lane line detection result comprises: detecting a lane line in an image space based on the original picture to obtain a lane line detection result.

3. The camera extrinsic parameter detection and adjustment method of claim 1, wherein, the IPM projection based on the lane line detection result to obtain a point column of the lane line in a vehicle body coordinate system comprises: obtaining lane line key points in an image space based on the lane line detection result; projecting the lane line key points to an aerial view space to obtain a point column of the lane line in the vehicle body coordinate system.

4. The camera extrinsic parameter detection and adjustment method of claim 1, wherein, the virtual camera adjustment comprises: adjusting the heading angle according to a preset algorithm when the extrinsic parameter alarm is triggered.

5. The camera extrinsic parameter detection and adjustment method of claim 1, wherein, the side-view camera is installed according to a preset installation condition.

6. An extrinsic parameter anomaly detection and adjustment system for a side-view camera, the system comprising: The system is used to execute the method of any one of claims 1-5, comprising: a picture acquisition module, which acquires an original picture collected by a side-view camera; an extrinsic parameter value determination module, which determines a value of camera extrinsic parameter change through frame subtraction based on the original picture; an extrinsic parameter value anomaly detection and adjustment module, which triggers an extrinsic parameter alarm when the value of the camera extrinsic parameter change is greater than a preset adjustment threshold, and adjusts a heading angle of the side-view camera through lane line adjustment and virtual camera adjustment.

7. An electronic device, comprising: comprise a memory and a processor, wherein the memory is used to store a program; the processor is coupled with the memory and is used to execute the program stored in the memory to implement the steps of the camera extrinsic parameter detection and adjustment method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, a computer-readable program or instruction is stored, and the program or instruction is executed by a processor to implement the steps of the camera extrinsic parameter detection and adjustment method in any one of claims 1-5.

Images