Camera online calibration method, device and vehicle

Abstract

The present application provides a camera online calibration method, device and vehicle, and relates to the field of camera calibration technology. When the target camera meets the online calibration conditions, the method obtains the current configuration information and historical calibration information of the target camera, and when it is determined based on the current configuration information and the historical calibration information that the target camera meets the continuous calibration conditions, the target camera is continuously calibrated based on the historical calibration information. Thus, when the target camera fails to exit the online calibration due to factors such as the road environment and the driving mode during the last online calibration process, the historical calibration information generated when the last online calibration exited can still be used during the current online calibration process for continuous calibration, that is, there is no need to perform the online calibration of the target camera from the beginning during the current online calibration process, which greatly improves the probability of successful online calibration of the target camera and reduces the computing power during the online calibration of the target camera.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and in particular to a camera online calibration method, device and vehicle. Background Art

[0002] Online camera calibration is an important process in machine vision products such as intelligent driving vehicles and mobile robots. By calibrating the camera online, the camera parameters can be dynamically adjusted and optimized to ensure that the camera can maintain high-precision performance in different environments and conditions.

[0003] At present, in the process of online camera calibration, factors such as road environment and driving mode cannot relatively continuously meet the camera online calibration conditions, resulting in frequent failures of camera online calibration, making it difficult to achieve successful camera calibration, and causing a significant increase in the computing power of camera online calibration. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a camera online calibration method, device and vehicle to solve the problems in the prior art that the success rate of camera online calibration is low and the computing power requirement is large.

[0005] To achieve the above technical objectives, the present application provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present specification provides a camera online calibration method, comprising:

[0007] When the target camera meets the online calibration condition, the current configuration information and historical calibration information of the target camera are obtained; the historical calibration information is generated when the target camera exits the last online calibration;

[0008] If it is determined based on the current configuration information and the historical calibration information that the target camera meets the continuous calibration condition, then in the current online calibration process, the target camera is continuously calibrated based on the historical calibration information.

[0009] In one embodiment, the historical calibration information includes the historical calibration state of the target camera when the last online calibration exited, and the historical configuration information of the target camera during the last online calibration process;

[0010] Determining whether the target camera meets a continuous calibration condition based on the current configuration information and the historical calibration information includes:

[0011] If the historical calibration status indicates that the online calibration of the target camera is not completed, determining whether the target camera meets the continuous calibration condition based on the comparison result between the current configuration information and the historical configuration information;

[0012] If the historical calibration status indicates that the online calibration of the target camera is completed, it is determined that the target camera does not meet the continued calibration condition.

[0013] In one implementation, the determining whether the target camera meets the continued calibration condition based on the comparison result between the current configuration information and the historical configuration information includes:

[0014] If the current configuration information is the same as the historical configuration information, it is determined that the target camera meets the continued calibration condition;

[0015] If the current configuration information is different from the historical configuration information, it is determined that the target camera does not meet the continued calibration condition.

[0016] In one embodiment, the historical calibration information includes historical intermediate state calibration data, and the historical intermediate state calibration data includes intermediate state calibration data of the target camera when the last online calibration exits;

[0017] The continuously calibrating the target camera based on the historical calibration information includes:

[0018] Based on the historical intermediate state calibration data, determining the missing calibration data of the target camera when the last online calibration was exited;

[0019] Based on the image data collected by the target camera during the current online calibration process, the missing calibration data is generated, and the generated missing calibration data is merged into the historical intermediate state calibration data to obtain current calibration data;

[0020] When the current calibration data meets the calibration completion condition, an online calibration result of the target camera is generated based on the current calibration data.

[0021] In one embodiment, it further includes:

[0022] When the target camera meets the online calibration exit condition and the current calibration data does not meet the calibration completion condition, acquiring the target calibration data in the current calibration data;

[0023] The target calibration data is used as the intermediate calibration data of the current online calibration.

[0024] In one embodiment, the current calibration data includes at least two levels of sub-calibration data; for two adjacent levels of sub-calibration data, the latter level of sub-calibration data corresponds to a plurality of previous level of sub-calibration data, and the latter level of sub-calibration data is generated based on the corresponding previous level of sub-calibration data;

[0025] The obtaining of target calibration data in the current calibration data includes:

[0026] If the current sub-calibration data is the last level of sub-calibration data, determining the current sub-calibration data as the target calibration data;

[0027] If the current sub-calibration data is not the last level sub-calibration data, whether to determine the current sub-calibration data as the target calibration data is determined based on the data state corresponding to the current sub-calibration data, and the data state corresponding to the current sub-calibration data represents whether the next level sub-calibration data corresponding to the current sub-calibration data exists in the current calibration data.

[0028] In one implementation, the determining whether to determine the current sub-calibration data as target calibration data based on the data state corresponding to the current sub-calibration data includes:

[0029] If the data state indicates that the subsequent level sub-calibration data corresponding to the current sub-calibration data does not exist, determining the current sub-calibration data as the target calibration data;

[0030] If the data state indicates that subsequent sub-calibration data corresponding to the current sub-calibration data exists, the current sub-calibration data is not determined as target calibration data.

[0031] In one implementation, after generating the online calibration result of the target camera based on the current calibration data, the method further includes:

[0032] The calibration parameters and / or calibration file of the target camera are updated based on the online calibration result.

[0033] In a second aspect, an embodiment of the present specification provides a camera online calibration device, including:

[0034] A first processing module is used to obtain current configuration information and historical calibration information of the target camera when the target camera meets the online calibration conditions; the historical calibration information is generated when the target camera exits the last online calibration;

[0035] The second processing module is configured to perform a continuous calibration on the target camera based on the historical calibration information during the current online calibration process if it is determined based on the current configuration information and the historical calibration information that the target camera meets the continuous calibration condition.

[0036] In a third aspect, an embodiment of the present specification provides a vehicle, including a camera and a camera online calibration device as described above.

[0037] In a fourth aspect, an embodiment of the present specification provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the camera online calibration method as described in any one of the above items is implemented.

[0038] In a fifth aspect, an embodiment of the present specification provides a computer program product or a computer program, wherein the computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium; the processor of the computer device reads the computer program from the computer-readable storage medium, and when the processor executes the computer program, it implements the camera online calibration method as described in any one of the above items.

[0039] It can be seen from the above technical solution that the embodiment of the present application provides a camera online calibration method, device and vehicle. When the target camera meets the online calibration conditions, the method obtains the current configuration information and historical calibration information of the target camera, and the historical calibration information is generated when the target camera exits the last online calibration, and determines whether the target camera meets the continuous calibration conditions based on the current configuration information and the historical calibration information, so that it can quickly and effectively determine whether the target camera meets the continuous calibration conditions, wherein, when the target camera meets the continuous calibration conditions, the target camera is continuously calibrated based on the historical calibration information. Thus, when the target camera fails to exit the online calibration due to factors such as the road environment and the driving mode during the last online calibration process, the historical calibration information generated when the last online calibration exited can still be used in the current online calibration process for continuous calibration, that is, there is no need to perform the online calibration of the target camera from the beginning during the current online calibration process, which greatly improves the probability of successful online calibration of the target camera and reduces the computing power during the online calibration of the target camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0041] Figure 1 A schematic diagram of an exemplary road environment.

[0042] Figure 2 A flowchart of a camera online calibration method provided for an implementation of this specification.

[0043] Figure 3A schematic diagram of the correspondence between the second-level sub-calibration data and the third-level sub-calibration data provided for the implementation manner of this specification.

[0044] Figure 4 A flowchart of another camera online calibration method provided for an implementation of this specification.

[0045] Figure 5 A schematic diagram of the structure of a camera online calibration device provided in an embodiment of the present specification. DETAILED DESCRIPTION

[0046] Unless otherwise defined, the technical terms or scientific terms used in the embodiments of this specification shall have the common meanings understood by persons with ordinary skills in the field to which this specification belongs. The words "first", "second" and similar words used in the embodiments of this specification do not indicate any order, quantity or importance, but are only used to avoid confusion of constituent elements.

[0047] Unless the context requires otherwise, throughout the specification, "plurality" means "at least two", and "including" is interpreted as an open, inclusive meaning, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" are intended to indicate that a particular feature, structure, material or characteristic associated with the embodiment or example is included in at least one embodiment or example of the specification. The schematic representation of the above terms does not necessarily refer to the same embodiment or example.

[0048] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0049] Overview

[0050] As described in the background technology, online camera calibration is an important process in machine vision products such as intelligent driving vehicles and mobile robots. By calibrating the camera online, the camera parameters can be dynamically adjusted and optimized to ensure that the camera can maintain high-precision performance in different environments and conditions.

[0051] At present, in the process of online calibration of the camera, the camera online calibration conditions are usually not met relatively continuously due to factors such as road environment and driving mode, resulting in frequent failures of the camera online calibration. Among them, when the camera online calibration fails, that is, the camera online calibration is not completed and abnormally exits, it waits for the next time when the online calibration conditions are met to re-enter and pull up the camera online calibration process to perform the camera online calibration. This cycle is repeated, making it difficult to achieve successful camera calibration, and causing a great increase in the computing power of the camera online calibration.

[0052] For example, Figure 1 Taking the road environment shown in the figure as an example, the vehicle's driving section is divided into two types: the first section and the second section ( Figure 1 In the example, only two first sections and two second sections are used as examples), each first section can only complete 80% of the feature data accumulation during the target online calibration operation, and each second section exits the camera online calibration operation because the external road environment does not meet the camera online calibration requirements for a period of time. Each time the first section is entered, the camera online calibration operation needs to be performed from the beginning, and each time the calibration fails due to insufficient feature data accumulation, so that the camera cannot be successfully calibrated in the entire driving section of the vehicle, and it requires a huge amount of computing power.

[0053] In order to solve the problem that the traditional method cannot effectively improve the low success rate of camera online calibration and the large computing power requirement, in the technical solution of the present application, the camera online calibration method obtains the current configuration information and historical calibration information of the target camera when the target camera meets the online calibration conditions, and the historical calibration information is generated when the target camera exits the last online calibration, and determines whether the target camera meets the continuous calibration conditions based on the current configuration information and the historical calibration information, so that it can quickly and effectively determine whether the target camera meets the continuous calibration conditions, wherein, when the target camera meets the continuous calibration conditions, the target camera is continuously calibrated based on the historical calibration information. Thus, when the target camera fails to exit the online calibration due to factors such as the road environment and the driving mode during the last online calibration process, the historical calibration information generated when the last online calibration exits can still be used in the current online calibration process for continuous calibration, that is, there is no need to perform the online calibration of the target camera from the beginning during the current online calibration process, which greatly improves the probability of the target camera's online calibration success and reduces the computing power during the target camera's online calibration process.

[0054] Based on the above-mentioned inventive concept, the camera online calibration method provided in the implementation manner of this specification is exemplarily described below.

[0055] Exemplary Methods

[0056] The embodiments of this specification provide a camera online calibration method, such as Figure 2 As shown, including:

[0057] S201. When a target camera meets an online calibration condition, obtain current configuration information and historical calibration information of the target camera; the historical calibration information is generated when the target camera exits the last online calibration.

[0058] Specifically, the target camera may be a camera to be calibrated online. The online calibration conditions may be set according to actual needs. For example, the driving mode of the vehicle where the target camera is located may be a human driving mode; the road environment of the driving route of the vehicle where the target camera is located may also meet preset conditions, for example, the straightness of the road is greater than or equal to the preset straightness, the slope of the road is less than or equal to the preset slope, the flatness of the road is greater than or equal to the preset flatness, the number of lane lines on the road is greater than or equal to the preset number of lane lines, the lane lines of the road are clearly visible (for example, the contrast of the lane lines can be determined by image recognition), and the number of environmental features on both sides of the road (such as static features such as trees and buildings) is greater than or equal to the predetermined number of features , the illumination intensity is greater than or equal to the predetermined intensity, etc.; it may also include that the driving state of the vehicle where the target camera is located meets the preset state, for example, the driving speed is in the preset speed range (such as 30km / h~80km / h), driving straight in the middle of the lane line, the turning frequency is lower than the preset turning frequency, and the frequency of rapid acceleration and deceleration is lower than the preset acceleration and deceleration frequency; it may also include meeting the preset calibration times or calibration timing, for example, within a life cycle from the start to the shutdown of the intelligent driving controller, the number of successful online calibrations of the target camera is less than or equal to the preset number (such as once), and the current moment is within the effective duration of the last online calibration of the target camera (such as 24h). It is understandable that after the software and / or hardware of the target camera is updated, the effective duration of the last online calibration of the target camera can be cleared.

[0059] The current configuration information of the target camera is the software and / or hardware configuration information of the target camera at the current moment. For example, the hardware configuration information of the target camera may include the SN (Serial Number), model, status information, etc. of each module of the target camera, and may also include the SN, model, status information, etc. of the intelligent driving controller. The software configuration information may include the version number, functional module version number, upgrade and update status, running status information, etc. of the intelligent driving software in the intelligent driving controller, and may also include the version number, upgrade and update status, running status information, etc. of the calibration software.

[0060] The historical calibration information of the target camera is generated when the target camera exits the last online calibration, for example, it may include the historical configuration information of the target camera during the last calibration process, it may also include the historical calibration state of the target camera when the last online calibration exits, and it may also include the intermediate state calibration data of the target camera when the last online calibration exits. Among them, the calibration state of the target camera may include online calibration completed and online calibration incomplete, and the intermediate state calibration data may include all or part of the feature data generated from the start time of online calibration to the exit time of online calibration for determining the online calibration result of the target camera, for example, it may include a lane line detection point set, a straight line equation for lane line detection, a vanishing point, an optical flow matching feature point, matching feature points of the common viewing area of ​​two cameras, initial calibration parameters obtained by calculating the image of a predetermined number of frames, and phased calibration parameters obtained by calibration parameter statistics and voting, etc.

[0061] During implementation, when it is determined that the target camera meets the online calibration conditions, the current configuration information and historical calibration information of the target camera may be obtained to determine whether the target camera meets the subsequent calibration conditions.

[0062] S202: If it is determined based on the current configuration information and the historical calibration information that the target camera meets the continuous calibration condition, then in the current online calibration process, the target camera is continuously calibrated based on the historical calibration information.

[0063] Specifically, whether the target camera meets the continued calibration condition can be determined based on the current configuration information and historical calibration information of the target camera. For example, when the historical calibration information indicates that the online calibration of the target camera was not completed when the last online calibration was exited, and the current configuration information is the same as the historical configuration information of the target camera during the last online calibration process, it is determined that the continued calibration condition is met; otherwise, it is determined that the continued calibration condition is not met, thereby being able to quickly and effectively determine whether the target camera meets the continued calibration condition.

[0064] Wherein, when the target camera does not meet the conditions for continued calibration, the target camera may be recalibrated online during the current online calibration process.

[0065] When the target camera meets the conditions for continued calibration, during the current online calibration process, the target camera can be calibrated continuously based on the historical calibration information of the target camera. For example, the historical calibration information can be substituted into the online calibration process of the target camera to realize continued calibration of the target camera. The continued calibration means continuing to calibrate the target camera online based on the intermediate state calibration data when the target camera exited the last online calibration.

[0066] Therefore, when the target camera exits the online calibration due to failure of online calibration in the last online calibration process due to factors such as road environment and driving mode, the historical calibration information generated when the last online calibration exited can still be used for continued calibration in the current online calibration process. That is, there is no need to perform the online calibration of the target camera from the beginning in the current online calibration process, which greatly improves the probability of successful online calibration of the target camera and reduces the computing power in the online calibration process of the target camera.

[0067] In a feasible implementation manner, the historical calibration information includes the historical calibration state of the target camera when the last online calibration exited, and the historical configuration information of the target camera during the last online calibration process;

[0068] Determining whether the target camera meets a continuous calibration condition based on the current configuration information and the historical calibration information includes:

[0069] If the historical calibration status indicates that the online calibration of the target camera is not completed, determining whether the target camera meets the continuous calibration condition based on the comparison result between the current configuration information and the historical configuration information;

[0070] If the historical calibration status indicates that the online calibration of the target camera is completed, it is determined that the target camera does not meet the continued calibration condition.

[0071] Specifically, the historical calibration information may include the historical calibration state of the target camera when the last online calibration exits, and may also include historical configuration information of the target camera during the last online calibration process.

[0072] During implementation, before each online calibration of the target camera, the configuration information of the target camera during the current online calibration process can be obtained and stored, so as to determine whether the current online calibration meets the conditions for continued calibration, and, when the current online calibration fails, to determine whether the target camera meets the conditions for continued calibration during the next online calibration.

[0073] At the same time, each time the online calibration of the target camera is exited, the calibration status of the current online calibration can be determined to facilitate determination of whether the calibration parameters and / or calibration files of the target camera need to be updated, and to facilitate determination of whether the target camera meets the conditions for continued calibration during the next online calibration.

[0074] Among them, when the historical calibration state of the target camera indicates that the online calibration of the target camera was not completed when the last online calibration was exited, the current configuration information of the target camera and the historical configuration information of the target camera during the last online calibration process can be further compared to determine whether the target camera meets the conditions for continued calibration based on the comparison result.

[0075] Therefore, according to the historical calibration state, current configuration information and historical configuration information of the target camera, the validity of the judgment result of whether the target camera meets the subsequent calibration conditions can be effectively guaranteed.

[0076] In a feasible implementation manner, the determining whether the target camera meets the continued calibration condition based on the comparison result between the current configuration information and the historical configuration information includes:

[0077] If the current configuration information is the same as the historical configuration information, it is determined that the target camera meets the continued calibration condition;

[0078] If the current configuration information is different from the historical configuration information, it is determined that the target camera does not meet the continued calibration condition.

[0079] Specifically, when the current configuration information of the target camera is the same as the historical configuration information, it indicates that the software and hardware of the target camera and the calibration device of the target camera (such as the intelligent driving controller) have not changed compared with the last online calibration of the target camera. At this time, it can be determined that the target camera meets the conditions for continued calibration, that is, in the current online calibration process of the target camera, the target camera can be calibrated based on the intermediate calibration data generated by the target camera in the last online calibration process.

[0080] When the current configuration information of the target camera is different from the historical configuration information, it indicates that the software and hardware of the target camera and / or the software and hardware of the calibration device of the target camera have changed compared to the last online calibration of the target camera. At this time, it can be determined that the target camera does not meet the conditions for continued calibration, thereby effectively avoiding the influence of the software and hardware differences of the target camera and / or the calibration device of the target camera on the overall calibration result of the target camera, further improving the effectiveness of the judgment result of whether the target camera meets the conditions for continued calibration.

[0081] In a feasible implementation manner, the historical calibration information includes historical intermediate state calibration data, and the historical intermediate state calibration data includes intermediate state calibration data of the target camera when the last online calibration is exited;

[0082] The continuously calibrating the target camera based on the historical calibration information includes:

[0083] Based on the historical intermediate state calibration data, determining the missing calibration data of the target camera when the last online calibration was exited;

[0084] Based on the image data collected by the target camera during the current online calibration process, the missing calibration data is generated, and the generated missing calibration data is merged into the historical intermediate state calibration data to obtain current calibration data;

[0085] When the current calibration data meets the calibration completion condition, an online calibration result of the target camera is generated based on the current calibration data.

[0086] Specifically, the historical calibration information of the target camera may also include intermediate state calibration data of the target camera when the last online calibration was exited, that is, historical intermediate state calibration data.

[0087] In implementation, the target camera can be continuously calibrated based on the historical intermediate state calibration data. For example, the missing calibration data of the target camera when the last online calibration exits can be determined based on the historical intermediate state calibration data. The missing calibration data can include feature data other than the historical intermediate state calibration data in the feature data required when the online calibration of the target camera is completed.

[0088] The current calibration data may include feature data used to determine an online calibration result of the target camera during the current online calibration process.

[0089] Among them, the missing calibration data can be generated based on the image data collected by the target camera in the current online calibration process, and the generated missing calibration data can be merged into the historical intermediate state calibration data to obtain the feature data required by the target camera in the current online calibration process, and use it as the current calibration data, so that on the basis of the feature data accumulated in the last online calibration process of the target camera, the feature data used for online calibration of the target camera can be continuously obtained, thereby effectively reducing the computing power in the camera online calibration process and reducing the probability of camera online calibration failure due to insufficient accumulation of feature data.

[0090] The calibration completion condition can be set according to actual needs. For example, the amount of target data in the current calibration data reaches a predetermined amount. The target data may include data used to generate the final online calibration result.

[0091] When the current calibration data meets the calibration completion conditions, an online calibration result of the target camera can be generated based on the current calibration data. The online calibration result of the target camera may include the parameter value of each external parameter of the target camera, and may also include the parameter values ​​of multiple calibration parameters. The calibration parameters may be parameters for evaluating the accuracy and reliability of the online calibration of the target camera, for example, may include error codes, reprojection error values, verification status codes, confidence levels, etc.

[0092] In a feasible implementation, it also includes:

[0093] When the target camera meets the online calibration exit condition and the current calibration data does not meet the calibration completion condition, acquiring the target calibration data in the current calibration data;

[0094] The target calibration data is used as the intermediate calibration data of the current online calibration.

[0095] Specifically, the online calibration exit condition may include that the current calibration data satisfies the calibration completion condition, and may also include that the online calibration condition is not satisfied, for example, the driving mode of the vehicle where the target camera is located is switched to the intelligent driving mode, or the road environment of the driving route of the vehicle where the target camera is located does not satisfy the preset condition for a preset period of time, or the driving state of the vehicle where the target camera is located does not satisfy the preset state for a preset period of time.

[0096] When the target camera meets the online calibration exit condition and the current calibration data does not meet the calibration completion condition, it indicates that the online calibration of the target camera has failed, that is, the online calibration is exited when the online calibration is not completed. At this time, the target calibration data in the current calibration data can be obtained. The target calibration data in the current calibration data can include feature data in the current calibration data that can be directly used for subsequent calibration, for example, it can include vanishing points, optical flow matching feature points, matching feature points in the common viewing area of ​​two cameras, initial calibration parameters calculated from images of a predetermined number of frames, and phased calibration parameters obtained by calibration parameter statistics and voting, etc. It can be understood that the target calibration data can also include all feature data in the current calibration data.

[0097] During implementation, the target calibration data can be saved as the intermediate calibration data of the current online calibration, so that during the next online calibration of the target camera, the target camera can be calibrated based on the intermediate calibration data. Thus, through the accumulation of the intermediate calibration data, the probability of successful online calibration of the target camera can be effectively improved, and the computing power during the online calibration of the target camera can be reduced.

[0098] In a feasible implementation manner, the current calibration data includes at least two levels of sub-calibration data; for two adjacent levels of sub-calibration data, the latter level of sub-calibration data corresponds to a plurality of previous level of sub-calibration data, and the latter level of sub-calibration data is generated based on the corresponding previous level of sub-calibration data;

[0099] The obtaining of target calibration data in the current calibration data includes:

[0100] If the current sub-calibration data is the last level of sub-calibration data, determining the current sub-calibration data as the target calibration data;

[0101] If the current sub-calibration data is not the last level sub-calibration data, whether to determine the current sub-calibration data as the target calibration data is determined based on the data state corresponding to the current sub-calibration data, and the data state corresponding to the current sub-calibration data represents whether the next level sub-calibration data corresponding to the current sub-calibration data exists in the current calibration data.

[0102] Specifically, all intermediate calibration data generated during the online calibration process, that is, all intermediate calibration data generated from the start time of the online calibration to the completion time of the online calibration, can be divided into at least two levels of sub-calibration data. In the multi-level sub-calibration data, for any adjacent two-level sub-calibration data, the latter-level sub-calibration data can correspond to a plurality of previous-level sub-calibration data, that is, the latter-level sub-calibration data can be generated based on the corresponding previous-level sub-calibration data. It is understandable that when the number of previous-level sub-calibration data reaches a specified number, the corresponding latter-level sub-calibration data is generated based on the specified number of previous-level sub-calibration data.

[0103] In implementation, the current sub-calibration data may be any sub-calibration data in the current calibration data. In the case where the current calibration data includes at least two levels of sub-calibration data, it may be determined in sequence whether each sub-calibration data needs to be saved as target calibration data.

[0104] Wherein, when the current sub-calibration data is the last level of sub-calibration data, the current sub-calibration data may be determined as the target calibration data.

[0105] In the case where the current sub-calibration data is not the last level sub-calibration data, it can be determined whether to determine the current sub-calibration data as the target calibration data based on the data state corresponding to the current sub-calibration data. The data state corresponding to the current sub-calibration data can be used to characterize whether the next level sub-calibration data corresponding to the current sub-calibration data exists in the current calibration data, so that when the target camera fails to perform online calibration and exits online calibration, the amount of saved intermediate calibration data can be effectively reduced, thereby saving storage resources and the computational complexity in the subsequent calibration process.

[0106] In a feasible implementation manner, the determining whether to determine the current sub-calibration data as the target calibration data based on the data state corresponding to the current sub-calibration data includes:

[0107] If the data state indicates that the subsequent level sub-calibration data corresponding to the current sub-calibration data does not exist, determining the current sub-calibration data as the target calibration data;

[0108] If the data state indicates that subsequent sub-calibration data corresponding to the current sub-calibration data exists, the current sub-calibration data is not determined as target calibration data.

[0109] Specifically, when the data state corresponding to the current sub-calibration data indicates that the subsequent sub-calibration data corresponding to the current sub-calibration data does not exist, it indicates that the amount of the current sub-calibration data is insufficient to generate the subsequent sub-calibration data, and the current sub-calibration data can be determined as the target calibration data.

[0110] When the data state corresponding to the current sub-calibration data indicates that the subsequent sub-calibration data corresponding to the current sub-calibration data exists, it indicates that the subsequent sub-calibration data has been generated based on the current sub-calibration data. In the subsequent calibration process, the next step of calibration can be directly based on the subsequent sub-calibration data corresponding to the current sub-calibration data. At this time, the current sub-calibration data may not be determined as the target calibration data, so that the target camera can be effectively calibrated through the target calibration data, and the amount of intermediate calibration data that needs to be saved is further reduced.

[0111] It is understandable that in the process of determining the target calibration data, the target calibration data can be determined only from the sub-calibration data of the target level. For example, it can be divided into three levels of sub-calibration data according to the order of the calibration steps, namely the first-level sub-calibration data, the second-level sub-calibration data and the third-level sub-calibration data from front to back. The first-level sub-calibration data may include basic feature data such as lane line detection point set and straight line equation of lane line detection. The second-level sub-calibration data may include vanishing points, optical flow matching feature points, matching feature points of the common viewing area of ​​two cameras, initial calibration parameters calculated from images of a predetermined number of frames, and other feature data generated based on basic feature data. The third-level sub-calibration data may include staged calibration parameters obtained by statistics and voting on the initial calibration parameters in the second-level sub-calibration data. In implementation, the target calibration data can be determined only from the second-level sub-calibration data and the third-level sub-calibration data.

[0112] For example, the correspondence between the second-level sub-calibration data and the third-level sub-calibration data can be as follows: Figure 3 The second-level sub-calibration data may include multiple groups of initial calibration parameters, and the jth group of initial calibration parameters may be expressed as P j , any set of initial calibration parameters in the second-level sub-calibration data can be generated by the lane line detection point set obtained by n-frame image detection, the straight line equation of lane line detection, etc.; the third-level sub-calibration data can include multiple sets of phased calibration parameters, and the i-th group of phased calibration parameters can be expressed as PP iAny set of phased calibration parameters in the third-level sub-calibration data can be generated by counting and voting the m sets of initial calibration parameters in the second-level sub-calibration data. When the number of phased calibration parameters in the third-level sub-calibration data reaches a preset number (e.g., 100 sets), it can be determined that the calibration completion condition is met. When the target camera fails to exit the online calibration, all the phased calibration parameters (e.g., the number is t) in the third-level sub-calibration data and the initial calibration parameters other than the m*t sets of initial calibration parameters in the second-level sub-calibration data that have been used to generate the phased calibration parameters can be saved as intermediate calibration data to facilitate the continued calibration of the target camera.

[0113] Assume that the intermediate state calibration data includes 79 sets of phased calibration parameters PP1~PP 79 , and m-2 groups of initial calibration parameters P 79m+1 ~P 80m-2 In the process of subsequent calibration of the target camera, the initial calibration parameters obtained by solving the first set of images can be written into P 80m-1 , and write the initial calibration parameters obtained by solving the second set of images into P 80m , so that P 79m+1 ~P 80m Generate PP 80 At this point, only 20 sets of phased calibration parameters need to be accumulated to complete the online calibration of the target camera. Figure 1 Taking the road environment shown as an example, when the vehicle enters the first section for the first time to perform online calibration on the target camera, 80% of the feature data can be accumulated. Although the online calibration is exited when encountering the second section, the vehicle only needs to accumulate 20% of the feature data to complete the online calibration when entering the first section again. That is, when the vehicle enters the first section again, the online calibration of the target camera can be completed without completing the section, which greatly improves the probability of successful online calibration of the target camera and reduces the computing power during the online calibration of the target camera. As a result, the stringent requirements of the target camera online calibration on the road environment are greatly reduced.

[0114] In a feasible implementation manner, after generating the online calibration result of the target camera based on the current calibration data, the method further includes:

[0115] The calibration parameters and / or calibration file of the target camera are updated based on the online calibration result.

[0116] Specifically, after generating the online calibration result of the target camera based on the current calibration data, the calibration parameters and / or calibration file of the target camera can be updated based on the online calibration result to complete the online calibration of the target camera. At the same time, the calibration state of the target camera can also be updated to online calibration completion, and the time of online calibration completion can be recorded, so as to determine whether the online calibration conditions are met and whether the continuous calibration conditions are met during the next online calibration process.

[0117] The following is an example of an optional implementation of the camera online calibration method of the present invention. Figure 4 ,The camera online calibration method includes:

[0118] S401, when the target camera meets the online calibration conditions, the target camera online calibration process is started;

[0119] S402, obtaining current configuration information of the target camera and historical calibration status of the target camera;

[0120] S403, determining whether the historical calibration status indicates that the last online calibration has been completed, if so, executing step S406, if not, executing step S404;

[0121] S404, determining whether the current configuration information of the target camera is the same as the historical configuration information, if so, executing step S405, if not, executing step S406;

[0122] S405, performing a subsequent calibration based on the historical intermediate state calibration data, and executing step S407;

[0123] S406, restarting a new online calibration;

[0124] S407, determining whether the target camera meets the online calibration exit condition, if yes, executing step S408, if no, repeating step S407;

[0125] S408, determining whether the online calibration has been completed, if not, executing step S409, if yes, executing step S410;

[0126] S409, saving the intermediate calibration data of the current online calibration, and setting the calibration state of the current online calibration to unfinished, and executing step S401;

[0127] S410, saving the calibration result and process information of the current online calibration, and setting the calibration status of the current online calibration to completed;

[0128] S411: Wait for the next restart of the intelligent driving controller, and execute step S401 after the restart.

[0129] Exemplary Devices

[0130] In an exemplary embodiment of the present specification, a camera online calibration device is also provided, such as Figure 5 As shown, including:

[0131] The first processing module 501 is used to obtain the current configuration information and historical calibration information of the target camera when the target camera meets the online calibration conditions; the historical calibration information is generated when the target camera exits the last online calibration;

[0132] The second processing module 502 is configured to perform a continuous calibration on the target camera based on the historical calibration information during the current online calibration process if it is determined based on the current configuration information and the historical calibration information that the target camera meets the continuous calibration condition.

[0133] In a feasible implementation manner, the historical calibration information includes the historical calibration state of the target camera when the last online calibration exited, and the historical configuration information of the target camera during the last online calibration process;

[0134] The second processing module 502 is specifically used for:

[0135] If the historical calibration status indicates that the online calibration of the target camera is not completed, determining whether the target camera meets the continuous calibration condition based on the comparison result between the current configuration information and the historical configuration information;

[0136] If the historical calibration status indicates that the online calibration of the target camera is completed, it is determined that the target camera does not meet the continued calibration condition.

[0137] In a feasible implementation manner, the second processing module 502 is specifically configured to:

[0138] If the current configuration information is the same as the historical configuration information, it is determined that the target camera meets the continued calibration condition;

[0139] If the current configuration information is different from the historical configuration information, it is determined that the target camera does not meet the continued calibration condition.

[0140] In a feasible implementation manner, the historical calibration information includes historical intermediate state calibration data, and the historical intermediate state calibration data includes intermediate state calibration data of the target camera when the last online calibration is exited;

[0141] The second processing module 502 is specifically used for:

[0142] Based on the historical intermediate state calibration data, determining the missing calibration data of the target camera when the last online calibration was exited;

[0143] Based on the image data collected by the target camera during the current online calibration process, the missing calibration data is generated, and the generated missing calibration data is merged into the historical intermediate state calibration data to obtain current calibration data;

[0144] When the current calibration data meets the calibration completion condition, an online calibration result of the target camera is generated based on the current calibration data.

[0145] In a feasible implementation manner, the second processing module 502 is further configured to:

[0146] When the target camera meets the online calibration exit condition and the current calibration data does not meet the calibration completion condition, acquiring the target calibration data in the current calibration data;

[0147] The target calibration data is used as the intermediate calibration data of the current online calibration.

[0148] In a feasible implementation manner, the current calibration data includes at least two levels of sub-calibration data; for two adjacent levels of sub-calibration data, the latter level of sub-calibration data corresponds to a plurality of previous level of sub-calibration data, and the latter level of sub-calibration data is generated based on the corresponding previous level of sub-calibration data;

[0149] The second processing module 502 is specifically used for:

[0150] If the current sub-calibration data is the last level of sub-calibration data, determining the current sub-calibration data as the target calibration data;

[0151] If the current sub-calibration data is not the last level sub-calibration data, whether to determine the current sub-calibration data as the target calibration data is determined based on the data state corresponding to the current sub-calibration data, and the data state corresponding to the current sub-calibration data represents whether the next level sub-calibration data corresponding to the current sub-calibration data exists in the current calibration data.

[0152] In a feasible implementation manner, the second processing module 502 is specifically configured to:

[0153] If the data state indicates that the subsequent level sub-calibration data corresponding to the current sub-calibration data does not exist, determining the current sub-calibration data as the target calibration data;

[0154] If the data state indicates that subsequent sub-calibration data corresponding to the current sub-calibration data exists, the current sub-calibration data is not determined as target calibration data.

[0155] In a feasible implementation manner, the second processing module 502 is further configured to:

[0156] The calibration parameters and / or calibration file of the target camera are updated based on the online calibration result.

[0157] The camera online calibration device provided in this embodiment belongs to the same application concept as the camera online calibration method provided in the above embodiments of this application, and can execute the camera online calibration method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the camera online calibration method. For technical details not fully described in this embodiment, please refer to the specific processing content of the camera online calibration method provided in the above embodiments of this application, and will not be repeated here.

[0158] Exemplary Devices

[0159] In an exemplary embodiment of the present specification, an electronic device is also provided, the electronic device comprising at least one processor and at least one memory, the memory storing a computer program, and the computer program, when executed by the processor, implements the camera online calibration method as described in any of the above embodiments.

[0160] Example Vehicles

[0161] In an exemplary embodiment of the present specification, a vehicle is further provided, comprising a camera and a camera online calibration device as described in any one of the above embodiments.

[0162] Exemplary computer program products and storage media

[0163] In addition to the above-mentioned methods and devices, the camera online calibration method provided in the embodiments of this specification may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the camera online calibration method according to various embodiments of this specification described in the above “Exemplary Method” section of this specification.

[0164] The computer program product can be written in any combination of one or more programming languages ​​to write program codes for executing the operations of the embodiments of this specification, and the programming languages ​​include object-oriented programming languages, such as Java, C++, etc., and also include conventional procedural programming languages, such as "C" language or similar programming languages.

[0165] In addition, an embodiment of the present specification also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the camera online calibration method according to various embodiments of the present specification described in the above "Exemplary Method" section of the present specification.

[0166] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this specification may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0167] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] The above-mentioned embodiments only express several implementation methods of this specification, and the descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the solutions provided by the embodiments of this specification. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of this specification, which all belong to the protection scope of this specification. Therefore, the protection scope of the patent of this specification shall be based on the attached claims.

Claims

1. A camera online calibration method, characterized in that: include: When the target camera meets the online calibration conditions, obtaining the current configuration information and historical calibration information of the target camera; The historical calibration information is generated when the target camera exits the last online calibration; If it is determined based on the current configuration information and the historical calibration information that the target camera meets the continuous calibration condition, then in the current online calibration process, the target camera is continuously calibrated based on the historical calibration information; Wherein, the historical calibration information includes historical intermediate state calibration data, and the historical intermediate state calibration data is the target calibration data in the current calibration data when the target camera's last online calibration meets the online calibration exit condition and does not meet the calibration completion condition; the current calibration data includes at least two levels of sub-calibration data; for two adjacent levels of sub-calibration data, the latter level of sub-calibration data corresponds to a plurality of previous level of sub-calibration data, and the latter level of sub-calibration data is generated based on the corresponding previous level of sub-calibration data; The process of determining the target calibration data in the current calibration data includes: if the current sub-calibration data is the last-level sub-calibration data, determining the current sub-calibration data as the target calibration data; if the current sub-calibration data is not the last-level sub-calibration data, determining whether to determine the current sub-calibration data as the target calibration data based on a data state corresponding to the current sub-calibration data, the data state corresponding to the current sub-calibration data representing whether the next-level sub-calibration data corresponding to the current sub-calibration data exists in the current calibration data.

2. The method according to claim 1, characterized in that The historical calibration information includes the historical calibration state of the target camera when the last online calibration was exited, and the historical configuration information of the target camera during the last online calibration process; Determining whether the target camera meets a continuous calibration condition based on the current configuration information and the historical calibration information includes: If the historical calibration status indicates that the online calibration of the target camera is not completed, determining whether the target camera meets the continuous calibration condition based on the comparison result between the current configuration information and the historical configuration information; If the historical calibration status indicates that the online calibration of the target camera is completed, it is determined that the target camera does not meet the continued calibration condition.

3. The method according to claim 2, characterized in that The determining whether the target camera meets the continuous calibration condition based on the comparison result between the current configuration information and the historical configuration information includes: If the current configuration information is the same as the historical configuration information, it is determined that the target camera meets the continued calibration condition; If the current configuration information is different from the historical configuration information, it is determined that the target camera does not meet the continued calibration condition.

4. The method according to any one of claims 1 to 3, characterized in that: The continuously calibrating the target camera based on the historical calibration information includes: Based on the historical intermediate state calibration data, determining the missing calibration data of the target camera when the last online calibration was exited; Based on the image data collected by the target camera during the current online calibration process, the missing calibration data is generated, and the generated missing calibration data is merged into the historical intermediate state calibration data to obtain current calibration data; When the current calibration data meets the calibration completion condition, an online calibration result of the target camera is generated based on the current calibration data.

5. The method according to claim 1, characterized in that The determining, based on the data state corresponding to the current sub-calibration data, whether to determine the current sub-calibration data as the target calibration data includes: If the data state indicates that the subsequent level sub-calibration data corresponding to the current sub-calibration data does not exist, determining the current sub-calibration data as the target calibration data; If the data state indicates that subsequent sub-calibration data corresponding to the current sub-calibration data exists, the current sub-calibration data is not determined as target calibration data.

6. The method according to claim 4, characterized in that After generating the online calibration result of the target camera based on the current calibration data, the method further includes: The calibration parameters and / or calibration file of the target camera are updated based on the online calibration result.

7. A camera online calibration device, characterized in that: include: A first processing module, configured to obtain current configuration information and historical calibration information of the target camera when the target camera meets the online calibration condition; The historical calibration information is generated when the target camera exits the last online calibration; A second processing module, configured to, if it is determined based on the current configuration information and the historical calibration information that the target camera meets the continuous calibration condition, then, in the current online calibration process, continuously calibrate the target camera based on the historical calibration information; Wherein, the historical calibration information includes historical intermediate state calibration data, and the historical intermediate state calibration data is the target calibration data in the current calibration data when the target camera's last online calibration meets the online calibration exit condition and does not meet the calibration completion condition; the current calibration data includes at least two levels of sub-calibration data; for two adjacent levels of sub-calibration data, the latter level of sub-calibration data corresponds to a plurality of previous level of sub-calibration data, and the latter level of sub-calibration data is generated based on the corresponding previous level of sub-calibration data; The process of determining the target calibration data in the current calibration data includes: if the current sub-calibration data is the last-level sub-calibration data, determining the current sub-calibration data as the target calibration data; if the current sub-calibration data is not the last-level sub-calibration data, determining whether to determine the current sub-calibration data as the target calibration data based on a data state corresponding to the current sub-calibration data, the data state corresponding to the current sub-calibration data representing whether the next-level sub-calibration data corresponding to the current sub-calibration data exists in the current calibration data.

8. A vehicle, characterized in that: It comprises a camera and the camera online calibration device as claimed in claim 7.

Images