Camera external parameter detection method and device, storage medium, terminal and computer program product

Abstract

The invention discloses a camera external parameter detection method and device, a storage medium, a terminal and a computer program product. The method comprises the following steps: determining a scene point cloud collected by a laser radar and a plurality of scene images collected by a plurality of cameras to be detected; for an overlapping area of each to-be-detected camera and an adjacent to-be-detected camera, determining a plurality of target points corresponding to each pixel point of the overlapping area in the scene point cloud and a first category and a second category of each target point based on external parameters of the to-be-detected camera and the category of each pixel point of the overlapping area; based on the first category and the second category of the plurality of target points, performing category consistency analysis to obtain a preliminary analysis result of the overlapping region; and detecting the accuracy of the external parameters of the to-be-detected camera based on the preliminary analysis result of each overlapping region of the to-be-detected camera and each adjacent to-be-detected camera. According to the scheme, the accuracy and reliability of external parameter detection of the camera can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of camera extrinsic parameter detection, and in particular to a camera extrinsic parameter detection method and device, storage medium, terminal, and computer program product. Background Art

[0002] In the intelligent driving system, the perception module is responsible for receiving the original information collected by the sensor, building a real-time environment model around the intelligent driving vehicle, and providing information about dynamic and static obstacles such as vehicles and pedestrians. The sensors deployed on the autonomous driving vehicle mainly include lidar, cameras, etc. Different sensors have different advantages in environmental modeling. For example, the camera can obtain rich color information, which is conducive to the classification and identification of objects and provides a basis for the vehicle driving plan.

[0003] Due to the limited field of view of a single camera, in order to reduce blind spots and increase the detection area, autonomous vehicles are usually equipped with multiple cameras. By calibrating the external parameters of each camera (used to represent the rotation and translation relationship between the camera coordinate system and the target coordinate system (usually the vehicle coordinate system)), the camera coordinate systems of each camera can be converted to a unified vehicle coordinate system. When the relative position of the camera and the vehicle changes (especially the direction), the camera's detection results will be abnormal in space, and the camera's external parameters need to be recalibrated.

[0004] Therefore, it is necessary to study how to accurately and reliably detect whether the camera's external parameters are accurate or abnormal to ensure the safety of autonomous driving. Summary of the invention

[0005] The technical problem solved by the embodiments of the present invention is how to accurately and reliably detect the accuracy of the camera extrinsic parameters on the autonomous driving vehicle.

[0006] To solve the above technical problems, an embodiment of the present invention provides a camera extrinsic parameter detection method, comprising the following steps: determining a scene point cloud collected by a laser radar and a plurality of scene images collected by a plurality of cameras to be tested, wherein the scene point cloud and the plurality of scene images are time-series synchronized, each camera to be tested has an adjacent relationship with at least one other camera to be tested, and the scene images collected by the adjacent cameras to be tested have an overlapping area, and each pixel point in the overlapping area has a respective category; for the overlapping area between each camera to be tested and its adjacent camera to be tested, based on the extrinsic parameters of the camera to be tested and the categories of each pixel point in the overlapping area, determining a plurality of target points corresponding to each pixel point in the overlapping area in the scene point cloud, and a first category and a second category of each target point, wherein the first category is a category corresponding to the camera to be tested, and the second category is a category corresponding to the adjacent camera to be tested; performing a category consistency analysis based on the first category and the second category of each of the plurality of target points to obtain a preliminary analysis result of the overlapping area; and detecting the accuracy of the extrinsic parameters of the camera to be tested based on the preliminary analysis results of each overlapping area between the camera to be tested and each of the adjacent cameras to be tested.

[0007] Optionally, the number of the first categories and the number of the second categories to which the multiple target points belong are both at least one; based on the first categories and the second categories of each of the multiple target points, a category consistency analysis is performed to obtain a preliminary analysis result of the overlapping area, including: for each first category to which the multiple target points belong, each target point belonging to the first category and the total number thereof among the multiple target points are determined, and the target points belonging to the first category are recorded as first target points; for each second category to which each first target point belongs, the number of first target points belonging to the second category among the first target points is counted, and recorded as the statistical number corresponding to the second category; based at least on the statistical number corresponding to each second category to which each first target point belongs and the total number, a score value corresponding to each second category is determined; the score values ​​of the various second categories to which the first target points belong are summed to obtain a total score of the first category; and a weighted operation is performed on the total scores of the various first categories to which the multiple target points belong to obtain a preliminary analysis result of the overlapping area.

[0008] Optionally, the scoring value corresponding to each second category is determined based at least on the statistical number corresponding to each second category to which the first target points belong and the total number, including: for each second category to which the first target points belong, calculating the ratio of the statistical number corresponding to the second category to the total number, and using the square of the ratio as the scoring value corresponding to the second category.

[0009] Optionally, the scoring value corresponding to each second category is determined based at least on the statistical quantity corresponding to each second category to which the first target points belong and the total quantity, including: sorting the statistical quantities corresponding to the various second categories to which the first target points belong from large to small to obtain the sorted second categories; for each sorted second category, determining the corresponding scoring coefficient based on the order to which the second category belongs, and calculating the product of the scoring coefficient and the statistical quantity corresponding to the second category, and using the ratio of the product to the total quantity as the scoring value corresponding to the second category.

[0010] Optionally, the method of detecting the accuracy of the external parameters of the camera to be tested based on the preliminary analysis results of the overlapping areas between the camera to be tested and each adjacent camera to be tested includes: for the overlapping area between the camera to be tested and each adjacent camera to be tested, counting the total number of the multiple target points corresponding to each pixel point in the overlapping area in the scene point cloud, and using the total number as the weight of the preliminary analysis result of the overlapping area; using the weight to perform a weighted operation on the preliminary analysis results of the overlapping areas between the camera to be tested and each adjacent camera to be tested to obtain a first weighted analysis result; and detecting the accuracy of the external parameters of the camera to be tested based at least on the first weighted analysis result.

[0011] Optionally, at least based on the first weighted analysis result, the accuracy of the external parameters of the camera to be tested is detected, including: using multiple different offset parameters to offset the external parameters of the camera to be tested respectively to obtain multiple offset external parameters of the camera to be tested; determining the corresponding second weighted analysis result based on each offset external parameter of the camera to be tested; determining the maximum value among the first weighted analysis result and each second weighted analysis result, and determining the offset parameter to which the maximum value belongs, recorded as a target offset parameter; if the target offset parameter is less than or equal to a preset offset parameter threshold, confirming that the external parameters of the camera to be tested are accurate; wherein, the scheme of determining the corresponding second weighted analysis result based on each offset external parameter of the camera to be tested is consistent with the scheme of determining the first weighted analysis result based on the external parameters of the camera to be tested.

[0012] Optionally, for each camera to be tested and its adjacent camera to be tested overlapping area, based on the extrinsic parameters of the camera to be tested and the categories of each pixel point in the overlapping area, determining a plurality of target points corresponding to each pixel point in the overlapping area in the scene point cloud, as well as the first category and the second category of each target point, comprises: using the extrinsic parameters of the camera to be tested to project the scene point cloud onto the scene image captured by the camera to be tested, to obtain a plurality of first projection points; using the extrinsic parameters of the adjacent camera to be tested to project the scene point cloud onto the scene image captured by the adjacent camera to be tested, to obtain a plurality of second projection points; determining the plurality of first The intersection of the projection point and the multiple second projection points, and the multiple original three-dimensional points corresponding to each projection point in the intersection in the scene point cloud are used as the multiple target points; for the first projection point corresponding to each target point, the first pixel point closest to the first projection point is determined in the overlapping area, and the category of the first pixel point is used as the first category of the target point relative to the camera to be tested; for the second projection point corresponding to each target point, the second pixel point closest to the second projection point is determined in the overlapping area, and the category of the second pixel point is used as the second category of the target point relative to the adjacent camera to be tested.

[0013] The embodiment of the present invention further provides an external parameter detection device for a camera, comprising: a scene data determination module, used to determine a scene point cloud collected by a laser radar and a plurality of scene images collected by a plurality of cameras to be tested, wherein the scene point cloud and the plurality of scene images are time-series synchronized, each camera to be tested has an adjacent relationship with at least one other camera to be tested, and the scene images collected by the adjacent cameras to be tested have an overlapping area, and each pixel point in the overlapping area has a respective category to which it belongs; a category determination module, used to determine, for each camera to be tested and its adjacent camera to be tested overlapping area, based on the external parameters of the camera to be tested and the category of each pixel point in the overlapping area; The method comprises the following steps: determining a plurality of target points corresponding to each pixel point in the overlapping area in the scene point cloud, and a first category and a second category of each target point, wherein the first category is a category corresponding to the camera to be tested, and the second category is a category corresponding to the adjacent camera to be tested; a category consistency analysis module, for performing category consistency analysis based on the first category and the second category of each of the plurality of target points to obtain a preliminary analysis result of the overlapping area; and an extrinsic parameter accuracy detection module, for detecting the accuracy of the extrinsic parameter of the camera to be tested based on the preliminary analysis results of each overlapping area between the camera to be tested and each adjacent camera to be tested.

[0014] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned camera external parameter detection method are executed.

[0015] An embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor runs the computer program, the steps of the above-mentioned camera external parameter detection method are executed.

[0016] An embodiment of the present invention further provides a computer program product, including a computer program, which executes the steps of the above-mentioned camera external parameter detection method when the computer program is executed by a processor.

[0017] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0018] In an embodiment of the present invention, the accuracy of the camera's extrinsic parameters is detected in combination with the scene point cloud collected by the laser radar and the overlapping area between the multiple scene images collected by the multiple cameras to be tested. Specifically: for the overlapping area between each camera to be tested and its adjacent camera to be tested, based on the extrinsic parameters of the camera to be tested and the categories of each pixel point in the overlapping area, the multiple target points corresponding to each pixel point in the overlapping area in the scene point cloud and their respective first and second categories are determined; then, the category consistency analysis is performed on each target point in the overlapping area, and according to the category consistency analysis results of each overlapping area between the camera to be tested and each adjacent camera to be tested, the accuracy of the extrinsic parameters of the camera to be tested is detected.

[0019] From the above, compared with the existing conventional scheme of using the data collected by the lidar and a single camera to perform camera extrinsic parameter detection, the scheme of the embodiment of the present invention comprehensively utilizes the scene point cloud collected by the lidar and the overlapping area image data of multiple cameras, which helps to avoid the problem of insufficient accuracy of extrinsic parameter detection caused by insufficient data volume, changes in the data collection environment, etc., so as to obtain more accurate and reliable detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of an external parameter detection method for a camera in an embodiment of the present invention;

[0021] Figure 2 yes Figure 1 A flowchart of a specific implementation of step S12;

[0022] Figure 3 yes Figure 1 A flowchart of a specific implementation of step S13;

[0023] Figure 4 yes Figure 1 A flowchart of a specific implementation of step S14;

[0024] Figure 5 It is a structural schematic diagram of an external parameter detection device for a camera in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] As mentioned in the background technology, in an autonomous driving system, when the relative position of the camera and the vehicle changes, the camera's detection results will be abnormal in spatial position, and the camera's external parameters need to be recalibrated. Therefore, it is necessary to study how to accurately and reliably detect whether the camera's external parameters are accurate or abnormal.

[0026] In the prior art, the camera's external parameters are usually detected using data collected by a laser radar and a single camera. For example, a conventional solution is to extract features from the point cloud collected by the laser radar (the external parameters of the laser radar have been accurately calibrated in advance) for a preset target (e.g., a checkerboard) and the image collected by the camera for the same target in a specific calibration environment, respectively, to obtain a point cloud feature point set and an image feature point set; then, the camera's external parameters (i.e., the external parameters to be detected) are used to project the point cloud feature point set to the image coordinate system where the scene image is located to obtain a point cloud projection feature point set; the overlap between the point cloud projection feature point set and the image feature point set in the image coordinate system is analyzed (for example, the average distance between each point cloud projection feature point and the image feature point can be calculated, and the inverse of the distance average is used as the overlap index); if the overlap is greater than a preset threshold, it can be confirmed that the camera's external parameters are accurate and valid.

[0027] However, the above-mentioned solutions that rely on external parameter detection using lidar and a single camera have limited data volume, high requirements for the data collection environment, and are easily affected by changes in the data collection environment. Therefore, there may be problems with insufficient accuracy and reliability of detection results due to insufficient data.

[0028] To solve the above technical problems, in an embodiment of the present invention, the accuracy of the camera's extrinsic parameters is detected in combination with the scene point cloud collected by the laser radar and the overlapping area between multiple scene images collected by multiple cameras to be tested. Specifically: for the overlapping area between each camera to be tested and its adjacent camera to be tested, based on the extrinsic parameters of the camera to be tested and the categories of each pixel point in the overlapping area, multiple target points corresponding to each pixel point in the overlapping area in the scene point cloud and their respective first and second categories are determined; then, a category consistency analysis is performed on each target point in the overlapping area, and based on the category consistency analysis results of each overlapping area between the camera to be tested and each adjacent camera to be tested, the accuracy of the extrinsic parameters of the camera to be tested is detected.

[0029] From the above, compared with the existing conventional scheme of using the data collected by the lidar and a single camera to perform camera extrinsic parameter detection, the scheme of the embodiment of the present invention comprehensively utilizes the scene point cloud collected by the lidar and the overlapping area image data of multiple cameras, which helps to avoid the problem of insufficient accuracy of extrinsic parameter detection caused by insufficient data volume, changes in the data collection environment, etc., so as to obtain more accurate and reliable detection results.

[0030] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0031] Reference Figure 1 , Figure 1 The flowchart of the external parameter detection method of a camera in an embodiment of the present invention is shown in FIG. The method can be applied to a terminal with point cloud and image data processing functions, such as a computer, a mobile phone, a tablet computer, a server, a cloud platform, etc., especially a vehicle-mounted terminal on an autonomous driving vehicle.

[0032] The camera external parameter detection method may include steps S11 to S13:

[0033] Step S11: determining a scene point cloud collected by a laser radar and a plurality of scene images collected by a plurality of cameras to be tested, wherein the scene point cloud and the plurality of scene images are time-series synchronized, each camera to be tested has an adjacent relationship with at least one other camera to be tested, and the scene images collected by adjacent cameras to be tested have an overlapping area, and each pixel point in the overlapping area has a category to which it belongs;

[0034] Step S12: for each camera to be tested and its adjacent camera to be tested overlapping area, based on the external parameters of the camera to be tested and the category of each pixel point in the overlapping area, determine a plurality of target points corresponding to each pixel point in the overlapping area in the scene point cloud, and a first category and a second category of each target point, wherein the first category is a category corresponding to the camera to be tested, and the second category is a category corresponding to the adjacent camera to be tested;

[0035] Step S13: performing category consistency analysis based on the first category and the second category of each of the plurality of target points to obtain a preliminary analysis result of the overlapping area;

[0036] Step S14: Based on the preliminary analysis results of the overlapping areas between the camera to be tested and each adjacent camera to be tested, the accuracy of the external parameters of the camera to be tested is detected.

[0037] The following is an explanation of the relevant concepts involved in this application.

[0038] Camera extrinsic parameters: Parameters representing the rotational relationship and translational relationship between the camera coordinate system and the target coordinate system (e.g., the vehicle coordinate system), which may include a rotation matrix and a translation vector. In some embodiments, a 6-DOF transformation matrix may be used to represent the camera extrinsic parameters.

[0039] Camera coordinate system: The origin is the optical center of the camera, the x-axis and y-axis are parallel to the X-axis and Y-axis of the image respectively, and the z-axis is the optical axis of the camera (which is perpendicular to the image plane). The intersection of the optical axis of the camera and the image plane is the origin of the image coordinate system. The image coordinate system is a two-dimensional rectangular coordinate system.

[0040] External parameters of the LiDAR: parameters that represent the rotational relationship and translational relationship between the LiDAR coordinate system and the target coordinate system (e.g., the vehicle coordinate system), which may include a rotation matrix and a translation vector.

[0041] LiDAR coordinate system: The origin is the center of mass or geometric center of the LiDAR or the laser beam emission center, the x-axis points horizontally to the front of the laser beam emission surface, the y-axis points horizontally to the left side of the LiDAR, and the z-axis points vertically upward.

[0042] Vehicle coordinate system: It can be a special coordinate system preset according to the actual application scenario and used to describe the movement of the vehicle. For example, when the vehicle is stationary on a horizontal road, the vehicle coordinate system can be determined by the following method: the origin is the center of mass or geometric center of the vehicle, the x-axis points horizontally to the front of the vehicle, the y-axis points horizontally to the left of the driver, and the z-axis points vertically upward.

[0043] In coordinate space transformation, the external parameters of the lidar can be used to realize the spatial transformation (or projection) between the lidar coordinate system and the vehicle coordinate system; the external parameters of the camera can be used to realize the spatial transformation between the vehicle coordinate system and the camera coordinate system; the intrinsic parameters of the camera can be used to realize the spatial transformation between the camera coordinate system and the image coordinate system.

[0044] In the specific implementation of step S11, the scene point cloud may be point cloud data collected by one or more laser radars at the same time for scenes in multiple directions (for example, 360° scanning directions) around the autonomous driving vehicle. In the case of using multiple laser radars, the scene point cloud may be fused point cloud data obtained after point cloud fusion processing of the original point clouds collected by each laser radar, wherein the multiple laser radars may be arranged in a circle around the top surface of the autonomous driving vehicle. The multiple cameras to be tested may include, but are not limited to, one or more front cameras (arranged at the front of the vehicle), one or more side-view cameras (arranged on the left and / or right side of the vehicle body), and one or more rear cameras (arranged at the rear of the vehicle).

[0045] In a specific implementation, the scene targeted by the scene point cloud overlaps at least partially with the scene targeted by each of the scene images. For the multiple cameras to be tested, each camera to be tested has an adjacent relationship with at least one other camera to be tested, and the scene images captured by the adjacent cameras to be tested have an overlapping area, and each pixel point in the overlapping area has a category to which it belongs.

[0046] In a specific implementation, each of the scene images may be segmented to determine the category to which each pixel in the image belongs. In a non-limiting manner, the category to which each pixel in the scene image belongs may be selected from but not limited to: motor vehicles, pedestrians, road surfaces, non-motor vehicles, buildings, and transportation facilities.

[0047] In the specific implementation of step S12, the overlapping area between each camera to be tested and its adjacent cameras to be tested specifically refers to: the overlapping area between the scene image captured by each camera to be tested and the scene image captured by each camera to be tested having an adjacent relationship (also called overlapping image area).

[0048] Reference Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific implementation of step S12 in the embodiment of the present invention, wherein step S12 may include the following steps S121 to S125.

[0049] In step S121, the scene point cloud is projected onto the scene image captured by the camera to be tested using the external parameters of the camera to be tested, to obtain a plurality of first projection points.

[0050] In a specific implementation, the step S121 may specifically include: first using the external parameters of the laser radar to project the scene point cloud to the vehicle coordinate system; then using the external parameters of the camera to be tested to project the scene point cloud projected to the vehicle coordinate system to the camera coordinate system of the camera to be tested; and then using the internal parameters of the camera to be tested to project the scene point cloud projected to the camera coordinate system to the image coordinate system where the scene image captured by the camera to be tested is located, to obtain multiple first projection points located in the image coordinate system.

[0051] It should be pointed out that in the present application, the external parameters of the laser radar and the internal parameters of the camera to be tested can be parameters that have been pre-calibrated and confirmed to be accurate and reliable, while the external parameters of the camera to be tested are parameters that need to be verified for abnormality or accuracy.

[0052] In step S122, the scene point cloud is projected onto the scene image captured by the adjacent camera to be tested using the external parameters of the adjacent camera to be tested, so as to obtain a plurality of second projection points.

[0053] Regarding the specific implementation method of obtaining the plurality of second projection points, refer to the description of obtaining the first projection point in the above step S121, which will not be repeated here.

[0054] In step S123, the intersection of the plurality of first projection points and the plurality of second projection points is determined, and a plurality of original three-dimensional points corresponding to each projection point in the intersection in the scene point cloud are used as the plurality of target points.

[0055] In step S124, for each first projection point corresponding to the target point, a first pixel point closest to the first projection point is determined in the overlapping area, and the category of the first pixel point is used as the first category of the target point relative to the camera to be tested.

[0056] In step S125, for each second projection point corresponding to the target point, a second pixel point closest to the second projection point is determined in the overlapping area, and the category of the second pixel point is used as the second category of the target point relative to the adjacent camera to be tested.

[0057] Specifically, the first projection point corresponding to each target point is obtained by projecting using the extrinsic parameters of the camera to be tested, so the first category of the target point determined in step S124 is related to the extrinsic parameters of the camera to be tested; similarly, the second projection point corresponding to each target point is obtained by projecting using the extrinsic parameters of the camera to be tested adjacent to the camera to be tested, so the second category of the target point determined in step S124 is related to the extrinsic parameters of the adjacent camera to be tested.

[0058] Specifically, the number of all first categories and all second categories to which the plurality of target points belong may be one or more. In other words, the plurality of target points may belong to n first categories and n second categories (n is a positive integer). The first category and the second category may be selected from, but not limited to, motor vehicles, pedestrians, road surfaces, non-motor vehicles, buildings, traffic facilities, etc.

[0059] It is understandable that, for a single target point, its first category and second category may be the same or different. For example, the first category of target point A is road surface, and the second category may be road surface, pedestrian, vehicle or other categories.

[0060] exist Figure 1 In another implementation of step S12, the following steps (1) to (4) may be used to determine a plurality of target points corresponding to each pixel point in the overlapping area in the scene point cloud, and the first category and the second category of each target point:

[0061] Step (1): using the external parameters of the camera to be tested, projecting the scene point cloud onto the scene image captured by the camera to be tested, to obtain a plurality of first projection points;

[0062] Step (2): for the multiple original three-dimensional points (referred to as first original three-dimensional points) corresponding to the multiple first projection points in the scene point cloud, the multiple first original three-dimensional points are projected to the overlapping area using the external parameters of the adjacent cameras to be tested to obtain multiple second projection points, and the multiple original three-dimensional points (referred to as second original three-dimensional points) corresponding to the multiple second projection points in the scene point cloud are used as the multiple target points;

[0063] Step (3): for each first projection point corresponding to the target point, determine the first pixel point closest to the first projection point in the overlapping area, and use the category of the first pixel point as the first category of the target point relative to the camera to be tested;

[0064] Step (4): for each second projection point corresponding to the target point, determine the second pixel point closest to the second projection point in the overlapping area, and use the category of the second pixel point as the second category of the target point relative to the adjacent camera to be tested.

[0065] Continue to refer to Figure 1 In the specific implementation of step S13, based on the first category and the second category of each of the plurality of target points, a category consistency analysis is performed to obtain a preliminary analysis result of the overlapping area. It can be understood that the larger the value of the preliminary analysis result, the higher the degree of consistency between the first category and the second category of each of the plurality of target points.

[0066] In a specific implementation, the following scheme may be used for category consistency analysis: count the number of target points in the first category and the second category that are consistent among the multiple target points, and directly use the ratio of this number to the total number of the multiple target points as the preliminary analysis result of the overlapping area.

[0067] In another specific embodiment, it is possible to use Figure 3 The steps shown are used to perform category consistency analysis. Figure 3 , Figure 3 yes Figure 1 A flowchart of a specific implementation of step S13 in the embodiment. In this implementation, the number of the first category and the number of the second category to which the multiple target points belong are at least one, and the step S13 may specifically include the following steps S131 to S135.

[0068] In step S131, for each first category to which the multiple target points belong, the target points belonging to the first category among the multiple target points and the total number thereof are determined, and the target points belonging to the first category are recorded as first target points.

[0069] In step S132, for each second category to which each first target point belongs, the number of first target points belonging to the second category among the first target points is counted, and recorded as the statistical number corresponding to the second category.

[0070] In step S133, a score value corresponding to each second category is determined based at least on the statistical quantity corresponding to each second category to which each first target point belongs and the total quantity.

[0071] For example, among the multiple target points, several first target points belong to the first category of road surface, among the several first target points, a1 target points may belong to the second category of road surface, and a2 target points may belong to the second category of vehicles or pedestrians, that is, each target point belonging to a single first category may belong to one or more second categories.

[0072] In one embodiment, the score value corresponding to each second category is determined based at least on the statistical quantity corresponding to each second category to which the first target points belong and the total quantity, including: for each second category to which the first target points belong, calculating the ratio of the statistical quantity corresponding to the second category to the total quantity, and using the square of the ratio as the score value corresponding to the second category.

[0073] Specifically, the following formula (1) can be used to determine the score value corresponding to each second category:

[0074] S j =(a j ÷sum i ) 2 (1);

[0075] Among them, i represents the i-th first category, j represents the j-th second category, S j represents the score value corresponding to the jth second category, sum i represents the total number of first target points belonging to the i-th first category among the multiple target points, a j It represents the number of first target points belonging to the jth second category among the first target points belonging to the i-th first category.

[0076] In another specific embodiment, the scoring value corresponding to each second category is determined based at least on the statistical quantity corresponding to each second category to which the first target points belong and the total quantity, including: sorting the statistical quantities corresponding to the various second categories to which the first target points belong from large to small to obtain the sorted second categories; for each sorted second category, determining the corresponding scoring coefficient based on the order to which the second category belongs, and calculating the product of the scoring coefficient and the statistical quantity corresponding to the second category, and using the ratio of the product to the total quantity as the scoring value corresponding to the second category.

[0077] Specifically, the score coefficient of the jth second category after sorting can be: (j-1) , wherein k is a preset constant, 0<k<1. In some non-limiting embodiments, k=0.4.

[0078] More specifically, the following formula (2) can be used to determine the score corresponding to each second category:

[0079] S j =(k (j-1) ×a j )÷sum i (2);

[0080] Among them, i represents the i-th first category, j represents the j-th second category after sorting, S j represents the score value corresponding to the jth second category after sorting, sum i represents the total number of first target points belonging to the i-th first category among the multiple target points, a j It represents the number of first target points belonging to the jth second category after sorting among the first target points belonging to the i-th first category.

[0081] It can be understood that, compared with formula (1), the above formula (2) adds a scoring coefficient, and the scoring coefficients of various second categories are related to the number of target points belonging to the second category, which helps to fine-tune the scoring value calculation process and improve the calculation accuracy of the scoring value.

[0082] In step S134, the score values ​​of the various second categories to which the first target points belong are summed to obtain a total score of the first category.

[0083] For example, for each first target point belonging to the first category of road surface, the second categories to which it belongs are road surface and pedestrians. The score corresponding to the second category of road surface is 0.2, and the score corresponding to the second category of pedestrians is 0.4. Then, the total score corresponding to the first category (road surface) to which each first target point belongs is 0.2+0.4=0.6.

[0084] In step S135, a weighted operation is performed on the total scores of various first categories to which the multiple target points belong, so as to obtain a preliminary analysis result of the overlapping area.

[0085] In a specific implementation, the number of target points belonging to each first category among the multiple target points can be counted and recorded as the statistical number corresponding to the first category; the ratio of the statistical numbers corresponding to the various first categories to which the multiple target points belong is used as the weight ratio, and the total scores of the various first categories are weighted (for example, weighted summation) to obtain the preliminary analysis results of the overlapping area.

[0086] Reference Figure 4 , Figure 4 yes Figure 1 The flowchart of a specific implementation of step S14 in the embodiment of the present invention is shown in FIG. 1 , wherein step S14 may specifically include the following steps S141 to S143.

[0087] In step S141, for the overlapping area between the camera to be tested and each adjacent camera to be tested, the total number of the multiple target points corresponding to each pixel point in the overlapping area in the scene point cloud is counted, and the total number is used as the weight of the preliminary analysis result of the overlapping area.

[0088] In step S142, the weights are used to perform a weighted operation on the preliminary analysis results of the overlapping areas of the camera to be tested and each adjacent camera to be tested, so as to obtain a first weighted analysis result.

[0089] Specifically, the weighted operation is, for example, weighted summation, weighted product, etc. The smaller the value of the first weighted analysis result is, the lower the accuracy of the external parameters of the camera to be tested is; conversely, the larger the value of the first weighted analysis result is, the higher the accuracy of the external parameters of the camera to be tested is.

[0090] In step S143, the accuracy of the external parameters of the camera to be tested is detected based at least on the first weighted analysis result.

[0091] In a specific implementation, the step S143 may include: comparing the first weighted analysis result with a preset score; if the first weighted analysis result is greater than the preset score, confirming that the external parameters of the camera under test are accurate; otherwise, confirming that the external parameters of the camera under test are abnormal. The preset score may be an empirical value determined based on several pre-experiments, for example, may be selected from the interval [0.8, 1).

[0092] In another specific implementation, the step S143 may include: using multiple different offset parameters to offset the external parameters of the camera to be tested (recorded as original external parameters) respectively, to obtain multiple offset external parameters of the camera to be tested; determining the corresponding second weighted analysis result based on each offset external parameter of the camera to be tested; determining the maximum value among the first weighted analysis result and each second weighted analysis result, and determining the offset parameter to which the maximum value belongs, recorded as the target offset parameter; if the target offset parameter is less than or equal to a preset offset parameter threshold, confirming that the external parameters of the camera to be tested are accurate.

[0093] Among them, the scheme for determining the corresponding second weighted analysis result based on each offset external parameter of the camera to be tested is consistent with the scheme for determining the first weighted analysis result based on the external parameters of the camera to be tested (ie, the original external parameters), and will not be described in detail here.

[0094] In a specific implementation, the various offset parameters for offsetting the original external parameters of the camera under test can be determined according to multiple sets of different pitch, yaw, and roll parameter values. For example, each offset parameter can be determined using , where pitch, yaw, and roll represent the rotation angles of the camera to be tested from its original position (i.e., the position corresponding to the original external parameter) around the x-axis, y-axis, and z-axis of the camera coordinate system.

[0095] It can be understood that the camera extrinsic parameters (which may be original extrinsic parameters or offset extrinsic parameters) corresponding to the maximum values ​​in the first weighted analysis result and each second weighted analysis result have the highest accuracy. Therefore, if the offset parameter to which the maximum value belongs is larger, it means that the difference between the original extrinsic parameter before offset and the extrinsic parameter with the highest accuracy after offset is greater, which means that the accuracy of the original extrinsic parameter is lower.

[0096] Based on the above principle, the embodiment of the present invention compares the offset parameter to which the maximum value in the weighted analysis result belongs with the preset offset parameter threshold. If the former is smaller than the latter, it can be confirmed that the difference between the original external parameter and the offset external parameter with the highest accuracy is within the expected range, that is, the original external parameter can be considered accurate and reliable; otherwise, it can be confirmed that the original external parameter is inaccurate or invalid. Compared with the implementation scheme of directly judging whether the camera extrinsic parameter is accurate by comparing the first weighted analysis result with the preset score value, the above scheme is beneficial to improving the accuracy and reliability of the external parameter detection result because it integrates the score results corresponding to different offset parameters (equivalent to giving different disturbances to the camera extrinsic parameter to be tested).

[0097] Reference Figure 5 , Figure 5 : is a schematic diagram of the structure of an external parameter detection device for a camera in an embodiment of the present invention. The device may include:

[0098] A scene data determination module 51 is used to determine a scene point cloud collected by a laser radar and a plurality of scene images collected by a plurality of cameras to be tested, wherein the scene point cloud and the plurality of scene images are time-series synchronized, each camera to be tested has an adjacent relationship with at least one other camera to be tested, and the scene images collected by the adjacent cameras to be tested have an overlapping area, and each pixel point in the overlapping area has a category to which it belongs;

[0099] A category determination module 52 is used to determine, for each camera under test and its adjacent camera under test overlapping area, a plurality of target points corresponding to each pixel point in the overlapping area in the scene point cloud, and a first category and a second category of each target point based on the external parameters of the camera under test and the category of each pixel point in the overlapping area, wherein the first category is the category corresponding to the camera under test, and the second category is the category corresponding to the adjacent camera under test;

[0100] A category consistency analysis module 53, configured to perform category consistency analysis based on the first category and the second category of each of the plurality of target points to obtain a preliminary analysis result of the overlapping area;

[0101] The external parameter accuracy detection module 54 is used to detect the accuracy of the external parameters of the camera to be tested based on the preliminary analysis results of each overlapping area between the camera to be tested and each adjacent camera to be tested.

[0102] For the principle, specific implementation and beneficial effects of the camera's external parameter detection device, please refer to the previous article and Figures 1 to 4 The relevant description of the external parameter detection method of the camera shown in any embodiment will not be repeated here.

[0103] The embodiment of the present invention further provides a storage medium, such as a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor. Figures 1 to 4 The steps of the camera external parameter detection method shown in any embodiment. The computer-readable storage medium may include a non-volatile memory or a non-transitory memory, and may also include an optical disk, a mechanical hard disk, a solid-state hard disk, etc.

[0104] Specifically, in the embodiment of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0105] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0106] The embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned Figures 1 to 4The steps of the camera external parameter detection method shown in any embodiment.

[0107] The embodiment of the present invention further provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, Figures 1 to 4 The steps of the camera external parameter detection method shown in any embodiment.

[0108] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0109] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0110] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0111] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0112] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A camera external parameter detection method, characterized in that: include: Determine a scene point cloud collected by a laser radar and a plurality of scene images collected by a plurality of cameras to be tested, wherein the scene point cloud and the plurality of scene images are time-series synchronized, each camera to be tested has an adjacent relationship with at least one other camera to be tested, and the scene images collected by the adjacent cameras to be tested have an overlapping area, and each pixel point in the overlapping area has a category to which it belongs; For the overlapping area between each camera to be tested and its adjacent camera to be tested, based on the extrinsic parameters of the camera to be tested and the categories of each pixel point in the overlapping area, determine a plurality of target points corresponding to each pixel point in the overlapping area in the scene point cloud, and a first category and a second category of each target point, wherein the first category is a category corresponding to the camera to be tested, and the second category is a category corresponding to the adjacent camera to be tested; Based on the first category and the second category of each of the plurality of target points, a category consistency analysis is performed to obtain a preliminary analysis result of the overlapping area; Based on the preliminary analysis results of the overlapping areas between the camera to be tested and each adjacent camera to be tested, the accuracy of the external parameters of the camera to be tested is detected.

2. The method according to claim 1, characterized in that The number of the first category and the number of the second category to which the plurality of target points belong are both at least one; Based on the first category and the second category of each of the plurality of target points, a category consistency analysis is performed to obtain a preliminary analysis result of the overlapping area, including: For each first category to which the plurality of target points belong, determining the target points belonging to the first category and the total number thereof among the plurality of target points, and recording the target points belonging to the first category as first target points; For each second category to which each first target point belongs, the number of first target points belonging to the second category among the first target points is counted, and recorded as the statistical number corresponding to the second category; Determine a score value corresponding to each second category based at least on the statistical quantity corresponding to each second category to which each first target point belongs and the total quantity; Sum the score values ​​of various second categories to which the first target points belong to obtain a total score of the first category; A weighted operation is performed on the total scores of various first categories to which the multiple target points belong, so as to obtain a preliminary analysis result of the overlapping area.

3. The method according to claim 2, characterized in that Determining a score value corresponding to each second category based at least on the statistical quantity corresponding to each second category to which each first target point belongs and the total quantity includes: For each second category to which each of the first target points belongs, the ratio of the statistical quantity corresponding to the second category to the total quantity is calculated, and the square of the ratio is used as the score value corresponding to the second category.

4. The method according to claim 2, characterized in that: Determining a score value corresponding to each second category based at least on the statistical quantity corresponding to each second category to which each first target point belongs and the total quantity includes: Sorting the second categories to which the first target points belong in descending order according to the statistical quantities corresponding to the second categories to obtain sorted second categories; For each sorted second category, the corresponding scoring coefficient is determined based on the order to which the second category belongs, and the product of the scoring coefficient and the statistical number corresponding to the second category is calculated, and the ratio of the product to the total number is used as the scoring value corresponding to the second category.

5. The method according to any one of claims 1 to 4, characterized in that: The detecting the accuracy of the external parameters of the camera to be tested based on the preliminary analysis results of the overlapping areas of the camera to be tested and each adjacent camera to be tested includes: For the overlapping area between the camera to be tested and each adjacent camera to be tested, counting the total number of the plurality of target points corresponding to each pixel point in the overlapping area in the scene point cloud, and using the total number as the weight of the preliminary analysis result of the overlapping area; Using the weights, a weighted operation is performed on preliminary analysis results of overlapping areas of the camera to be tested and each adjacent camera to be tested to obtain a first weighted analysis result; At least based on the first weighted analysis result, the accuracy of the external parameters of the camera to be tested is detected.

6. The method according to claim 5, characterized in that At least based on the first weighted analysis result, detecting the accuracy of the external parameters of the camera to be tested includes: Using a plurality of different offset parameters, the external parameters of the camera to be tested are offset respectively to obtain a plurality of offset external parameters of the camera to be tested; Determine a corresponding second weighted analysis result based on each offset external parameter of the camera to be tested; Determine a maximum value among the first weighted analysis result and each second weighted analysis result, and determine an offset parameter to which the maximum value belongs, recorded as a target offset parameter; If the target offset parameter is less than or equal to a preset offset parameter threshold, it is confirmed that the external parameters of the camera to be tested are accurate; The scheme of determining the corresponding second weighted analysis result based on each offset external parameter of the camera to be tested is consistent with the scheme of determining the first weighted analysis result based on the external parameter of the camera to be tested.

7. The method according to claim 1, characterized in that The method further comprises: determining, for each camera to be tested and its adjacent camera to be tested in an overlapping area, a plurality of target points corresponding to each pixel point in the overlapping area in the scene point cloud, and a first category and a second category of each target point based on the external parameters of the camera to be tested and the category of each pixel point in the overlapping area, including: Using the external parameters of the camera to be tested, projecting the scene point cloud onto the scene image captured by the camera to be tested, to obtain a plurality of first projection points; Using the external parameters of the adjacent camera to be tested, projecting the scene point cloud onto the scene image captured by the adjacent camera to be tested, to obtain a plurality of second projection points; Determine the intersection of the plurality of first projection points and the plurality of second projection points, and use the plurality of original three-dimensional points corresponding to each projection point in the intersection in the scene point cloud as the plurality of target points; for each first projection point corresponding to the target point, determine the first pixel point closest to the first projection point in the overlapping area, and use the category of the first pixel point as the first category of the target point relative to the camera to be tested; For each second projection point corresponding to the target point, a second pixel point closest to the second projection point is determined in the overlapping area, and the category of the second pixel point is used as the second category of the target point relative to the adjacent camera to be tested.

8. A camera external parameter detection device, characterized in that: include: A scene data determination module, used to determine a scene point cloud collected by a laser radar and a plurality of scene images collected by a plurality of cameras to be tested, wherein the scene point cloud and the plurality of scene images are time-series synchronized, each camera to be tested has an adjacent relationship with at least one other camera to be tested, and the scene images collected by adjacent cameras to be tested have an overlapping area, and each pixel point in the overlapping area has a category to which it belongs; A category determination module is used to determine, for each camera under test and its adjacent camera under test overlapping area, a plurality of target points corresponding to each pixel point in the overlapping area in the scene point cloud, and a first category and a second category of each target point based on the external parameters of the camera under test and the category of each pixel point in the overlapping area, wherein the first category is the category corresponding to the camera under test, and the second category is the category corresponding to the adjacent camera under test; A category consistency analysis module, used for performing category consistency analysis based on the first category and the second category of each of the plurality of target points to obtain a preliminary analysis result of the overlapping area; The external parameter accuracy detection module is used to detect the accuracy of the external parameters of the camera to be tested based on the preliminary analysis results of each overlapping area between the camera to be tested and each adjacent camera to be tested.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the external parameter detection method of the camera according to any one of claims 1 to 7 are executed.

10. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the external parameter detection method of the camera according to any one of claims 1 to 7 are performed.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the external parameter detection method of the camera according to any one of claims 1 to 7 are executed.

Images