A method for calibrating camera extrinsic parameters and a storage medium

Abstract

The camera extrinsic parameter calibration method and the storage medium provided by the embodiment of the application can be applied to the field of information technology, can acquire a monitoring image collected by a target camera, can perform feature extraction on the monitoring image, and can create a target feature map according to the extracted features; initial values of target point cloud data of a scene where the target camera is located and installation position information of the target camera are acquired; an initial projection matrix is created according to the initial values of the target point cloud data and the installation position information, the target point cloud is projected onto the target feature map through the initial projection matrix, and a first projection result is obtained; the projection matrix is corrected according to the first projection result, a target projection matrix is obtained, and the extrinsic parameter of the target camera is calculated according to the target projection matrix. It can be seen that, by using the method of the embodiment of the application, the projection matrix can be solved, the extrinsic parameter of the target camera is calculated according to the finally obtained target projection matrix, and the automatic calibration of the target camera is realized.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a method for calibrating camera extrinsic parameters and a storage medium. Background Technology

[0002] Currently, in image measurement and machine vision applications, to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. The parameters of this geometric model are the camera parameters. Among them, the camera extrinsic parameters describe the correspondence between the image acquired by the camera and the world coordinate system, determining the camera's position and orientation in a certain three-dimensional space, such as the camera's position and rotation direction.

[0003] However, currently, obtaining camera extrinsic parameters often requires experiments and calculations, which not only incurs high labor costs but also takes a long time and results in low calibration efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a camera extrinsic parameter calibration method and storage medium to solve the problems of high labor costs, long time consumption, and low calibration efficiency in the camera extrinsic parameter calibration process. The specific technical solution is as follows:

[0005] A first aspect of this application provides a camera extrinsic parameter calibration method, including:

[0006] Acquire surveillance images captured by the target camera, wherein the surveillance images include power transmission lines;

[0007] The edge structure features of the power transmission line in the surveillance image are extracted, and a target feature map is created based on the edge structure features of the power transmission line.

[0008] Initial values ​​are obtained for target point cloud data of the scene where the target camera is located and the installation location information of the target camera, wherein the scene where the target camera is located includes the power transmission line and the target point cloud includes the power transmission line point cloud;

[0009] An initial projection matrix is ​​created based on the initial values ​​of the power transmission line point cloud and the installation location information, and the power transmission line point cloud is projected onto the target feature map through the initial projection matrix to obtain a first projection result. The initial projection matrix represents the correspondence between the world coordinate system and the camera coordinate system of the target camera.

[0010] The projection matrix is ​​corrected based on the first projection result to obtain the target projection matrix, and the extrinsic parameters of the target camera are calculated based on the target projection matrix.

[0011] A second aspect of this application provides another method for calibrating camera extrinsic parameters, including:

[0012] Acquire surveillance images captured by the target camera;

[0013] Feature extraction is performed on the captured images, and a target feature map is created based on the extracted features;

[0014] Initial values ​​are obtained for the target point cloud data of the scene where the target camera is located and the installation location information of the target camera;

[0015] An initial projection matrix is ​​created based on the initial values ​​of the target point cloud data and the installation location information, and the target point cloud is projected onto the target feature map through the initial projection matrix to obtain a first projection result.

[0016] The projection matrix is ​​corrected based on the first projection result to obtain the target projection matrix, and the extrinsic parameters of the target camera are calculated based on the target projection matrix.

[0017] A third aspect of this application provides a camera extrinsic parameter calibration device, comprising:

[0018] A monitoring image acquisition module is used to acquire monitoring images captured by a target camera, wherein the monitoring images include power transmission lines;

[0019] The structural feature extraction module is used to extract the edge structural features of the power transmission line in the surveillance image, and to obtain and create a target feature map based on the edge structural features of the power transmission line.

[0020] The point cloud acquisition module is used to acquire the initial values ​​of target point cloud data of the scene where the target camera is located and the installation location information of the target camera, wherein the scene where the target camera is located includes the power transmission line, and the target point cloud includes the power transmission line point cloud;

[0021] The point cloud projection module is used to create an initial projection matrix based on the initial values ​​of the power transmission line point cloud and the installation location information, and to project the power transmission line point cloud onto the target feature map through the initial projection matrix to obtain a first projection result. The initial projection matrix represents the correspondence between the world coordinate system and the camera coordinate system of the target camera.

[0022] The camera extrinsic parameter calculation module is used to correct the projection matrix based on the first projection result to obtain the target projection matrix, and to calculate the extrinsic parameters of the target camera based on the target projection matrix.

[0023] A fourth aspect of this application provides another camera extrinsic parameter calibration device, comprising:

[0024] The image acquisition module is used to acquire monitoring images captured by the target camera;

[0025] The matrix creation module is used to extract features from the captured images and create a target feature map based on the extracted features.

[0026] The location acquisition module is used to acquire the initial values ​​of target point cloud data of the scene where the target camera is located and the installation location information of the target camera;

[0027] The projection result acquisition module is used to create an initial projection matrix based on the initial values ​​of the target point cloud data and the installation location information, and to project the target point cloud onto the target feature map through the initial projection matrix to obtain a first projection result.

[0028] The matrix correction module is used to correct the projection matrix according to the first projection result to obtain the target projection matrix, and to calculate the extrinsic parameters of the target camera according to the target projection matrix.

[0029] In another aspect of this application, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0030] Memory, used to store computer programs;

[0031] When the processor executes the program stored in the memory, it implements any of the camera extrinsic parameter calibration method steps described above.

[0032] In another aspect of the embodiments of this application, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the camera extrinsic parameter calibration method steps described above.

[0033] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the camera extrinsic parameter calibration method steps described above.

[0034] Beneficial effects of the embodiments in this application:

[0035] This application provides a camera extrinsic parameter calibration method and storage medium, which can acquire monitoring images captured by a target camera; extract features from the monitoring images and create a target feature map based on the extracted features; acquire initial values ​​of target point cloud data and the installation location information of the target camera in the scene where the target camera is located; create an initial projection matrix based on the initial values ​​of the target point cloud data and the installation location information, and project the target point cloud onto the target feature map through the initial projection matrix to obtain a first projection result; correct the projection matrix based on the first projection result to obtain a target projection matrix, and calculate the extrinsic parameters of the target camera based on the target projection matrix. Therefore, the method of this application can automatically calibrate the target camera by acquiring target point cloud data and the installation location information of the target camera in the scene where the target camera is located, as well as monitoring images captured by the target camera; identifying and projecting the target point cloud onto the target feature map through the initial projection matrix to obtain a first projection result; solving the projection matrix; and calculating the extrinsic parameters of the target camera based on the final target projection matrix. This solves the problems of high manual costs, long time consumption, and low calibration efficiency in the camera extrinsic parameter calibration process.

[0036] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0038] Figure 1 A schematic flowchart of a camera extrinsic parameter calibration method provided in an embodiment of this application;

[0039] Figure 2 A schematic diagram illustrating the process of obtaining the installation location information of the target camera as provided in an embodiment of this application;

[0040] Figure 3 An example diagram illustrating the acquisition of the installation location information of the target camera provided in an embodiment of this application;

[0041] Figure 4 A schematic diagram illustrating the process of creating a target feature map provided in an embodiment of this application;

[0042] Figure 5 A schematic diagram of the process for obtaining the features corresponding to the power transmission line provided in an embodiment of this application;

[0043] Figure 6A flowchart illustrating the calculation of the extrinsic parameters of the target camera provided in an embodiment of this application;

[0044] Figure 7 Another schematic flowchart of the camera extrinsic parameter calibration method provided in the embodiments of this application;

[0045] Figure 8 A schematic diagram of a camera extrinsic parameter calibration device provided in an embodiment of this application;

[0046] Figure 9 This is another schematic diagram of the camera extrinsic parameter calibration device provided in the embodiments of this application;

[0047] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0048] Figure 11 This is another structural schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0050] A first aspect of this application provides a method for calibrating camera extrinsic parameters, see [link to relevant documentation]. Figure 1 ,include:

[0051] Step S11: Acquire the monitoring image captured by the target camera.

[0052] In this embodiment, the target camera is the camera whose extrinsic parameters need to be calibrated, and the monitoring image is an image captured by the target camera. Specifically, acquiring the monitoring image captured by the target camera can be done after the target camera is installed and extrinsic parameters need to be calibrated. The monitoring image in this embodiment may include a specified reference object. Specifically, the specified reference object may be a pre-defined object. For example, when the target camera in this embodiment is installed on a power transmission line tower, the camera monitors roads or squares below the tower, and the specified reference object may be a combination of the tower adjacent to the tower where the target camera is located and the power transmission lines erected on the adjacent tower.

[0053] The method described in this application is applied to a smart terminal, which can be implemented through the smart terminal. Specifically, the smart terminal can be a computer, mobile phone, smart camera, smart video recorder, or server, etc.

[0054] Step S12: Extract features from the captured image and create a target feature map based on the extracted features.

[0055] In this process, feature extraction of the surveillance image can be performed to extract edge structure features. Specifically, when the surveillance image in this embodiment includes a specified object, feature extraction of the surveillance image can be performed to extract the edge structure features of the specified object in the surveillance image, and a target feature map can be created based on the extracted edge structure features.

[0056] For example, when the method of this application embodiment is applied to a target camera installed on a transmission line tower, feature extraction can be performed on the image acquired by summing. For example, the image may include transmission lines and towers. Feature extraction can be performed on the monitoring image, which can extract the edge structure features of the transmission lines and / or the edge structure features of the towers. Then, a corresponding Map can be constructed based on the extracted edge structure features. For example, a corresponding Map can be constructed based on one or both of the edge structure features of the transmission lines and the edge structure features of the towers.

[0057] Step S13: Obtain the initial values ​​of the target point cloud data of the scene where the target camera is located and the installation location information of the target camera.

[0058] The target point cloud data in this embodiment can be target point cloud data of the scene where the target camera is located, collected by a target point cloud data acquisition device such as a drone. The target point cloud data of the scene where the target camera is located can include the point cloud of the entirety or a portion of the specified object. For example, the scene where the target camera is located might be a power transmission line scene. The installation location information of the target camera can include information such as the installation height of the target camera. When obtaining the initial value of the installation location information of the target camera, the environmental characteristics of the installation environment of the target camera can be obtained. For example, when the target camera is installed on a pole, the height of the pole can be estimated based on the type of pole, and then the approximate installation location of the target camera can be estimated based on the height of the pole. For example, if the pole height is 18m, the installation height of the target camera is approximately 15m.

[0059] Step S14: Create an initial projection matrix based on the initial values ​​of the target point cloud data and installation location information, and project the target point cloud onto the target feature map using the initial projection matrix to obtain the first projection result.

[0060] The initial projection matrix can be a matrix that projects the target point cloud onto the same coordinate system as the target feature map. In one example, the target point cloud in this embodiment is a point cloud corresponding to the world coordinate system, and the target feature map is an image corresponding to the camera coordinate system of the target camera. In actual use, when creating the target feature map based on the extracted features, the image created based on the extracted features is an image in the image coordinate system of the corresponding camera. Then, an inverse transformation is performed on this image to obtain the target feature map in the camera coordinate system corresponding to the image.

[0061] The process involves creating an initial projection matrix based on the initial values ​​of the target point cloud data and installation location information. This allows for the acquisition of the initial value of the target camera's installation height based on the installation location, and the acquisition of the Euler angles of the target camera's installation location based on the target point cloud data. Specifically, the target point cloud data can be projected onto a horizontal plane, and the Euler angles of the camera's installation location can be obtained based on the angles between the specified object and the horizontal plane's x-axis and y-axis in the projection result. Optionally, creating an initial projection matrix based on the initial values ​​of the target point cloud data and installation location information, and then projecting the target point cloud onto the target feature map using the initial projection matrix, yields a first projection result. This includes: creating an initial projection matrix based on the initial values ​​of the target camera's installation angle and installation height; and projecting the target point cloud onto the target feature map using the initial projection matrix to obtain the first projection result.

[0062] Step S15: Correct the projection matrix according to the first projection result to obtain the target projection matrix, and calculate the extrinsic parameters of the target camera according to the target projection matrix.

[0063] The process involves correcting the projection matrix based on the first projection result. After projecting the target point cloud onto the target feature map using the initial projection matrix, the deviation between the target feature map and the projected point cloud is identified. This deviation is then used to modify the projection matrix. The projected point cloud is then re-projected using the corrected projection matrix, and the deviation is calculated to obtain the projection matrix with the smallest deviation, thus yielding the target projection matrix. Since the projection matrix is ​​created using the camera's Euler angles and height, identifying the target projection matrix allows us to determine the camera's mounting angle and height, i.e., the camera's extrinsic parameters.

[0064] As can be seen, the method of this application embodiment can obtain target point cloud data of the scene where the target camera is located, the installation location information of the target camera, and the monitoring images captured by the target camera. By identifying the target point cloud and projecting it onto the target feature map through an initial projection matrix to obtain the first projection result, the projection matrix is ​​solved, and the extrinsic parameters of the target camera are calculated based on the final target projection matrix. This achieves automatic calibration of the target camera and solves the problems of high manual cost, long time consumption, and low calibration efficiency in the process of camera extrinsic parameter calibration.

[0065] Optional, see Figure 2 The scene where the target camera is located includes a first target object and a second target object. The target point cloud data of the scene where the target camera is located includes the point cloud of the first target object. The initial value of the target camera's installation position information includes the initial value of the target camera's installation height. Obtaining the target point cloud data of the scene where the target camera is located and the initial value of the target camera's installation position information includes:

[0066] Step S21: Obtain target point cloud data of the scene where the target camera is located;

[0067] Step S22: Project the target point cloud data onto a horizontal plane to obtain a second projection result;

[0068] Step S23: Identify and calculate the initial value of the installation angle of the target camera based on the position of the first target object in the second projection result;

[0069] Step S24: Obtain the preset attribute information of the second target object, and calculate the initial value of the installation height of the target camera based on the preset attribute information.

[0070] Optionally, step S22 projects the target point cloud data onto a horizontal plane to obtain a second projection result, including: identifying the point cloud of the first target object in the scene point cloud according to a preset height threshold; and projecting the point cloud of the first target object onto the XOY plane to obtain a second projection result, wherein the XOY plane is a horizontal plane, and the XOY plane is the plane containing the X-axis, the Y-axis, and the intersection point O of the X-axis and the Y-axis.

[0071] Step S23 identifies and calculates the initial value of the installation angle of the target camera based on the position of the first target object in the second projection result, including: identifying the projection corresponding to the first target object in the second projection result; calculating the direction of the projection of the first target object in the second projection result and the angle between it and the X-axis and / or Y-axis to obtain the initial value of the installation angle of the target camera. For example, when the method of this embodiment is applied to the tower of a power transmission line, the target point cloud data is adjusted, the power transmission line point cloud model is projected along the Z-axis onto the horizontal XOY plane, the power transmission line straight line equation is fitted on the XOY plane, the angle between the straight line equation and the X-axis or Y-axis is calculated based on the straight line equation, and the target point cloud data is rotated so that the fitted straight line is parallel to the X-axis. At this time, the power transmission line target point cloud data is simultaneously perpendicular to the YOZ plane. At this time, the Euler angle between the point cloud and the camera pose is approximately [-90°, 0°, 90°] or [90°, 0°, 90°], and this Euler angle is used as the initial value of the installation angle of the target camera. Here, the target point cloud is point cloud data obtained based on the world coordinate system.

[0072] See Figure 3For the purpose of explanation, the reference objects are specified as transmission lines and towers:

[0073] 1. Transmission line and tower point cloud classification: The user-input point cloud model contains transmission lines, towers, and ground point clouds. First, a plane fitting method is used to approximately fit the ground point cloud; then, the tower and transmission line point clouds are classified based on a height threshold, thus dividing the remaining point cloud model into transmission line point clouds and tower point clouds.

[0074] 2. Adjust the power line point cloud model so that the power line is parallel to the X-axis and perpendicular to the YOZ plane. Adjust the input scene point cloud model and project the power line point cloud model along the Z-axis onto the XYO plane. Fit the power line straight line equation on the XYO plane. Calculate the angle between the straight line equation and the X-axis or Y-axis based on the straight line equation. Rotate the target point cloud data so that the fitted straight line is parallel to the X-axis. At this time, the power line target point cloud data is also perpendicular to the YOZ plane. The Euler angle between the point cloud and the camera pose is approximately [-90°, 0°, 90°] or [90°, 0°, 90°].

[0075] 3. Based on the camera's orientation and installation height, determine the camera's initial extrinsic parameters R,t. Specifically, based on the camera's installation tower number and the number of the monitored opposing tower, obtain the centroid coordinates of the camera's installation tower. Use the corresponding x,y coordinates as the camera's translation vector x,y coordinates. Construct the initial extrinsic parameters t based on the approximate installation height of the camera. The Euler angles of the camera's pose can be further confirmed using the camera's installation tower number and the number of the monitored opposing tower. When the camera's world coordinate x is greater than the opposing tower's world coordinate x, the initial Euler angles are [-90°, 0°, 90°]; otherwise, they are [90°, 0°, 90°]. This yields the camera's initial extrinsic parameters R,t.

[0076] As can be seen, the method of this application embodiment can project target point cloud data onto a horizontal plane to obtain a second projection result, identify and calculate the initial value of the installation angle of the target camera based on the position of the first target object in the second projection result, and calculate the initial value of the installation height of the target camera based on preset attribute information, thereby obtaining the extrinsic parameters composed of the initial value of the installation angle and the initial value of the installation height, and then create the projection matrix through the initial extrinsic parameters.

[0077] Optional, see Figure 4 Step S12 involves extracting features from the captured image and creating a target feature map based on the extracted features, including:

[0078] Step S121: Extract edge features from the first target object and the second target object in the monitored image to obtain the first target feature and the second target feature;

[0079] Step S122: Construct a feature image based on the first target features and / or the second target features;

[0080] Step S123: Perform an inverse transformation on the feature image according to the preset inverse transformation formula to obtain the target feature map.

[0081] Optionally, edge features are extracted from the first target object and the second target object in the surveillance image to obtain the first target feature and the second target feature, including: converting the surveillance image to grayscale to obtain a grayscale image; applying linear feature filtering to the first target object in the grayscale image to obtain a filtered image; extracting edge structure features from the filtered image to obtain the first target feature; detecting the position information of the second target object in the grayscale image using a pre-trained feature extraction model; and extracting edge structure features from the grayscale image based on the detected position information to obtain the second target feature.

[0082] In this process, edge features of the first and second target objects in the surveillance image are extracted. Before obtaining the first and second target features, feature extraction can be performed on the surveillance image. This can be done by preprocessing the surveillance image and then extracting features from the preprocessed image. For example, the surveillance image can be preprocessed by grayscale conversion, image rotation, etc., and then edge features can be extracted from the processed image.

[0083] Specifically, taking the first target module as the transmission line and the second target module as the pole tower for illustration, for example, see... Figure 5 The steps for obtaining the features corresponding to the transmission line may include:

[0084] 1. Convert surveillance images to grayscale;

[0085] 2. Grayscale image edge feature extraction: After converting the input surveillance image to grayscale, edge feature extraction operators are used to extract the corresponding edge features. Specifically, edge feature extraction can be performed using operators such as the Canny operator and the Laplacian operator.

[0086] 3. Line segment detector extracts line segment features and filters: After converting the input surveillance image to grayscale, the line segment detector detects line segment features in the image and filters them based on the length and direction features of the line segment features, filtering out line segment features with a length less than a preset length threshold and horizontal line segment features.

[0087] 4. Extraction of transmission line edge structure features: The intersection of line segment features and edge features is obtained to obtain the final transmission line edge structure features. All transmission lines are retained and other noise is filtered out to finally extract the transmission line edge structure.

[0088] The steps for obtaining the features corresponding to the poles may include: pole target detection, using deep learning methods to train the corresponding detector and detect the position information of the poles in the image; pole edge structure feature extraction, based on the position information and edge feature map of the poles, directly extracting the edge features of the poles, and finally obtaining the pole edge features.

[0089] Specifically, by performing an inverse transform on the feature image according to the inverse transform formula to obtain the target feature map, an image edge inverse transform Map module can be constructed. Its characteristic is the construction of an image edge inverse transform Map for a given target, and the corresponding inverse transform formula is as follows:

[0090]

[0091] Where i,j are the pixel coordinates of the monitored image, and D i, For the inverse transformation Map, E i, Let x and y be the edge structure features corresponding to pixel coordinates (i,j) in the monitoring image, where x and y are the pixel coordinates selected from the monitoring image during the calculation process, and E is the edge structure features. x, The edge structure features corresponding to the selected pixel coordinates (x, y) can also be represented by (u, v) in practical applications, where a and b are preset coefficients. Based on the extracted edge structure features of transmission lines and towers, corresponding inverse image edge transformation maps can be constructed respectively; alternatively, an inverse image edge transformation map can be constructed based on the union of the edge structure features of transmission lines and towers.

[0092] As can be seen, the method of this application embodiment can extract edge features of the first target object and the second target object in the surveillance image, then construct a feature image based on the extracted first target features and second target features, and finally perform an inverse transformation on the feature image according to a preset inverse transformation formula to obtain a target feature map, thereby projecting the target point cloud onto the target feature map, detecting the error of the projection matrix based on the projection result, and then updating the projection matrix.

[0093] Optional, see Figure 6 Step S14 creates an initial projection matrix based on the initial values ​​of the target point cloud data and installation location information, and projects the target point cloud onto the target feature map using the initial projection matrix to obtain the first projection result, including:

[0094] Step S151: Create an initial projection matrix based on the initial values ​​of the target camera's mounting angle and mounting height.

[0095] Step S152: Project the target point cloud onto the target feature map using the initial projection matrix to obtain the first projection result;

[0096] Step S15 corrects the projection matrix based on the first projection result to obtain the target projection matrix, and calculates the extrinsic parameters of the target camera based on the target projection matrix, including: calculating the inverse edge transform value of the coordinate position of each point in the target point cloud corresponding to the target feature map based on the first projection result; creating an integral function corresponding to the inverse edge transform value of each point in the target point cloud to obtain the target function; calculating the projection matrix corresponding to the maximum value of the target function through a preset nonlinear parameter estimation algorithm to obtain the target projection matrix, wherein the target function represents the deviation between the projected target point cloud and the target feature map, and the larger the value of the target function, the smaller the corresponding deviation; and calculating the extrinsic parameters of the target camera based on the target projection matrix.

[0097] The process of calculating the extrinsic parameters of the target camera based on the target projection matrix includes: calculating the target camera's output extrinsic parameters based on the target projection matrix; updating the initial projection matrix based on the output extrinsic parameters; and continuing to execute the step of projecting the target point cloud onto the target feature map through the initial projection matrix to obtain the first projection result until a preset iteration stop condition is met. The output extrinsic parameters corresponding to the preset iteration stop condition are then used as the target camera's extrinsic parameters.

[0098] The process involves calculating the projection integral of the target point cloud onto a predetermined matrix to obtain the first projection result. This can be achieved by using the projection integral of the corresponding point cloud onto the initial projection matrix Map as the objective function, maximizing the objective function using a nonlinear parameter estimation method, and then calculating the projection integral of the target point cloud onto the predetermined matrix to obtain the first projection result. The corresponding objective function Jc is:

[0099]

[0100] N is the number of point clouds, Xp is the image coordinates corresponding to the reprojection of the point cloud coordinates; D Xp This represents the inverse edge transform value corresponding to the image coordinates of the point. In this embodiment, the world coordinates (x, y, z) of the point cloud are reprojected onto the image coordinates through the camera intrinsic and extrinsic parameter matrices, which is the aforementioned Xp. The optimized Jc function includes Xp, which in turn includes the camera extrinsic parameters (camera extrinsic parameters R, t; projection matrix). Therefore, the output extrinsic parameters of the target camera can be calculated based on the target projection matrix. The initial projection matrix is ​​updated based on the output extrinsic parameters, and the step of projecting the target point cloud onto the target feature map through the initial projection matrix to obtain the first projection result continues until a preset iteration stop condition is met. The output extrinsic parameters corresponding to the preset iteration stop condition are then used as the extrinsic parameters of the target camera. Specifically, the iteration stop condition can be that the difference between the results of two iterations is less than a preset threshold or the number of iterations has been reached.

[0101] As can be seen, the method of this application embodiment can calculate the projection matrix corresponding to the maximum value of the objective function by using a preset nonlinear parameter estimation algorithm, obtain the target projection matrix, and calculate the extrinsic parameters of the target camera based on the target projection matrix, thereby realizing the automatic calibration of the target camera in the power transmission line scenario and solving the problems of high manual cost, long time consumption and low calibration efficiency in the process of camera extrinsic parameter calibration.

[0102] To illustrate the method of this application embodiment, the following description uses a power transmission line as a reference object. (See attached image.) Figure 5 :

[0103] 1. Obtain the initial extrinsic parameters of the camera; Since the camera is mounted in the same direction as the power line and the camera is mounted at a certain height, you only need to adjust the point cloud model to the corresponding position to obtain the corresponding initial extrinsic parameters of the camera.

[0104] 1.1 Transmission Line and Tower Point Cloud Classification Module: The user-input point cloud model contains transmission lines, towers, and ground point clouds. The point cloud model is classified. First, a plane fitting method is used to approximately fit the ground point cloud. Then, based on a height threshold, the tower and transmission line point clouds are classified, and the remaining point cloud model is divided into transmission line point clouds and tower point clouds.

[0105] 1.2 Adjust the power line point cloud model so that the power line is parallel to the X-axis and perpendicular to the YOZ plane; adjust the input scene point cloud model, project the power line point cloud model along the Z-axis onto the XYO plane, fit the power line straight line equation on the XYO plane, calculate the angle between the straight line equation and the X-axis or Y-axis based on the straight line equation, rotate the target point cloud data so that the fitted straight line is parallel to the X-axis, at this time the power line target point cloud data is also perpendicular to the YOZ plane, and the Euler angle between the point cloud and the camera pose is approximately [-90°, 0°, 90°] or [90°, 0°, 90°].

[0106] 1.3 Based on the camera's orientation and installation height, determine the initial extrinsic parameters R and t of the camera; based on the camera's installation tower number and the monitored opposing tower number, obtain the centroid coordinates of the camera's installation tower, and use the corresponding x and y coordinates as the camera's translation vector x and y coordinates; construct the initial extrinsic parameters t of the camera. The Euler angles of the camera's pose can be further confirmed by the camera's installation tower number and the monitored opposing tower number. When the camera's world coordinate x is greater than the opposing tower's world coordinate x, the initial Euler angles are [-90°, 0°, 90°]; otherwise, they are [90°, 0°, 90°].

[0107] 2. Automated camera extrinsic parameter calibration: Based on the acquired initial camera extrinsic parameters, the camera extrinsic parameters are optimized using the edge structural features of power lines and towers in the scene, and automated camera extrinsic parameter calibration is performed.

[0108] 2.1. Transmission line edge structure feature extraction module; including: converting surveillance images to grayscale; grayscale image edge feature extraction, after converting the input surveillance image to grayscale, using an edge feature extraction operator to extract the corresponding edge features, including edge features of transmission lines, towers, and noise; line segment detector extracting line segment features and filtering, after converting the input surveillance image to grayscale, using a line segment detector to detect line segment features in the image, and filtering based on the length and direction features of the line segment features to filter out short line segment features and horizontal line segment features; transmission line edge structure feature extraction, taking the intersection of line segment features and edge features to obtain the final transmission line edge structure features, retaining all transmission lines and filtering out other noise, finally extracting the transmission line edge structure features.

[0109] 2.2. Tower edge structure feature extraction module; including: tower target detection, which uses deep learning to train the corresponding detector to detect the position information of the tower in the image; tower edge structure feature extraction, which directly extracts the edge features of the tower based on the position information and edge feature map of the tower.

[0110] 2.3. Construct the Image Edge Inverse Transformation Map Module; The corresponding formula for constructing the Image Edge Inverse Transformation Map module is as follows:

[0111]

[0112] Based on the extracted edge structure features of transmission lines and towers, a corresponding inverse edge transformation map can be constructed; alternatively, an inverse edge transformation map can be constructed based on the union of the edge structure features of transmission lines and towers.

[0113] After constructing each submodule, the projection integral of the corresponding point cloud in the inverse transform map of the image edges is used as the corresponding objective function. A nonlinear parameter estimation method is employed to maximize the objective function, thereby optimizing the camera extrinsic parameters. Specifically, the camera extrinsic parameters are updated based on the calculation results of the optimized objective function. Then, the point cloud model is reprojected based on the updated camera extrinsic parameters, and the optimized objective function is calculated based on the reprojection results. This process continues until the required number of iterations is met, yielding the final camera extrinsic parameters.

[0114] Define the optimization objective function:

[0115]

[0116] Where N is the number of point clouds, and Xp is the image coordinates corresponding to the reprojected point cloud coordinates. The objective function is maximized by nonlinear optimization to obtain the inverse edge transformation value corresponding to the reprojection coordinate position, and then used for automatic calibration of camera extrinsic parameters.

[0117] A second aspect of this application provides another method for calibrating camera extrinsic parameters, see [link to relevant documentation]. Figure 7 ,include:

[0118] Step S71: Acquire the monitoring image captured by the target camera, wherein the monitoring image includes the power transmission line;

[0119] Step S72: Extract edge structure features from the monitored image of the power transmission line, and create a target feature map based on the edge structure features of the power transmission line.

[0120] Step S73: Obtain the initial values ​​of the target point cloud data of the scene where the target camera is located and the installation location information of the target camera, wherein the scene where the target camera is located includes power transmission lines and the target point cloud includes power transmission line point clouds.

[0121] Step S74: Create an initial projection matrix based on the initial values ​​of the transmission line point cloud and installation location information, and project the transmission line point cloud onto the target feature map through the initial projection matrix to obtain the first projection result. The initial projection matrix represents the correspondence between the world coordinate system and the camera coordinate system of the target camera.

[0122] Step S75: Correct the projection matrix according to the first projection result to obtain the target projection matrix, and calculate the extrinsic parameters of the target camera according to the target projection matrix.

[0123] Optionally, the projection matrix is ​​corrected based on the first projection result to obtain the target projection matrix, and the extrinsic parameters of the target camera are calculated based on the target projection matrix, including:

[0124] An objective function is created based on the first projection result, where the objective function represents the deviation between the projected target point cloud and the target feature map. The larger the value of the objective function, the smaller the corresponding deviation.

[0125] Calculate the target projection matrix corresponding to the maximum value of the objective function, and label the extrinsic parameters corresponding to the target projection matrix as the extrinsic parameters of the target camera.

[0126] Optionally, initial values ​​for obtaining target point cloud data of the scene where the target camera is located and initial values ​​for the installation location information of the target camera include:

[0127] Identify power line point clouds in target point cloud data based on a preset height threshold;

[0128] The point cloud of the transmission line is projected onto the XOY plane to obtain the second projection result. The XOY plane is a horizontal plane and is the plane containing the X-axis, Y-axis, and the intersection point O of the X-axis and Y-axis.

[0129] Identify the projection corresponding to the transmission line in the second projection result;

[0130] Calculate the direction of the projection of the transmission line in the second projection result and the angle between it and the X-axis and / or Y-axis to obtain the initial value of the installation angle of the target camera.

[0131] Optionally, an initial projection matrix is ​​created based on the initial values ​​of the transmission line point cloud and installation location information, including:

[0132] Obtain the initial value of the installation height of the target camera, where;

[0133] An initial projection matrix is ​​created based on the initial values ​​of the target camera's installation angle and installation height.

[0134] Optionally, the monitored image may also include the tower. After acquiring the monitored image captured by the target camera, the method further includes:

[0135] Edge structure features of power transmission lines and towers in surveillance images are extracted to obtain the edge structure features of power transmission lines and towers.

[0136] Create feature images based on the edge structure features of transmission lines and / or towers;

[0137] The feature image is inversely transformed according to the preset inverse transform formula to obtain the target feature map.

[0138] Optionally, edge structure features of power lines and towers in the surveillance images are extracted to obtain the edge structure features of power lines and towers, including:

[0139] The captured images are converted to grayscale to obtain grayscale images;

[0140] Linear feature filtering is applied to the transmission line in the grayscale image to obtain the filtered image; edge structure features are extracted from the filtered image to obtain the edge structure features of the transmission line.

[0141] The location information of the tower in the grayscale image is detected by a pre-trained feature extraction model; based on the detected location information, the edge structure features of the grayscale image are extracted to obtain the edge structure features of the tower.

[0142] Optionally, an initial projection matrix is ​​created based on the initial values ​​of the transmission line point cloud and installation location information, and the transmission line point cloud is projected onto the target feature map using the initial projection matrix to obtain a first projection result, including:

[0143] An initial projection matrix is ​​created based on the initial values ​​of the target camera's installation angle and installation height.

[0144] The target point cloud is projected onto the target feature map using the initial projection matrix to obtain the first projection result.

[0145] Optionally, the projection matrix is ​​corrected based on the first projection result to obtain the target projection matrix, and the extrinsic parameters of the target camera are calculated based on the target projection matrix, including:

[0146] Based on the first projection result, calculate the edge inverse transform value of the coordinate position of each point in the target point cloud corresponding to the target feature map;

[0147] Create the integral function corresponding to the inverse edge transform value of each point in the target point cloud to obtain the target function;

[0148] By using a pre-defined nonlinear parameter estimation algorithm, the projection matrix corresponding to the maximum value of the objective function is calculated to obtain the target projection matrix. Here, the objective function represents the deviation between the projected target point cloud and the target feature map. The larger the value of the objective function, the smaller the corresponding deviation.

[0149] Calculate the extrinsic parameters of the target camera based on the target projection matrix.

[0150] Optionally, based on the target projection matrix, the extrinsic parameters of the target camera are calculated, including:

[0151] Calculate the target camera's extrinsic parameters based on the target projection matrix;

[0152] The process of updating the initial projection matrix based on the extrinsic parameters to be output, and then returning to the step of projecting the target point cloud onto the target feature map through the initial projection matrix to obtain the first projection result, continues until the preset iteration stop condition is met. The extrinsic parameters to be output corresponding to the preset iteration stop condition are then used as the extrinsic parameters of the target camera.

[0153] As can be seen, the apparatus of this application embodiment can acquire target point cloud data and installation location information of the target camera in the power transmission line scene, as well as the monitoring images captured by the target camera. By identifying the target point cloud and projecting it onto the target feature map through an initial projection matrix to obtain the first projection result, the projection matrix is ​​solved, and the extrinsic parameters of the target camera are calculated based on the final target projection matrix. This achieves automatic calibration of the target camera in the power transmission line scene, solving the problems of high manual cost, long time consumption, and low calibration efficiency in the camera extrinsic parameter calibration process.

[0154] A third aspect of this application provides a camera extrinsic parameter calibration device, see [link to relevant documentation]. Figure 8 ,include:

[0155] Image acquisition module 801 is used to acquire monitoring images captured by the target camera;

[0156] The matrix creation module 802 is used to extract features from the captured images and create a target feature map based on the extracted features.

[0157] The location acquisition module 803 is used to acquire the initial values ​​of the target point cloud data of the scene where the target camera is located and the installation location information of the target camera;

[0158] The projection result acquisition module 804 is used to create an initial projection matrix based on the initial values ​​of the target point cloud data and the installation location information, and to project the target point cloud onto the target feature map through the initial projection matrix to obtain the first projection result.

[0159] The matrix correction module 805 is used to correct the projection matrix according to the first projection result to obtain the target projection matrix, and to calculate the extrinsic parameters of the target camera according to the target projection matrix.

[0160] Optionally, the scene where the target camera is located includes a first target object and a second target object; the target point cloud data of the scene where the target camera is located includes the point cloud of the first target object; the initial value of the target camera's installation position information includes the initial value of the target camera's installation height; the position acquisition module 803 includes:

[0161] The point cloud data acquisition submodule is used to acquire target point cloud data of the scene where the target camera is located;

[0162] The point cloud data projection submodule is used to project the target point cloud data onto a horizontal plane to obtain a second projection result;

[0163] The initial angle calculation submodule is used to identify and calculate the initial value of the installation angle of the target camera based on the position of the first target object in the second projection result.

[0164] The initial height calculation submodule is used to obtain the preset attribute information of the second target object and calculate the initial value of the installation height of the target camera based on the preset attribute information.

[0165] Optional, the point cloud data projection submodule includes:

[0166] The point cloud recognition submodule is used to identify the point cloud of the first target object in the scene point cloud according to a preset height threshold.

[0167] The point cloud projection submodule is used to project the point cloud of the first target object onto the XOY plane to obtain the second projection result. The XOY plane is a horizontal plane and is the plane containing the X-axis, Y-axis, and the intersection point O of the X-axis and Y-axis.

[0168] The initial angle calculation submodule includes:

[0169] A projection recognition unit is used to identify the projection corresponding to the first target object in the second projection result;

[0170] Angle recognition unit is used to calculate the direction of the projection of the first target object in the second projection result, and the angle between the projection and the X-axis and / or Y-axis, to obtain the initial value of the installation angle of the target camera.

[0171] Optionally, the matrix creation module 802 includes:

[0172] The edge structure feature extraction submodule is used to extract the edge features of the first target object and the second target object in the surveillance image to obtain the first target feature and the second target feature.

[0173] The feature image construction submodule is used to construct a feature image based on the first target feature and / or the second target feature;

[0174] The image inverse transform submodule is used to perform an inverse transform on the feature image according to a preset inverse transform formula to obtain the target feature map.

[0175] Optional, edge structure feature extraction submodule, including:

[0176] The grayscale unit is used to perform grayscale processing on the captured image to obtain a grayscale image.

[0177] The feature filtering unit is used to perform linear feature filtering on the first target object in the grayscale image to obtain the filtered image; and to extract the edge structure features of the filtered image to obtain the first target features.

[0178] The position detection unit is used to detect the position information of the second target object in the grayscale image through a pre-trained feature extraction model; based on the detected position information, the edge structure features of the grayscale image are extracted to obtain the features of the second target.

[0179] Optionally, the projection result acquisition module 804 includes:

[0180] The initial projection matrix creation submodule is used to create an initial projection matrix based on the initial values ​​of the target camera's mounting angle and mounting height.

[0181] The initial matrix projection submodule is used to project the target point cloud onto the target feature map using an initial projection matrix to obtain the first projection result.

[0182] Optionally, the matrix correction module 805 includes:

[0183] The transformation value calculation submodule is used to calculate the edge inverse transformation value of each point in the target point cloud and the corresponding coordinate position of the target feature map based on the first projection result.

[0184] The integral function creation submodule is used to create the integral function corresponding to the inverse edge transform value of each point in the target point cloud, and obtain the target function.

[0185] The maximum value calculation submodule is used to calculate the projection matrix corresponding to the maximum value of the objective function through a preset nonlinear parameter estimation algorithm, and obtain the target projection matrix. Here, the objective function represents the deviation between the projected target point cloud and the target feature map. The larger the value of the objective function, the smaller the corresponding deviation.

[0186] The extrinsic parameter calculation submodule is used to calculate the extrinsic parameters of the target camera based on the target projection matrix.

[0187] Optional extrinsic parameter calculation submodule, including:

[0188] The extrinsic parameter calculation unit is used to calculate the extrinsic parameters to be output by the target camera based on the target projection matrix.

[0189] The extrinsic parameter generation unit is used to update the initial projection matrix according to the extrinsic parameters to be output, and return the step of projecting the target point cloud onto the target feature map through the initial projection matrix to obtain the first projection result. This process continues until a preset iteration stop condition is met, and the extrinsic parameters to be output corresponding to the preset iteration stop condition are used as the extrinsic parameters of the target camera.

[0190] As can be seen, the apparatus of this application embodiment can obtain target point cloud data of the scene where the target camera is located, the installation location information of the target camera, and the monitoring images captured by the target camera. By identifying the target point cloud and projecting it onto the target feature map through an initial projection matrix to obtain the first projection result, the projection matrix is ​​solved, and the extrinsic parameters of the target camera are calculated based on the final target projection matrix. This achieves automatic calibration of the target camera and solves the problems of high manual cost, long time consumption, and low calibration efficiency in the process of camera extrinsic parameter calibration.

[0191] A fourth aspect of the embodiments of this application provides another camera extrinsic parameter calibration device, see [link to relevant documentation]. Figure 9 ,include:

[0192] The monitoring image acquisition module 901 is used to acquire monitoring images captured by the target camera, wherein the monitoring images include power transmission lines;

[0193] The structural feature extraction module 902 is used to extract the edge structural features of the power transmission line in the surveillance image, and to obtain and create a target feature map based on the edge structural features of the power transmission line.

[0194] The point cloud acquisition module 903 is used to acquire the initial values ​​of the target point cloud data of the scene where the target camera is located and the installation location information of the target camera. The scene where the target camera is located includes power transmission lines, and the target point cloud includes the power transmission line point cloud.

[0195] The point cloud projection module 904 is used to create an initial projection matrix based on the initial values ​​of the transmission line point cloud and installation location information, and to project the transmission line point cloud onto the target feature map through the initial projection matrix to obtain the first projection result. The initial projection matrix represents the correspondence between the world coordinate system and the camera coordinate system of the target camera.

[0196] The camera extrinsic parameter calculation module 905 is used to correct the projection matrix based on the first projection result to obtain the target projection matrix, and to calculate the extrinsic parameters of the target camera based on the target projection matrix.

[0197] Optionally, the camera extrinsic parameter calculation module 905 includes:

[0198] The function creation submodule is used to create an objective function based on the first projection result. The objective function represents the deviation between the projected target point cloud and the target feature map. The larger the value of the objective function, the smaller the corresponding deviation.

[0199] The extrinsic parameter calculation submodule is used to calculate the target projection matrix corresponding to the maximum value of the objective function, and the extrinsic parameters corresponding to the target projection matrix are marked as the extrinsic parameters of the target camera.

[0200] Optionally, the point cloud acquisition module 903 includes:

[0201] The threshold judgment submodule is used to identify power line point clouds in the target point cloud data based on a preset height threshold.

[0202] The horizontal plane projection submodule is used to project the transmission line point cloud onto the XOY plane to obtain a second projection result. The XOY plane is a horizontal plane, which is the plane containing the X-axis, Y-axis, and the intersection point O of the X-axis and Y-axis.

[0203] The projection recognition submodule is used to identify the projection corresponding to the transmission line in the second projection result;

[0204] The angle recognition submodule is used to calculate the direction of the projection of the transmission line in the second projection result and the angle between it and the X-axis and / or Y-axis to obtain the initial value of the installation angle of the target camera.

[0205] Optionally, the point cloud projection module 904 includes:

[0206] The installation height acquisition submodule is used to obtain the initial value of the installation height of the target camera;

[0207] The projection matrix creation submodule is used to create an initial projection matrix based on the initial values ​​of the target camera's mounting angle and mounting height.

[0208] Optionally, the monitored image may also include the tower, and the aforementioned device may also include:

[0209] The structural feature extraction module is used to extract the edge structural features of power transmission lines and towers in the surveillance images, and obtain the edge structural features of power transmission lines and towers.

[0210] The feature map creation module is used to create feature images based on the edge structure features of transmission lines and / or towers.

[0211] The image inverse transformation module is used to perform an inverse transformation on the feature image according to a preset inverse transformation formula to obtain the target feature map.

[0212] Optional, the structural feature extraction module includes:

[0213] The grayscale processing submodule is used to convert the captured images to grayscale to obtain grayscale images;

[0214] The linear filtering submodule is used to perform linear feature filtering on the transmission lines in the grayscale image to obtain the filtered image; and to extract the edge structure features of the transmission lines from the filtered image.

[0215] The feature acquisition submodule is used to detect the position information of the tower in the grayscale image through a pre-trained feature extraction model; based on the detected position information, the edge structure features of the grayscale image are extracted to obtain the edge structure features of the tower.

[0216] Optionally, the point cloud projection module 904 includes:

[0217] The projection matrix creation submodule is used to create an initial projection matrix based on the initial values ​​of the target camera's mounting angle and mounting height.

[0218] The projection result acquisition submodule is used to project the target point cloud onto the target feature map using the initial projection matrix to obtain the first projection result.

[0219] Optionally, the camera extrinsic parameter calculation module 905 includes:

[0220] The transformation value acquisition submodule is used to calculate the edge inverse transformation value of each point in the target point cloud and the corresponding coordinate position of the target feature map based on the first projection result.

[0221] The objective function creation submodule is used to create the integral function corresponding to the inverse edge transform value of each point in the target point cloud, thus obtaining the objective function.

[0222] The matrix acquisition submodule is used to calculate the projection matrix corresponding to the maximum value of the objective function through a preset nonlinear parameter estimation algorithm, and obtain the target projection matrix. Here, the objective function represents the deviation between the projected target point cloud and the target feature map. The larger the value of the objective function, the smaller the corresponding deviation.

[0223] The extrinsic parameter calculation submodule is used to calculate the extrinsic parameters of the target camera based on the target projection matrix.

[0224] Optional extrinsic parameter calculation submodule, including:

[0225] The extrinsic parameter calculation unit is used to calculate the output extrinsic parameters of the target camera based on the target projection matrix.

[0226] The extrinsic parameter acquisition unit is used to update the initial projection matrix according to the extrinsic parameters to be output, and return the step of projecting the target point cloud onto the target feature map through the initial projection matrix to obtain the first projection result. The process continues until the preset iteration stop condition is met, and the extrinsic parameters to be output corresponding to the preset iteration stop condition are used as the extrinsic parameters of the target camera.

[0227] As can be seen, the apparatus of this application embodiment can acquire target point cloud data and installation location information of the target camera in the power transmission line scene, as well as the monitoring images captured by the target camera. By identifying the target point cloud and projecting it onto the target feature map through an initial projection matrix to obtain the first projection result, the projection matrix is ​​solved, and the extrinsic parameters of the target camera are calculated based on the final target projection matrix. This achieves automatic calibration of the target camera in the power transmission line scene, solving the problems of high manual cost, long time consumption, and low calibration efficiency in the camera extrinsic parameter calibration process.

[0228] This application also provides an electronic device, such as... Figure 10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0229] Memory 1003 is used to store computer programs;

[0230] When processor 1001 executes a program stored in memory 1003, it performs the following steps:

[0231] Acquire surveillance images captured by the target camera;

[0232] Feature extraction is performed on the surveillance images, and a target feature map is created based on the extracted features;

[0233] Initial values ​​for target point cloud data of the scene where the target camera is located and initial values ​​for the installation location information of the target camera;

[0234] An initial projection matrix is ​​created based on the initial values ​​of the target point cloud data and installation location information. The target point cloud is then projected onto the target feature map using the initial projection matrix to obtain the first projection result.

[0235] The projection matrix is ​​corrected based on the first projection result to obtain the target projection matrix, and the extrinsic parameters of the target camera are calculated based on the target projection matrix. This application also provides an electronic device, such as... Figure 11 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104. The processor 1101, communication interface 1102, and memory 1103 communicate with each other via the communication bus 1104.

[0236] Memory 1103 is used to store computer programs;

[0237] When processor 1101 executes the program stored in memory 1103, it performs the following steps:

[0238] Acquire surveillance images captured by the target camera, wherein the surveillance images include power transmission lines;

[0239] Edge structure features of power transmission lines in surveillance images are extracted, and a target feature map is created based on the edge structure features of the power transmission lines.

[0240] Initial values ​​are obtained for the target point cloud data of the scene where the target camera is located and the installation location information of the target camera. The scene where the target camera is located includes power lines, and the target point cloud includes the power line point cloud.

[0241] An initial projection matrix is ​​created based on the initial values ​​of the transmission line point cloud and installation location information. The transmission line point cloud is then projected onto the target feature map using the initial projection matrix to obtain the first projection result. The initial projection matrix represents the correspondence between the world coordinate system and the camera coordinate system of the target camera.

[0242] The projection matrix is ​​corrected based on the first projection result to obtain the target projection matrix, and the extrinsic parameters of the target camera are calculated based on the target projection matrix.

[0243] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0244] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0245] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0246] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0247] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described camera extrinsic parameter calibration methods.

[0248] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the camera extrinsic parameter calibration methods described above.

[0249] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0250] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0251] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of devices, electronic devices, storage media, and computer program products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0252] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for calibrating camera extrinsic parameters, characterized in that, The method comprises the following steps: acquiring a monitoring image collected by a target camera, wherein the monitoring image comprises a power transmission line; extracting edge structure features of the power transmission line in the monitoring image to obtain a target feature map created according to the edge structure features of the power transmission line; acquiring target point cloud data of a scene where the target camera is located and an initial value of installation position information of the target camera, wherein the scene where the target camera is located comprises the power transmission line, and the target point cloud data comprises power transmission line point cloud data; creating an initial projection matrix according to the power transmission line point cloud data and the initial value of the installation position information, and projecting the power transmission line point cloud data onto the target feature map through the initial projection matrix to obtain a first projection result, wherein the initial projection matrix represents a corresponding relationship between a world coordinate system and a camera coordinate system of the target camera; correcting the projection matrix according to the first projection result to obtain a target projection matrix, and calculating extrinsic parameters of the target camera according to the target projection matrix.

2. The method of claim 1, wherein, The method of correcting the projection matrix according to the first projection result to obtain a target projection matrix, and calculating extrinsic parameters of the target camera according to the target projection matrix comprises: creating a target function according to the first projection result, wherein the target function represents a deviation between the projected target point cloud data and the target feature map, and the greater the value of the target function is, the smaller the corresponding deviation is; calculating a target projection matrix corresponding to a maximum value of the target function, and marking the target projection matrix as the extrinsic parameters of the target camera.

3. The method of claim 1, wherein, The method of acquiring target point cloud data of a scene where the target camera is located and an initial value of installation position information of the target camera comprises: identifying the power transmission line point cloud data in the target point cloud data according to a preset height threshold; projecting the power transmission line point cloud data onto an XOY plane to obtain a second projection result, wherein the XOY plane is a horizontal plane, and the XOY plane is a plane where an X-axis, a Y-axis and an intersection O point of the X-axis and the Y-axis are located; identifying a projection corresponding to the power transmission line in the second projection result; calculating an angle between a pointing direction of the projection of the power transmission line in the second projection result and an X-axis and / or a Y-axis to obtain an initial value of an installation angle of the target camera.

4. The method of claim 3, wherein, The method of creating an initial projection matrix according to the power transmission line point cloud data and the initial value of the installation position information comprises: acquiring an initial value of an installation height of the target camera; creating the initial projection matrix according to the initial value of the installation angle of the target camera and the initial value of the installation height of the target camera.

5. The method of claim 1, wherein, The monitoring image further comprises a tower, and after the step of acquiring the monitoring image collected by the target camera, the method further comprises: extracting edge structure features of the power transmission line and the tower in the monitoring image to obtain edge structure features of the power transmission line and edge structure features of the tower; creating a feature image according to the edge structure features of the power transmission line and / or the edge structure features of the tower; performing inverse transformation on the feature image according to a preset inverse transformation formula to obtain the target feature map.

6. The method of claim 5, wherein, The edge structure feature extraction of the transmission line and the tower in the monitoring image comprises: The monitoring image is subjected to a gray processing to obtain a gray image; The transmission line in the gray image is subjected to a linear feature filtering to obtain a filtered image, and the edge structure feature extraction of the filtered image is performed to obtain the edge structure feature of the transmission line; The position information of the tower in the gray image is detected through a pre-trained feature extraction model, and the edge structure feature extraction of the gray image is performed according to the detected position information to obtain the edge structure feature of the tower.

7. The method of claim 1, wherein, The initial projection matrix is created according to the initial value of the transmission line point cloud and the installation position information, and the target point cloud is projected onto the target feature map through the initial projection matrix to obtain a first projection result, which comprises: The initial projection matrix is created according to the initial value of the installation angle of the target camera and the initial value of the installation height of the target camera; The target point cloud is projected onto the target feature map through the initial projection matrix to obtain the first projection result.

8. The method of claim 7, wherein, The target projection matrix is obtained by correcting the projection matrix according to the first projection result, and the extrinsic parameter of the target camera is calculated according to the target projection matrix, which comprises: The edge inverse transformation value corresponding to the coordinate position of the target feature map of each point in the target point cloud is calculated according to the first projection result; An integral function corresponding to the edge inverse transformation value of each point in the target point cloud is created to obtain a target function; The target projection matrix is obtained by calculating the projection matrix corresponding to the maximum value of the target function through a preset nonlinear parameter estimation algorithm, wherein the target function represents the deviation of the projected target point cloud and the target feature map, and the greater the value of the target function, the smaller the corresponding deviation; The extrinsic parameter of the target camera is calculated according to the target projection matrix.

9. The method of claim 8, wherein, The extrinsic parameter of the target camera is calculated according to the target projection matrix, which comprises: The to-be-output extrinsic parameter of the target camera is calculated according to the target projection matrix; The initial projection matrix is updated according to the to-be-output extrinsic parameter, and the step of projecting the target point cloud onto the target feature map through the initial projection matrix to obtain the first projection result is continued to be executed until a preset iteration stopping condition is met, and the to-be-output extrinsic parameter corresponding to the preset iteration stopping condition is taken as the extrinsic parameter of the target camera. 10.A method for calibrating camera extrinsic parameters, characterized in that, It comprises: A monitoring image collected by a target camera is obtained; Features are extracted from the monitoring image, and a target feature map is created according to the extracted features; Initial values of target point cloud data of a scene where the target camera is located and installation position information of the target camera are obtained; An initial projection matrix is created according to the initial values of the target point cloud data and the installation position information, and the target point cloud is projected onto the target feature map through the initial projection matrix to obtain a first projection result. According to the first projection result, the projection matrix is corrected to obtain a target projection matrix, and the extrinsic parameter of the target camera is calculated according to the target projection matrix.

11. The method of claim 10, wherein, The scene where the target camera is located includes a first target object and a second target object, target point cloud data of the scene where the target camera is located includes point cloud of the first target object, and the initial value of the installation position information of the target camera includes an initial value of the installation height of the target camera, the obtaining of the target point cloud data of the scene where the target camera is located and the initial value of the installation position information of the target camera includes: Obtaining target point cloud data of a scene where a target camera is located; Projecting the target point cloud data to a horizontal plane to obtain a second projection result; Identifying and calculating an initial value of an installation angle of the target camera according to a position of the first target object in the second projection result; Obtaining preset attribute information of the second target object, and calculating an initial value of the installation height of the target camera according to the preset attribute information.

12. The method of claim 11, wherein, The projecting of the target point cloud data to a horizontal plane to obtain a second projection result includes: Identifying point cloud of the first target object in the scene point cloud according to a preset height threshold; Projecting the point cloud of the first target object to an XOY plane to obtain a second projection result, wherein the XOY plane is a horizontal plane, and the XOY plane is a plane where an X-axis, a Y-axis and an intersection O point of the X-axis and the Y-axis are located; The identifying and calculating of the initial value of the installation angle of the target camera according to the position of the first target object in the second projection result includes: Identifying a projection corresponding to the first target object in the second projection result; Calculating an angle between a pointing direction of the projection of the first target object in the second projection result and the X-axis and / or the Y-axis to obtain the initial value of the installation angle of the target camera.

13. The method of claim 11, wherein, The feature extraction from the monitoring image and the creation of a target feature map according to the extracted features include: Extracting edge features of the first target object and the second target object in the monitoring image to obtain first target features and second target features; Constructing a feature image according to the first target features and / or the second target features; Performing inverse transformation on the feature image according to a preset inverse transformation formula to obtain the target feature map.

14. The method of claim 13, wherein, The extracting of the first target features and the second target features from the first target object and the second target object in the monitoring image includes: Performing grayscale processing on the monitoring image to obtain a grayscale image; Performing linear feature filtering on the first target object in the grayscale image to obtain a filtered image, and extracting edge structural features from the filtered image to obtain the first target features; Detecting position information of the second target object in the grayscale image through a pre-trained feature extraction model, and extracting edge structural features from the grayscale image according to the detected position information to obtain the second target features.

15. The method of claim 10, wherein, The initial projection matrix is created according to the initial value of the target point cloud data and the installation position information, and the target point cloud is projected onto the target feature map through the initial projection matrix to obtain a first projection result, including: The initial projection matrix is created according to the initial value of the installation angle of the target camera and the initial value of the installation height of the target camera; The target point cloud is projected onto the target feature map through the initial projection matrix to obtain the first projection result.

16. The method of claim 15, wherein, The target projection matrix is obtained by correcting the projection matrix according to the first projection result, and the extrinsic parameter of the target camera is calculated according to the target projection matrix, including: According to the first projection result, the edge inverse transformation value corresponding to the coordinate position of each point in the target point cloud on the target feature map is calculated; An integral function corresponding to the edge inverse transformation value of each point in the target point cloud is created to obtain a target function; The target projection matrix is obtained by calculating the maximum value of the target function corresponding to the projection matrix through a preset nonlinear parameter estimation algorithm, wherein the target function represents the deviation of the projected target point cloud and the target feature map, and the greater the value of the target function, the smaller the corresponding deviation; The extrinsic parameter of the target camera is calculated according to the target projection matrix.

17. The method of claim 16, wherein, The extrinsic parameter of the target camera is calculated according to the target projection matrix, including: The target camera output extrinsic parameter is calculated according to the target projection matrix; The initial projection matrix is updated according to the target camera output extrinsic parameter, and the step of projecting the target point cloud onto the target feature map through the initial projection matrix to obtain the first projection result is continued to be executed until a preset iteration stop condition is met, and the target camera output extrinsic parameter corresponding to the preset iteration stop condition is taken as the extrinsic parameter of the target camera.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps of any one of claims 1-9 or 10-17. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps of any one of claims 1-9 or 10-17.

Images