A camera preset position correction method, device and storage medium

Abstract

The application discloses a camera preset position correction method, equipment and a storage medium. The method comprises the following steps: acquiring a first acquisition image corresponding to a target preset position of a camera, and acquiring a reference image and reference depth data corresponding to the target preset position, wherein the first acquisition image is collected after a holder of the camera rotates according to holder attitude parameter corresponding to the target preset position; calculating a current absolute pose of the camera by using a matching result between the first acquisition image and the reference image, and the reference depth data; determining a first pose deviation of the camera at the target preset position based on the current absolute pose; and correcting at least the target preset position of the camera based on the first pose deviation. In this way, the application can improve the camera preset position correction efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a camera preset position correction method, device, and storage medium. Background Technology

[0002] Video capture systems are commonly used for capturing images and recording status in specific scenes and areas. Taking a substation as an example, this system uses cameras to capture images or record videos, enabling real-time monitoring of the equipment status within the station. To facilitate accurate coverage of specific locations within the area, preset positions need to be set for the cameras. In related technologies, preset positions are set manually. Specifically, this involves operating the camera's pan / tilt joystick, directional buttons, or 3D positioning function to adjust the camera's rotation angle and zoom level, and then combining this with the on-site footage to complete the preset position setting.

[0003] However, due to the large rotational inertia of the camera gimbal load, the gimbal drive motor requires significant torque. The backlash in the mechanical gears and the precision of structural components in the drive motor can lead to accuracy errors in the preset position. Furthermore, prolonged use of the preset position can cause these errors to accumulate, and due to factors such as mechanical wear, the preset position may deviate from its initial setting, thus affecting the system's ability to properly view and record the device's status. Currently, preset position correction mainly relies on manual operation, which is time-consuming, labor-intensive, and consumes a large amount of human resources. Especially in scenarios such as substations and chemical energy plants, a single scenario often has thousands of cameras, each with an average of hundreds of preset positions. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a camera preset position correction method, apparatus, and storage medium to improve the efficiency and accuracy of camera preset position correction.

[0005] According to one embodiment of this application, a camera preset position correction method is provided, including:

[0006] Acquire a first image of the camera corresponding to a target preset position, and acquire a reference image and reference depth data corresponding to the target preset position, wherein the first image is acquired after the camera's gimbal is rotated according to the gimbal attitude parameters corresponding to the target preset position;

[0007] The current absolute pose of the camera is calculated using the matching result between the first acquired image and the reference image, as well as the reference depth data.

[0008] Based on the current absolute pose, determine the first pose deviation of the camera at the target preset position;

[0009] Based on the first pose deviation, at least the target preset position of the camera is corrected.

[0010] To solve the above-mentioned technical problems, one technical solution adopted in this application is to provide an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the camera preset position correction method in the above-mentioned technical solution.

[0011] To solve the above-mentioned technical problems, one technical solution adopted in this application is to provide a computer-readable storage medium for storing a computer program, which, when executed by a processor, is used to implement the camera preset position correction method in the above-mentioned technical solution.

[0012] The beneficial effects of this application through the above scheme are as follows: The camera preset position correction method provided by this application acquires a first acquired image of the camera corresponding to the target preset position, as well as a reference image and reference depth data corresponding to the target preset position. Using the matching result between the first acquired image and the reference image and the reference depth data, the current absolute pose of the camera is calculated. Based on the current absolute pose, the first pose deviation of the camera at the target preset position is determined. Based on the first pose deviation, at least the target preset position of the camera is corrected. In this way, this application, through the collaborative assisted positioning of image matching and depth data, can more accurately calculate the absolute pose deviation of the camera without relying on human visual judgment, and implements automatic correction of the preset position based on the quantified deviation. This eliminates the inefficiency and error interference of traditional manual operation, thereby significantly improving the efficiency and accuracy of camera preset position correction. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0014] Figure 1 This is a flowchart illustrating an embodiment of the camera preset position correction method provided in this application;

[0015] Figure 2 This is a flowchart illustrating another embodiment of the camera preset position correction method provided in this application;

[0016] Figure 3 A schematic diagram illustrating a method for acquiring a reference depth image provided in this application;

[0017] Figure 4 This is a flowchart illustrating a specific embodiment of the camera preset position correction method provided in this application;

[0018] Figure 5This is a schematic diagram of a camera preset position correction system provided in this application;

[0019] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application;

[0020] Figure 7 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0021] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the application. Similarly, the following embodiments are only some, not all, embodiments of the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0022] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] It should be noted that the terms "first," "second," etc., used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0024] Please see Figure 1 , Figure 1 This is a schematic flowchart of an embodiment of the camera preset position correction method provided in this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily reflect that result. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this embodiment includes:

[0025] S110: Acquire the first image of the target preset position corresponding to the camera, and acquire the reference image and reference depth data corresponding to the target preset position.

[0026] A target preset position refers to a baseline working state pre-set for the camera to meet specific business needs, including gimbal attitude parameters and imaging parameters, corresponding to a specific area of ​​interest in the scene. The first acquired image is captured after the camera's gimbal rotates according to the gimbal attitude parameters corresponding to the target preset position. That is, the real-time image data captured by the camera at the target preset position in response to the preset position call command or periodic acquisition command within the current working cycle, reflecting the scene state of the camera in its current pose.

[0027] The reference image is the baseline image data acquired and stored by the camera in the initial configuration stage of the target preset position, when it is in a standard pose, for the same shooting area. Its shooting scene and imaging parameters are consistent with the first acquired image. The reference depth data is three-dimensional spatial position data from the same source as the reference image. It consists of the spatial coordinate information of each feature point in the standard scene of the target preset position. It can be directly acquired by a depth camera or generated by processing the reference image based on a binocular vision algorithm. Alternatively, it can be obtained through depth simulation based on a twin 3D scene of the current scene, and is used to assist in the conversion calculation from two-dimensional image features to three-dimensional spatial pose.

[0028] In one embodiment, the reference image and reference depth data corresponding to the target preset position are configured through the following steps: After determining that the camera is located at the target preset position, the image captured by the camera is acquired as the reference image corresponding to the target preset position, and the current gimbal attitude parameters of the gimbal are acquired as the gimbal attitude parameters configured for the target preset position. Using the camera's calibration parameters and the gimbal attitude parameters configured for the target preset position, the first intrinsic parameter and the first extrinsic parameter of the camera at the target preset position are calculated, wherein the first extrinsic parameter serves as the initial absolute pose configured for the target preset position, and the calibration parameters include the gimbal attitude parameters of the camera during calibration and the second extrinsic parameter. In the scene 3D model corresponding to the scene where the camera is located, the parameters of the viewport camera are set using the first intrinsic parameter and the first extrinsic parameter, and the depth data corresponding to the target preset position is acquired using the set viewport camera as the reference depth data for the target preset position.

[0029] In one example, the gimbal attitude parameters configured for the target preset position are used as the first gimbal attitude parameters, and the gimbal attitude parameters of the camera during calibration are used as the second gimbal attitude parameters. The first intrinsic parameters and the first extrinsic parameters configured for the camera at the target preset position are calculated using the second gimbal attitude parameters of the camera during calibration and the first gimbal attitude parameters configured for the target preset position.

[0030] In one embodiment, the gimbal attitude deviation between the first gimbal attitude parameters and the second gimbal attitude parameters is obtained. Using the horizontal rotation deviation and vertical rotation deviation in the gimbal attitude deviation, a second pose deviation of the camera between the target preset position and the calibration position is calculated. The camera's first intrinsic parameter is calculated using the lens scaling deviation, the second pose deviation, and the second extrinsic parameter in the gimbal attitude deviation; the camera's first extrinsic parameter is calculated using the second pose deviation and the second extrinsic parameter.

[0031] In one embodiment, before configuring the camera's initial preset position, the camera is calibrated to obtain its intrinsic and extrinsic parameters during calibration. These parameters are then used to configure the camera parameters corresponding to the initial preset position. In another embodiment, the camera is rotated to a viewing angle that meets the calibration requirements. A third acquired image obtained by the camera, along with the camera's current gimbal attitude parameters, are used as the camera's gimbal attitude parameters during calibration. A 3D scene model corresponding to the scene where the camera is located is loaded, and a current rendered image is generated that meets the overlap requirements with the third acquired image. The viewport camera is used to acquire the current rendered image corresponding to the current rendered image, and the depth data corresponding to the current rendered image is obtained as the current depth data. Using the matching result between the third acquired image and the current rendered image, as well as the current depth data, the second extrinsic parameter and the second intrinsic parameter of the camera during calibration are calculated.

[0032] S120: Calculate the current absolute pose of the camera using the matching result between the first acquired image and the reference image, as well as the reference depth data.

[0033] The matching result between the first acquired image and the reference image includes several matching point pairs between the first acquired image and the reference image. The matching point pair can be a two-dimensional pixel point of the first acquired image and a two-dimensional pixel point of the reference image, or it can be a three-dimensional spatial coordinate point corresponding to a two-dimensional pixel point of the first acquired image and a two-dimensional pixel point of the reference image. The three-dimensional spatial coordinate point of the reference image is located in the world coordinate system of the twin three-dimensional model corresponding to the current scene of the camera.

[0034] In one embodiment, the current extrinsic parameters of the camera are calculated using several matching point pairs, reference depth data, and camera intrinsic parameters, and are used as the current absolute pose. For example, pixel coordinates of matching points on the reference image side are extracted from the matching point pairs between the reference image and the first acquired image. These coordinates are then converted into three-dimensional spatial coordinates in the world coordinate system by combining them with the reference depth data. The two-dimensional pixel coordinates of the matching points on the first acquired image side, the three-dimensional spatial coordinates corresponding to the two-dimensional pixel coordinates of the reference image, and the pre-configured camera intrinsic parameters of the target preset position are input into the camera projection model. By minimizing the reprojection error, the current extrinsic parameters of the camera are calculated, including a rotation matrix R describing the rotation state and a translation vector t describing the translation state. The calculated current extrinsic parameters of the camera characterize the camera's position and attitude relative to the world coordinate system, and can be used as the current absolute pose.

[0035] In one embodiment, a matching result between a first acquired image and a reference image is obtained before calculating the current absolute pose of the camera. In another embodiment, the first acquired image and the reference image are matched to obtain a matching result. For example, image matching can be achieved by extracting local stable features to achieve feature-based matching; it can also be achieved by relying on local grayscale information comparison to achieve region-based matching; or it can be achieved by learning deep features through neural networks to achieve deep learning-based matching, etc.

[0036] In one embodiment, a first acquired image, a reference image, and reference depth data are used to match the first acquired image and the reference image to obtain a matching result. In one example, the depth information of the first acquired image can be obtained directly or through image depth estimation to determine the depth value set of the first acquired image and the reference image. Simultaneously, the three primary color information sets of the first acquired image and the reference image are obtained. Matching is then performed on the first acquired image and the reference image using the depth value set and the three primary color information set. For example, for any first pixel in the first acquired image and its corresponding second pixel in the reference image, a first matching probability is determined using the first target depth value of the first pixel and the second target depth value of the second pixel. A second matching probability is determined using the three primary color information of both. The two probabilities are weighted and summed to obtain the target matching probability. If the target matching probability is greater than or equal to a preset probability threshold, the pixel pair is determined to be a matching point pair. Finally, all matching point pairs that meet the conditions are counted, and erroneous matching points can be filtered using the RANSAC algorithm to obtain the final matching result, i.e., multiple valid matching points.

[0037] In one embodiment, before calculating the current absolute pose of the camera, it can be determined whether the target preset position has shifted. In response to the pixel deviation distribution of each matching point pair satisfying the offset condition, it is determined that the camera has shifted at the target preset position, and the current absolute pose of the camera and subsequent steps are performed using the matching result between the first acquired image and the reference image, as well as reference depth data. The offset condition is that the number of matching point pairs with pixel deviations greater than a deviation threshold exceeds a certain threshold; and / or, the pixel deviation of a matching point pair is the sum of the absolute values ​​of the differences in the coordinates of each axis of the matching point pair.

[0038] In one example, after acquiring matching point pairs between the first acquired image and the reference image, the sum of the absolute values ​​of the coordinate differences of each axis of the matching point pair is calculated to determine the pixel coordinate deviation of each pair of matching points. Different deviation thresholds can be set according to different pixel deviation distribution ranges of the matching point pairs. If the pixel deviation does not exceed the deviation threshold, the matching point pair is considered not to have shifted at a preset position; otherwise, it is determined that the matching point pair has shifted at the target preset position. Further, the number of matching point pairs with pixel deviations greater than the deviation threshold is counted. If the number exceeds a preset threshold, it is determined that the camera has shifted at the target preset position, thus allowing for further calculation of the camera's offset.

[0039] S130: Based on the current absolute pose, determine the first pose deviation of the camera at the target preset position.

[0040] The current absolute pose refers to the set of three-dimensional spatial position and attitude parameters of the camera relative to the world coordinate system in the preset twin 3D model under the current working state. Specifically, it is obtained by solving through PnP or other algorithms using image matching results, reference depth data and camera intrinsic parameters. It includes the rotation matrix (R) describing the rotation state and the translation vector (t) describing the translation state.

[0041] In one embodiment, a reference absolute pose of the camera at a preset target position is obtained, and the deviation between the current absolute pose and the reference absolute pose is calculated to obtain the first pose deviation. The reference absolute pose is either the initial absolute pose pre-configured for the preset target position, or the latest absolute pose updated for the preset target position after the last correction. For example, assume the current absolute pose is (R... new | t new The reference absolute pose is (R0|t0), and the first pose deviation (δR|δt) can be calculated by the following formula (1).

[0042] (1)

[0043] S140: Based on the first pose deviation, at least the target preset position of the camera is corrected.

[0044] To drive the camera gimbal to achieve preset position correction, a correspondence between the camera pose and the gimbal motion parameters is established to determine the adjustment amount of the gimbal pose.

[0045] In one embodiment, the first pose deviation is converted into a gimbal pose adjustment amount. For each preset position in at least one preset position, the gimbal position parameters corresponding to the preset position are adjusted using the gimbal pose adjustment amount. The at least one preset position includes a target preset position. The preset position deviations within a single camera are consistent; therefore, adjusting the deviation of one preset position will correct the deviations of all preset positions.

[0046] In one embodiment, the first pose deviation includes rotational deviation. The rotational deviation is decomposed into an n*m rotation matrix. The horizontal rotational adjustment in the gimbal pose adjustment is calculated using the first and second elements of the rotation matrix and trigonometric relationships. The vertical rotational adjustment in the gimbal pose adjustment is calculated using the third and fourth elements of the rotation matrix and trigonometric relationships. The first and second elements are the elements in the third row and first column of the rotation matrix, and the third row and third column of the rotation matrix, respectively. The third and fourth elements are the elements in the first row and second column of the rotation matrix, and the second row and second column of the rotation matrix, respectively. For example, assuming the calculated camera rotational deviation is δR, δR is decomposed into a 3*3 rotation matrix as shown in the matrix representation (2) below. Then, the horizontal rotational adjustment and vertical rotational adjustment in the gimbal pose adjustment are calculated using formula (3).

[0047] (2)

[0048] (3)

[0049] In one embodiment, the gimbal position parameters include horizontal rotation values, vertical rotation values, and lens scaling values. The gimbal pose adjustment amounts include horizontal rotation adjustment amounts and vertical rotation adjustment amounts. These horizontal and vertical rotation adjustment amounts are used to adjust the horizontal rotation values, vertical rotation values, and lens scaling values ​​corresponding to a preset position. In one example, the sum of the horizontal rotation value and the horizontal rotation adjustment amount in the gimbal position parameters before adjustment is used as the horizontal rotation value in the adjusted gimbal position parameters; the sum of the vertical rotation value and the vertical rotation adjustment amount in the gimbal position parameters before adjustment is used as the vertical rotation value in the adjusted gimbal position parameters; and the lens scaling value in the gimbal position parameters before adjustment is used as the lens scaling value in the adjusted gimbal position parameters.

[0050] After correcting the target preset position of the camera, the current state of the preset position can be verified. In one embodiment, a corrected preset position is used as a reference preset position. A second acquired image is acquired using the camera after correcting the reference preset position. A cross-correlation map of the second acquired image and the reference image of the reference preset position in the spatial domain is obtained. The distance between the pixel coordinates of the preset pixel value in the cross-correlation map and the position point of the reference image is obtained as the image offset distance. If the image offset distance is less than a distance threshold, the correction is determined to be valid. In one embodiment, the second acquired image can be used as the first image, and the reference image of the reference preset position can be used as the second image. In one embodiment, if the preset position is initially configured with the imaging pixel box information of the target object, the area where the target object is located in the second acquired image can also be used as the first image, and the area where the target object is located in the reference image of the reference preset position can be used as the second image.

[0051] In one example, the cross-correlation plot can be obtained through the following steps: convert the first image and the second image into grayscale images, perform Fourier transform on the converted first image and the second image respectively to obtain the spectrum corresponding to the first image and the second image respectively, calculate the conjugate product of the spectrum corresponding to the first image and the second image respectively, and perform inverse Fourier transform on the conjugate product to obtain the cross-correlation plot. Alternatively, the inverse Fourier transform can be performed after normalizing the conjugate product to obtain the cross-correlation plot.

[0052] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the camera preset position correction method provided in this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily reflect that outcome. Figure 2 The illustrated process sequence is limited. For example... Figure 2 As shown, this embodiment includes:

[0053] S201: Camera initialization configuration.

[0054] The initial configuration of the camera at least completes the initial calibration and initial preset position configuration of the gimbal camera. In this embodiment, the initial calibration and initial preset position configuration of each camera are completed using the twin scene corresponding to the scene where the current camera is located. The initial calibration process can determine the intrinsic and extrinsic parameters of the camera in the twin three-dimensional coordinate system. The initial preset position configuration can obtain the reference RGB (three primary colors) image, reference depth image (or reference depth information), and corresponding gimbal attitude parameters, i.e., PTZ value, of the camera at each preset position. Optionally, the pixel bounding box coordinates of the target object in the image at the preset position can also be configured.

[0055] In one embodiment, the camera is initially calibrated, the camera gimbal is rotated, a textured viewpoint is selected, an image is acquired, and the current gimbal parameters are recorded. Assuming the gimbal parameters are [P0, T0, Z0] and the captured image is Img0, then a twin 3D model of the scene is loaded in any 3D visualization platform. The current rendering screen is interactively adjusted so that the overlap between the rendering screen and the captured image Img0 reaches a preset percentage threshold, such as more than 50%. The parameters of the rendering image, the rendering depth map, and the viewport camera are saved. The resolution setting of the viewport camera must be consistent with the resolution of the captured image.

[0056] The captured image Img0 is matched with the rendered image to obtain several matching point pairs of the captured image Img0, the rendered image, and the corresponding depth information, which are recorded as [u0, v0, u, v, depth_value]. In the rendering engine, the three-dimensional coordinates [x0, y0, z0] of the model corresponding to the current pixel can be easily obtained by intersecting the direction vectors. That is, the matching point pair can also be described as [u0, v0, x0, y0, z0].

[0057] Based on the matching relationship between the acquired 2D camera pixels and 3D model coordinates, the camera intrinsic and extrinsic parameters corresponding to the captured image can be solved using existing open-source calibration libraries, such as OpenCV. The camera intrinsic parameters describe its own optical and digital characteristics, including focal lengths fx and fy, and image center points cx and cy. The camera extrinsic parameters describe the position and orientation of the camera in the 3D world coordinate system, including the rotation matrix R ∈ SO (3) (3×3) and the translation vector t (3×1). For example, the intrinsic parameter matrix K is shown in the matrix representation (4) below, and the extrinsic parameter matrix [R | t] can be determined based on the camera projection model (5) below, where [u, v] are pixel coordinates, [X, Y, Z] are 3D world coordinates, and s is the scale factor.

[0058] (4)

[0059] (5)

[0060] After initial camera calibration, several preset positions of the camera can be initially configured. The reference RGB image is a camera image in the preset position state that has been manually verified. This image is used as a reference for judging the subsequent preset position offset. For example, the camera is controlled to acquire and save images at each preset position.

[0061] In one embodiment, a reference depth image is obtained based on the camera's initial gimbal calibration parameters and a twin scene 3D model. (See [link to relevant documentation]). Figure 3 , Figure 3This is a schematic diagram of a method for acquiring a reference depth image provided in this application. Based on the deviation values ​​(δP, δT, δZ) between the gimbal pose parameters (P, T, Z) at the current preset position and the gimbal pose parameters (P0, T0, Z0) at the initial calibration, (δP, δT) are converted into a rotation matrix, and δZ is converted into the focal length change of the intrinsic parameter matrix.

[0062] (6) (7)

[0063] Based on the above formulas (6) and (7), perform depth simulation at the current preset position, load the current scene 3D model data in any 3D engine, and use the camera intrinsic and extrinsic parameters calculated by the formula at the preset position to set the parameters of the rendering viewport camera. The camera resolution must always remain consistent. After setting, the rendering depth buffer data, i.e. the reference depth image, can be obtained through the model.

[0064] S202: Monitoring of preset position offset.

[0065] In this embodiment, the monitoring of preset position offset can be set to judge each preset position during the business inspection process, or it can be set to judge once at a certain time interval. The specific judgment rules can be set according to the actual situation of the equipment and the site, and are not limited here.

[0066] In one embodiment, the latest captured image Img(new) at the current preset position is acquired, along with the reference image Img0 and the reference depth image depth_Img0 stored at the current preset position. Image matching is performed on the latest captured image Img(new) and the reference image Img0 to obtain N matching point pairs between Img(new) and Img0. The pixel coordinate deviation [δu, δv] of each pair of pixels is counted, and the absolute value of the pixel coordinate deviation (|δu|+|δv|) of each pair of pixels is calculated and counted. Simultaneously, the distribution range needs to be set according to device performance and historical experience. Considering the different stability of gimbals of different products and the varying degrees of stability of device installation in different scenarios, different deviation thresholds S can be set for different distribution ranges. If the absolute value of the pixel deviation does not exceed the deviation threshold S, the matching point pair is considered not to have shifted at the preset position. The percentage of matching point pairs not exceeding the deviation threshold is counted out of the total number of matching point pairs. If the percentage exceeds 70% (which can be adjusted according to actual conditions), no shift has occurred; otherwise, a preset position shift is considered to have occurred.

[0067] S203: Calculate the deviation of the preset position.

[0068] For cameras whose preset position offset is determined to have occurred, the extrinsic parameters (R) of the camera's current state are recalculated based on the current preset position reference RGB image, reference depth image, the latest captured image, and several matching point pairs calculated in step S202. new | t new For example, the latest extrinsic parameters of the camera can also be obtained by using OpenCV's nonlinear optimization of the minimum reprojection error in step S201, or by employing methods such as EPNP (Efficient Perspective-n-Point, an efficient improved PnP algorithm).

[0069] Based on the absolute pose of the camera in the initial preset position (assumed to be R0|t0) and the absolute pose in the current state after the preset position offset (R new | t new The camera pose deviation value, i.e. the preset position deviation value (δR | δt), is obtained by using formula (1) in step S130.

[0070] S204: Update of preset bit information.

[0071] Camera preset position deviation is mainly caused by factors such as mechanical wear, physical resistance, and return error generated during long-term operation of the camera gimbal. This directly manifests as a change in the actual shooting orientation of the gimbal even with the same PTZ parameters, resulting in a shift in the image content. Since the root cause of this type of deviation is the common influence of the gimbal's mechanical system, the deviation is mainly reflected in the PT values ​​(horizontal rotation and vertical pitch parameters). Furthermore, the deviation of all preset positions within a single camera is consistent; that is, if a preset position has a deviation, the other preset positions will exhibit a corresponding synchronous deviation.

[0072] After calculating the current preset position deviation value (δR | δt), the rotation matrix of the camera rotation deviation δR is determined as shown in the matrix representation (2) in step 140 above. The horizontal rotation adjustment amount δP and the vertical rotation adjustment amount δT of the preset position deviation are calculated using formula (3). Finally, based on δP and δT, the camera's preset positions can be updated by using P_new=P0+δP, T_new=T0+δT, and keeping the Z value unchanged.

[0073] S205: Verification of preset bits after correction.

[0074] If the imaging pixel frame information of the target object is saved during the initial preset position configuration, the effect of preset position correction can be verified only for the image area within the imaging pixel frame. If the current preset position is not saved, the effect of preset position correction can be verified directly for the entire area of ​​the acquired image.

[0075] In one embodiment, the verification step of the preset position after correction includes: (1) performing image preprocessing on the original reference image and the image recaptured after preset position adjustment, converting the RGB image to a single-channel grayscale image; (2) performing two-dimensional fast Fourier transform (FFT) on the two preprocessed images respectively; (3) calculating the conjugate product of the two spectra, and then normalizing the result to remove the amplitude influence; (4) performing inverse Fourier transform on the result to obtain the cross-correlation map in the spatial domain, wherein the distance between the pixel coordinates corresponding to the point with the largest peak and the center point of the image is the image offset distance; (5) if the image offset distance is less than the offset distance threshold, the preset position correction is considered effective. The phase correlation method provided in this embodiment can effectively suppress the influence of texture such as image brightness and darkness and color shift, and can focus on the image contour to improve the accuracy of displacement calculation.

[0076] Finally, for cameras that have been effectively corrected, the latest internal and external parameters of the camera after correction, as well as the configuration information of each preset position, are updated and stored as the initial values ​​for the next preset position judgment.

[0077] For a better explanation of the camera preset position correction method in this application, please refer to the following. Figure 4 , Figure 4 The following specific embodiments of the camera preset position correction method are provided as examples:

[0078] The gimbal camera captures images in real time at preset positions, acquiring the latest captured image for each preset position. This image is then combined with a reference RGB image and a reference depth image for that preset position to monitor its position and determine if any position offset has occurred. If no offset has occurred, monitoring continues. If an offset has occurred, the deviation is calculated based on the reference RGB and depth images. All camera presets are updated according to the calculated deviation. After the update, the presets are validated. If the validation passes, the updated camera parameters and preset configuration take effect, and monitoring continues.

[0079] Please see Figure 5 , Figure 5This is a schematic diagram of a camera preset position correction system provided in this application. The camera preset position correction system includes a preset position configuration module, a preset position monitoring module, a preset position deviation calculation module, and a preset position update verification module. The preset position configuration module is used to calibrate the initial parameters of several cameras in the current scene and configure the initial preset positions. In this embodiment, a twin scene of the current scene is preferably used for preset position configuration. The preset position monitoring module is used to obtain the matching result between the latest acquired image and the reference image, and to determine whether the current preset position has shifted. The preset position deviation calculation module is used to calculate the current absolute pose of the camera using the matching result between the latest acquired image and the reference image, as well as reference depth data, and to determine the pose deviation of the camera at the current preset position based on the current absolute pose. The preset position update verification module is used to update the camera's intrinsic and extrinsic parameters and the initial preset position configuration information, and to verify the updated camera preset position to determine whether the preset position correction is effective.

[0080] Please see Figure 6 , Figure 6 This is a schematic diagram of an embodiment of the electronic device provided in this application. The electronic device 60 includes a memory 61 and a processor 62 connected to each other. The memory 61 is used to store a computer program. When the computer program is executed by the processor 62, it is used to implement the camera preset position correction method in the above embodiment.

[0081] The methods described in the above embodiments can exist in the form of a computer program; therefore, this application proposes a computer-readable storage medium. Please refer to [link / reference needed]. Figure 7 , Figure 7 This is a schematic diagram of an embodiment of a computer-readable storage medium provided in this application. The computer-readable storage medium 80 is used to store a computer program 81, which can be executed to implement the camera preset position correction method in the above embodiment.

[0082] The computer-readable storage medium 80 can be any medium capable of storing program code, such as a server, USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0083] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for correcting a camera preset position, characterized in that, The method includes: The system acquires a first image of the camera corresponding to a target preset position, and acquires a reference image and reference depth data corresponding to the target preset position. The first image is acquired after the camera's gimbal rotates according to the gimbal attitude parameters corresponding to the target preset position. The camera has a first intrinsic parameter and a first extrinsic parameter corresponding to the target preset position during calibration. The reference depth data is obtained based on the following steps: in the scene 3D model corresponding to the scene where the camera is located, the parameters of the viewport camera are set using the first intrinsic parameter and the first extrinsic parameter, and the depth data corresponding to the target preset position is acquired using the set viewport camera as the reference depth data of the target preset position. Using the matching result between the first acquired image and the reference image, and the reference depth data, the current absolute pose of the camera is calculated; the matching result includes several matching point pairs between the first acquired image and the reference image; the matching point pairs include the three-dimensional spatial coordinate points corresponding to the two-dimensional pixels of the first acquired image and the two-dimensional pixels of the reference image, and the three-dimensional spatial coordinate points of the reference image are located in the world coordinate system of the scene three-dimensional model corresponding to the current scene of the camera; Based on the current absolute pose, determine the first pose deviation of the camera at the target preset position; Based on the first pose deviation, at least the target preset position of the camera is corrected.

2. The method according to claim 1, characterized in that, The step of determining the first pose deviation of the camera at the target preset position based on the current absolute pose includes: The reference absolute pose of the camera at the target preset position is obtained, wherein the reference absolute pose is the initial absolute pose pre-configured for the target preset position, or the latest absolute pose updated for the target preset position after the last correction. The deviation between the current absolute pose and the reference absolute pose is calculated to obtain the first pose deviation.

3. The method according to claim 1, characterized in that, The step of correcting at least the target preset position of the camera based on the first pose deviation includes: The first pose deviation is converted into gimbal pose adjustment amount; For each preset position in at least one preset position, the gimbal position parameters corresponding to the preset position are adjusted using the gimbal pose adjustment amount, wherein the at least one preset position includes the target preset position.

4. The method according to claim 3, characterized in that, The first pose deviation includes rotational deviation; the step of converting the first pose deviation into gimbal pose adjustment includes: Decompose the rotational deviation into n m-rotation matrix; Using the first and second elements of the rotation matrix and trigonometric relationships, the horizontal rotation adjustment amount in the gimbal pose adjustment is calculated, wherein the first and second elements are the elements in the third row and first column of the rotation matrix, and the elements in the third row and third column, respectively; and, Using the third and fourth elements of the rotation matrix and trigonometric relationships, the vertical rotation adjustment amount in the gimbal pose adjustment amount is calculated, wherein the third and fourth elements are the elements in the first row and second column of the rotation matrix, and the elements in the second row and second column, respectively. And / or, the gimbal position parameters include horizontal rotation values, vertical rotation values, and lens zoom values, and the gimbal pose adjustment amounts include horizontal rotation adjustment amounts and vertical rotation adjustment amounts; adjusting the gimbal position parameters corresponding to the preset position using the gimbal pose adjustment amounts includes: The sum of the horizontal rotation value and the horizontal rotation adjustment amount in the gimbal position parameters before adjustment is used as the horizontal rotation value in the gimbal position parameters after adjustment. The sum of the vertical rotation value and the vertical rotation adjustment amount in the gimbal position parameters before adjustment is used as the vertical rotation value in the gimbal position parameters after adjustment. The lens scaling value in the gimbal position parameters before adjustment is used as the lens scaling value in the gimbal position parameters after adjustment.

5. The method according to claim 1, characterized in that, The matching result includes several pairs of matching points between the first acquired image and the reference image; calculating the current absolute pose of the camera using the matching result between the first acquired image and the reference image, and the reference depth data, includes: Using the several matching point pairs, the reference depth data, and the camera's intrinsic parameters, the current extrinsic parameters of the camera are calculated and used as the current absolute pose; And / or, before calculating the current absolute pose of the camera using the matching result between the first acquired image and the reference image, and the reference depth data, the method further includes: The matching result is obtained by matching the first acquired image and the reference image using the first acquired image and the reference image; or, the matching result is obtained by matching the first acquired image and the reference image using the first acquired image, the reference image, and the reference depth data.

6. The method according to claim 1, characterized in that, Before calculating the current absolute pose of the camera using the matching result between the first acquired image and the reference image, and the reference depth data, the method further includes: In response to the pixel deviation distribution of each of the matching point pairs satisfying the offset condition, it is determined that the camera has shifted at the target preset position, and the current absolute pose of the camera and subsequent steps are performed using the matching result between the first acquired image and the reference image, as well as the reference depth data. Wherein, the offset condition is that the number of matching point pairs with pixel deviation greater than the deviation threshold exceeds the number threshold; and / or, the pixel deviation of the matching point pair is the sum of the absolute values ​​of the differences in the coordinates of each axis of the matching point pair.

7. The method according to claim 1, characterized in that, After correcting at least the target preset position of the camera based on the first pose deviation, the method further includes: Using a corrected preset position as a reference preset position, a second image is acquired using a camera that has corrected the reference preset position. Obtain the cross-correlation map in the spatial domain between the second acquired image and the reference image at the reference preset position; The distance between the pixel coordinates of the preset pixel value in the cross-correlation graph and the position point in the reference image is obtained as the image offset distance; If the image offset distance is less than a distance threshold, the correction is deemed valid.

8. The method according to claim 7, characterized in that, The step of obtaining the cross-correlation map in the spatial domain between the second acquired image and the reference image at the reference preset position includes: The second acquired image can be used as the first image, and the reference image of the reference preset position can be used as the second image; or, the area where the target object is located in the second acquired image can be used as the first image, and the area where the target object is located in the reference image of the reference preset position can be used as the second image. Convert the first image and the second image into grayscale images; Perform Fourier transforms on the transformed first image and second image respectively to obtain the spectra corresponding to the first image and the second image respectively; Calculate the conjugate product of the spectra corresponding to the first image and the second image, respectively; The cross-correlation graph is obtained by performing an inverse Fourier transform on the conjugate product, or by performing an inverse Fourier transform on the conjugate product after normalization.

9. The method according to claim 1, characterized in that, The reference image corresponding to the target preset position and the first intrinsic parameter and the first extrinsic parameter of the target preset position are configured through the following steps: After determining that the camera is located at the target preset position, the image captured by the camera is obtained as the reference image corresponding to the target preset position, and the current gimbal attitude parameters of the gimbal are obtained as the gimbal attitude parameters configured for the target preset position. Using the camera's calibration parameters and the gimbal attitude parameters configured for the target preset position, the first intrinsic parameter and the first extrinsic parameter of the camera at the target preset position are calculated. The first extrinsic parameter serves as the initial absolute pose configured for the target preset position. The calibration parameters include the gimbal attitude parameters of the camera during calibration and the second extrinsic parameter.

10. The method according to claim 9, characterized in that, The calculation of the first intrinsic and first extrinsic parameters of the camera at the target preset position using the camera's calibration parameters and the gimbal attitude parameters configured for the target preset position includes: The gimbal attitude parameters configured for the target preset position are used as the first gimbal attitude parameters, and the gimbal attitude parameters of the camera during calibration are used as the second gimbal attitude parameters. The gimbal attitude deviation between the first gimbal attitude parameters and the second gimbal attitude parameters is obtained. Using the horizontal rotation deviation and vertical rotation deviation in the gimbal attitude deviation, the second pose deviation of the camera between the target preset position and the calibration position is calculated; Using the lens scaling deviation, the second pose deviation, and the second extrinsic parameter from the gimbal attitude deviation, the first intrinsic parameter of the camera is calculated; and, The first extrinsic parameter of the camera is calculated using the second pose deviation and the second extrinsic parameter. And / or, the camera calibration steps include: Control the camera to rotate to a viewing angle that meets the calibration requirements, and use the third acquired image obtained by the camera, as well as the current gimbal attitude parameters of the camera, as the gimbal attitude parameters of the camera during calibration. Load the 3D scene model corresponding to the scene where the camera is located, and generate the current rendered image whose visual overlap with the third acquired image meets the overlap requirements; The viewport camera is used to acquire the current rendered image corresponding to the current rendered screen, and the depth data corresponding to the current rendered screen is acquired as the current depth data; Using the matching result between the third acquired image and the currently rendered image, as well as the current depth data, the second extrinsic parameter and the second intrinsic parameter of the camera during calibration are calculated.

11. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor being coupled to the memory, the processor being configured to execute one or more steps of the camera preset position correction method according to any one of claims 1 to 10 based on instructions stored in the memory.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the steps of the camera preset position correction method as described in any one of claims 1 to 10.

Images