Method, device and electronic equipment for calibrating zoom lens

Abstract

The present disclosure relates to a calibration method, device, and electronic device for a zoom lens, wherein the method may include: obtaining images captured by the zoom lens at multiple focal lengths and tracking data corresponding to each image captured; the tracking data includes position and rotation information; obtaining a preset internal parameter model, wherein the internal parameter model represents the numerical value of the internal parameter corresponding to different focal lengths of the zoom lens; and optimizing the values ​​of the parameters in the internal parameter model based on the images captured at the multiple focal lengths and the tracking data corresponding to each image captured. In the present disclosure, the zoom lens installed on the image acquisition device in the virtual shooting scene is calibrated by optimizing the values ​​of the parameters in the internal parameter model; and in the process of optimizing the values ​​of the parameters in the internal parameter model, the corresponding tracking data for each image captured is used to assist, thereby increasing constraints and greatly improving the accuracy and stability of the optimization.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a calibration method, device, and electronic equipment for a zoom lens. Background Art

[0002] When filming TV series / performances, the traditional method of constructing real sets is extremely labor-intensive, time-consuming, and financially intensive. Furthermore, the sets cannot be reused; they must be completely dismantled after filming a scene and rebuilt for subsequent shoots. Furthermore, for science fiction, fantasy, and period dramas, the set design itself is extremely challenging. Replacing traditional real-life set construction with virtual filming technology significantly reduces set construction costs. Only a single set of digital assets is required, which can be reused across similar scenes. Adjustments can also be made quickly and in real time during filming to meet on-set filming requirements, saving time and effort. During actual virtual filming, a scene image rendered by a virtual engine is projected onto a screen. Actors then perform using the screen as a backdrop. An image capture device simultaneously captures both the actors and the screen. The captured images are then combined with the original scene image, placing the real actors within the virtual set, creating the illusion of filming exterior scenes or sci-fi backdrops in a studio.

[0003] Before performing virtual shooting, it is usually necessary to calibrate the lens of the image acquisition device. Existing calibration technologies are aimed at fixed-focus lenses, but lack calibration for zoom lenses. Summary of the Invention

[0004] In view of this, the present disclosure proposes a calibration method, apparatus, electronic device, storage medium, and computer program product for a zoom lens.

[0005] According to one aspect of the present disclosure, a method for calibrating a zoom lens is provided for calibrating a zoom lens installed on an image acquisition device in a virtual shooting scene, the method comprising: obtaining images captured by the zoom lens at multiple focal lengths and tracking data corresponding to each image captured; the tracking data comprising position and rotation information; obtaining a preset intrinsic parameter model, the intrinsic parameter model representing values ​​of intrinsic parameters corresponding to different focal lengths of the zoom lens; and optimizing parameter values ​​in the intrinsic parameter model based on the images captured at the multiple focal lengths and the tracking data corresponding to each image captured.

[0006] In a possible implementation, the internal parameter model includes: a polynomial curve corresponding to each internal parameter.

[0007] In one possible implementation, the image acquisition device is equipped with a tracker for measuring the tracking data; the optimization of the parameter values ​​in the intrinsic parameter model based on the images acquired at the multiple focal lengths and the tracking data corresponding to each image acquired includes: obtaining the external parameters corresponding to each image based on the tracking data corresponding to each image acquired and a preset offset transformation matrix; wherein the offset transformation matrix represents the deviation between the tracking data corresponding to when the zoom lens acquires the image and the external parameters when the zoom lens acquires the image; determining the projection transformation matrix corresponding to each image based on the external parameters corresponding to each image and the intrinsic parameter model; and optimizing the parameter values ​​in the intrinsic parameter model based on each image corresponding to each image and the projection transformation matrix.

[0008] In one possible implementation, each image includes feature points; the method also includes: determining the two-dimensional coordinates of the feature points in each image and the three-dimensional coordinates of the feature points on the screen; optimizing the values ​​of the parameters in the intrinsic parameter model based on each image and the projection transformation matrix corresponding to each image, including: based on a back projection error algorithm, using the projection transformation matrix corresponding to each image, the two-dimensional coordinates of the feature points in each image and the three-dimensional coordinates of the feature points on the screen, to optimize the values ​​of the parameters in the intrinsic parameter model.

[0009] In one possible implementation, the back-projection error algorithm utilizes the projection transformation matrix corresponding to each image, the two-dimensional coordinates of the feature points in each image, and the three-dimensional coordinates of the feature points on the screen to optimize the values ​​of the parameters in the internal parameter model, including: determining the two-dimensional reference coordinates corresponding to the three-dimensional coordinates of the feature points in each image on the screen according to the projection transformation matrix corresponding to each image; and calculating the back-projection error value based on the two-dimensional reference coordinates and the two-dimensional coordinates of the feature points; and iteratively optimizing the values ​​of the parameters in the internal parameter model based on the back-projection error value until the back-projection error value is less than a preset threshold.

[0010] In one possible implementation, optimizing the values ​​of the parameters in the intrinsic parameter model based on each image and the projection transformation matrix corresponding to each image includes optimizing the values ​​of the parameters in the intrinsic parameter model and the values ​​of the parameters in the offset transformation matrix based on each image and the projection transformation matrix corresponding to each image.

[0011] In a possible implementation, the method further includes: calibrating the internal parameters corresponding to the main focal length among the multiple focal lengths to obtain the calibration values ​​of the internal parameters corresponding to the main focal length; determining the initial values ​​of the parameters in the internal parameter model based on the calibration values ​​of the internal parameters corresponding to the main focal length; optimizing the values ​​of the parameters in the internal parameter model based on the images captured at the multiple focal lengths and the tracking data corresponding to when each image is captured, including: optimizing the values ​​of the parameters in the internal parameter model based on the initial values ​​of the parameters in the internal parameter model based on the images captured at the multiple focal lengths and the tracking data corresponding when each image is captured.

[0012] In a possible implementation, the method further includes: normalizing the identification value of the focal length of the zoom lens, wherein the normalized value of the identification value corresponding to the main focal length is 0.

[0013] In a possible implementation, the images captured at the multiple focal lengths include images obtained by shooting the image on the screen once at each of the multiple focal lengths.

[0014] According to another aspect of the present disclosure, a zoom lens calibration device is provided for calibrating a zoom lens installed on an image acquisition device in a virtual shooting scene. The device includes:

[0015] an acquisition module, configured to acquire images captured by the zoom lens at multiple focal lengths and tracking data corresponding to each image captured; the tracking data including position and rotation information;

[0016] The acquisition module is further configured to acquire a preset internal parameter model, wherein the internal parameter model represents the values ​​of the internal parameters corresponding to different focal lengths of the zoom lens;

[0017] The optimization module is used to optimize the values ​​of the parameters in the internal reference model based on the images collected at the multiple focal lengths and the tracking data corresponding to each image collected.

[0018] In a possible implementation, the internal parameter model includes: a polynomial curve corresponding to each internal parameter.

[0019] In one possible implementation, the image acquisition device is equipped with a tracker for measuring the tracking data; the optimization module is further used to: obtain the external parameters corresponding to each image based on the tracking data corresponding to the acquisition of each image and a preset offset transformation matrix; wherein the offset transformation matrix represents the deviation between the tracking data corresponding to the image acquired by the zoom lens and the external parameters when the image is acquired by the zoom lens; determine the projection transformation matrix corresponding to each image based on the external parameters corresponding to each image and the internal parameter model; and optimize the values ​​of the parameters in the internal parameter model based on each image and the projection transformation matrix corresponding to each image.

[0020] In one possible implementation, each of the images includes feature points; the optimization module is further used to: determine the two-dimensional coordinates of the feature points in each image and the three-dimensional coordinates of the feature points on the screen; based on the back projection error algorithm, use the projection transformation matrix corresponding to each image, the two-dimensional coordinates of the feature points in each image and the three-dimensional coordinates of the feature points on the screen to optimize the values ​​of the parameters in the internal parameter model.

[0021] In one possible implementation, the optimization module is further used to: determine the two-dimensional reference coordinates corresponding to the three-dimensional coordinates of the feature points in each image on the screen based on the projection transformation matrix corresponding to each image; and calculate the back projection error value based on the two-dimensional reference coordinates and the two-dimensional coordinates of the feature points; and iteratively optimize the values ​​of the parameters in the internal parameter model based on the back projection error value until the back projection error value is less than a preset threshold.

[0022] In a possible implementation, the optimization module is further used to optimize the values ​​of the parameters in the intrinsic parameter model and the values ​​of the parameters in the offset transformation matrix based on each image and the projection transformation matrix corresponding to each image.

[0023] In one possible implementation, the optimization module is further used to: calibrate the internal parameters corresponding to the main focal length among the multiple focal lengths to obtain the calibration value of the internal parameter corresponding to the main focal length; determine the initial value of the parameter in the internal parameter model based on the calibration value of the internal parameter corresponding to the main focal length; and optimize the value of the parameter in the internal parameter model based on the initial value of the parameter in the internal parameter model according to the images captured at the multiple focal lengths and the tracking data corresponding to the capture of each image.

[0024] In a possible implementation, the optimization module is further configured to: perform normalization processing on the identification value of the focal length of the zoom lens, wherein the normalized value of the identification value corresponding to the main focal length is 0.

[0025] In a possible implementation, the images captured at the multiple focal lengths include images obtained by shooting the image on the screen once at each of the multiple focal lengths.

[0026] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0027] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0028] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0029] In the disclosed embodiment, a zoom lens installed on an image acquisition device in a virtual shooting scene is calibrated by optimizing the values ​​of the parameters in the internal reference model. Furthermore, considering that each image can be acquired at a different position, the corresponding external parameters and internal parameters are different. In the process of optimizing the values ​​of the parameters in the internal reference model, the corresponding tracking data when each image is acquired is used as an aid, thereby adding constraints and greatly improving the accuracy and stability of the optimization. This allows for more accurate values ​​of the parameters in the internal reference model to be optimized, and there is no need to fix the position of the image acquisition device when acquiring images, greatly improving flexibility. As an example, the internal reference model can be a polynomial curve, so that the internal reference calibration of the zoom lens can be achieved through curve fitting. Compared with linear interpolation and other methods, the internal reference of each focal length can be directly calculated through the polynomial curve, which is more accurate and more efficient. As another example, only one image needs to be acquired at each focal length to complete the calibration, which greatly reduces the amount of data collected and improves the efficiency of zoom lens calibration.

[0030] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0032] Figure 1 A schematic diagram of a virtual shooting scene according to an embodiment of the present disclosure is shown.

[0033] Figure 2 A flowchart of a zoom lens calibration method according to an embodiment of the present disclosure is shown.

[0034] Figure 3 (a)-(b) are schematic diagrams showing collected images according to an embodiment of the present disclosure.

[0035] Figure 4 A flow chart of a method for optimizing the values ​​of parameters in an internal reference model according to an embodiment of the present disclosure is shown.

[0036] Figure 5 A flowchart of a zoom lens calibration method according to an embodiment of the present disclosure is shown.

[0037] Figure 6 The figure shows a structural diagram of a calibration device for a zoom lens according to an embodiment of the present disclosure.

[0038] Figure 7 A schematic structural diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0039] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0040] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present disclosure. Thus, phrases such as "exemplary," "in one embodiment," "in some other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0041] In the present disclosure, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0042] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0043] The following first provides an illustrative description of the applicable application scenarios of the embodiments of the present disclosure.

[0044] Figure 1 A schematic diagram of a virtual shooting scene according to an embodiment of the present disclosure is shown. Figure 1 As shown, the virtual shooting scene may include a display device 10 and an image acquisition device 20; wherein the display device 10 is used to display the rendered scene, and the image acquisition device 20 is used to capture the picture displayed by the display device 10. For example, an actor can stand in an appropriate position in front of the display device 10 and perform with the display device 10 as the background. The image acquisition device 20 can simultaneously capture the actor and the picture displayed by the display device 10, thereby completing the shooting of the virtual scene.

[0045] Exemplarily, the display device 10 may be a LED (Light-Emitting Diode) screen or a screen made of other materials. The shape of the screen may be a flat screen, a curved screen, etc. The display device 10 may also be a projection screen, etc., which is not limited in the embodiments of the present disclosure.

[0046] Exemplarily, the image acquisition device 20 may be a device with a shooting function such as a camera or a video camera. The image acquisition device 20 is equipped with a zoom lens, that is, the focal length of the lens of the image acquisition device 20 can be changed to meet different shooting requirements.

[0047] For a zoom lens, when the focal length changes, its imaging parameters usually change. Therefore, during the process of virtual shooting using the image acquisition device 20, it is crucial to correctly track the changes in the focal length of the zoom lens and the corresponding changes in the imaging parameters. In order to accurately track the imaging parameters of the zoom lens of the image acquisition device 20 at different focal lengths during the virtual shooting process, it is necessary to calibrate the zoom lens of the image acquisition device 20 before using the image acquisition device 20 for virtual shooting, that is, to solve the imaging parameters of the zoom lens of the image acquisition device 20 corresponding to different focal lengths. For example, the imaging parameters of the zoom lens may include intrinsic parameters and / or extrinsic parameters, wherein the extrinsic parameters may include pose, that is, the position of the zoom lens in space and the orientation of the zoom lens, which can be represented by a rotation matrix R and a translation matrix T; the intrinsic parameters may include the focal length, center offset, field of view (FOV), principal point position, distortion parameters, etc. of the zoom lens.

[0048] For example, a method for calibrating a zoom lens is as follows: multiple focal lengths are selected within the zoom range of the zoom lens, and the fixed-focus lens is calibrated for each focal length to obtain the internal parameters of the lens at each focal length, forming an internal parameter table; in actual use, the internal parameter table is searched according to the real-time focal length information, and the real-time lens internal parameters are obtained by interpolation. However, this method requires collecting a large number of pictures (generally about 10 pictures) for each focal length that needs to be calibrated, and the calibration time is long. Especially when the zoom range of the zoom lens is large, the amount of data collected is even greater, which is very time-consuming and the calibration efficiency is low. Another method for calibrating a zoom lens is as follows: by collecting images at different focal lengths at a fixed position, and then calibrating the internal parameters of each focal length by interpolation. However, the acquisition position of the image acquisition device in this method is fixed, which is not flexible enough, and the internal parameters determined by interpolation are not accurate enough.

[0049] In order to solve the above technical problems, the present disclosure proposes a calibration method for a zoom lens (detailed description see below), which can be used to calibrate the above Figure 1The zoom lens mounted on the image acquisition device 20 in the virtual shooting scene is calibrated. Calibration of the zoom lens mounted on the image acquisition device in the virtual shooting scene is achieved by optimizing the values ​​of the parameters in the intrinsic reference model. Furthermore, considering that each image can be captured at a different position, corresponding to different external and internal parameters, the optimization of the parameters in the intrinsic reference model is assisted by the tracking data corresponding to each captured image, adding constraints and significantly improving the accuracy and stability of the optimization. This allows for more accurate optimization of the parameters in the intrinsic reference model, and eliminates the need to fix the position of the image acquisition device during image capture, significantly increasing flexibility. As an example, the intrinsic reference model can be a polynomial curve, enabling intrinsic calibration of the zoom lens through curve fitting. Compared to methods such as linear interpolation, the intrinsic parameters for each focal length can be directly calculated using this polynomial curve, resulting in more accurate and efficient calculations. As another example, calibration can be completed by capturing only one image at each focal length, significantly reducing the amount of data collected and improving the efficiency of zoom lens calibration.

[0050] It should be noted that the above-mentioned application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. A person skilled in the art will appreciate that the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems in response to the emergence of other similar or new scenarios, such as virtual studio scenarios.

[0051] The calibration method provided by the embodiment of the present disclosure is described in detail below.

[0052] Figure 2 A flowchart of a zoom lens calibration method according to an embodiment of the present disclosure is shown. The method can be executed by an electronic device with data processing capabilities, such as a processor or a server, and is used to calibrate a zoom lens installed on an image acquisition device in a virtual shooting scene, such as Figure 2 As shown, the method may include the following steps:

[0053] Step 201: Acquire images captured by the zoom lens at multiple focal lengths and corresponding tracking data when each image is captured; the tracking data includes position and rotation information.

[0054] The focal length can be obtained using a set of encoders fixed to the zoom lens. The zoom value of the encoder indicates the current position within the entire zoom range, i.e., the current focal length. The number of focal lengths included in the multiple focal lengths and the intervals between adjacent focal lengths can be set according to actual needs. For example, since the zoom range of the zoom lens has more than one focal length, multiple focal lengths can be randomly selected from the zoom range of the zoom lens. For example, multiple focal lengths can be selected at equal intervals within the zoom range of the zoom lens so that the calibrated focal lengths are evenly distributed within the zoom range. For example, if the zoom range of the zoom lens is 10mm-110mm, 10 focal lengths can be selected at intervals of 10mm, i.e., the multiple focal lengths are 10mm, 20mm, 30mm, 40mm, 50mm, 60mm, 70mm, 80mm, 90mm, and 100mm.

[0055] For example, the images captured at multiple focal lengths may be images captured by capturing the on-screen image in the virtual shooting scene at multiple focal lengths; wherein the on-screen image refers to the image displayed on the screen, and may also be referred to as the upper screen image; as an example, the image capture device may be the above-mentioned Figure 1 The image acquisition device 20 may have a screen as described above. Figure 1 The display device 10 is shown in FIG.

[0056] For example, the position of the zoom lens when capturing images at different focal lengths can be different. This means that the position of the image capture device does not need to be fixed, and the image capture device can be moved during the image capture process, thereby reducing restrictions on the capture position and greatly improving flexibility. For example, during image capture by the image capture device, the focal length of the zoom lens can be adjusted, sequentially switching to each of the multiple focal lengths selected above, and capturing the image on the screen at each focal length, thereby obtaining images captured at each focal length. The position of the image capture device when capturing the image on the screen at different focal lengths can be the same or different. For example, the field of view captured at each focal length can cover the image on the screen.

[0057] For example, the images captured by the zoom lens at multiple focal lengths include images obtained by capturing the image on the screen once at each of the multiple focal lengths. That is, for each focal length, only one image of the image on the screen captured by the zoom lens at that focal length needs to be captured. This saves image capture time and reduces the amount of data required for processing, thereby reducing calibration time and effort and improving zoom lens calibration efficiency.

[0058] As an example, each image captured by the zoom lens at different focal lengths includes feature points, and the number of feature points can be one or more. A feature point represents a point in the captured image where the grayscale value changes dramatically or a point on the edge of the captured image where the curvature is large. This point can reflect the essential characteristics of the image and can identify information such as objects or locations in the captured image. In the disclosed embodiment, the image on the screen can include multiple preset feature points. In this way, the images captured by the zoom lens when capturing the image on the screen at each focal length contain multiple feature points. In addition, the image on the screen also includes a preset positioning identifier with known position information; in this way, the images captured by the zoom lens when capturing the image on the screen at each focal length also include the positioning identifier and the multiple feature points. Exemplarily, the positioning identifier can be an identifier generated based on an Aruco code (a type of QR code), or a combination of the two. The Aruco code is a synthetic square mark consisting of a wide black border and an internal binary matrix that can determine its identifier. In the embodiment of the present disclosure, a positioning identifier can be generated based on the Aruco code. For example, the Aruco code can be used as a positioning identifier alone; other identification information can be added to the Aruco code as a positioning identifier; the Aruco code can also be modified in a certain shape and the modified graphic can be used as a positioning identifier. For example, Figure 3 (a)-(b) are schematic diagrams showing a collected image according to an embodiment of the present disclosure, wherein the collected image includes a combination of a positioning identifier and feature points, wherein: Figure 3 (a) is a dot matrix diagram, which consists of white dots. Each white dot is a feature point, and the four circles in the middle are positioning identifiers. Figure 3 (b) is a picture combining the Aruco code with a checkerboard. The checkerboard consists of black and white squares. The point where two adjacent squares of the same color meet is called a corner point, which is a feature point. A single Aruco code is embedded in a white square on the checkerboard. The white square with the embedded ArUco code is the positioning identifier.

[0059] In one possible implementation, the method further includes determining the two-dimensional coordinates of feature points in each image captured by the zoom lens at multiple focal lengths and the three-dimensional coordinates of the feature points on the screen. Exemplarily, feature point detection is performed on each image captured by the zoom lens at each of the multiple focal lengths to determine the two-dimensional coordinates of the detected feature points in the captured image. Exemplarily, feature point detection can be performed using a variety of different algorithms, such as a corner detection algorithm or a spot detection algorithm, to determine multiple feature points in the captured image at each of the multiple focal lengths. For example, for a feature point on an image captured at each focal length, the two-dimensional coordinates of the feature point can be represented by the coordinate values ​​of the feature point in the image coordinate system. The image coordinate system is a two-dimensional rectangular coordinate system, and the origin can be the center of the captured image, a point in the lower left corner, or a point in the lower right corner of the captured image. The X-axis and Y-axis of the image coordinate system are parallel to the X-axis and Y-axis of the image acquisition device coordinate system, respectively. The image acquisition device coordinate system can have the focal center of the image acquisition device as its origin, and the optical axis of the image acquisition device as its Z-axis. For example, the intersection of the optical axis of the image acquisition device and the plane containing the captured image can be used as the origin of the image coordinate system. Thus, based on the established image coordinate system, after detecting each feature point in the captured image, the coordinate values ​​of each feature point in the image coordinate system (i.e., the two-dimensional coordinates in the captured image) can be determined. Furthermore, a location identifier can be identified, and the three-dimensional coordinates of each detected feature point on the screen can be determined based on the identified location identifier. For example, for a certain feature point on an image captured at each focal length, the three-dimensional coordinates of the feature point on the screen can be represented by the coordinate value of the feature point in the screen coordinate system on the screen captured by the zoom lens; wherein the screen coordinate system is a three-dimensional coordinate system set when the screen is modeled, and the definition of the screen coordinate system can be set according to the actual situation, for example: the center of the screen can be used as the coordinate origin, or a point in the lower left corner of the screen, or a point in the lower right corner of the screen can be used as the coordinate origin, etc. For example, based on the multiple feature points and positioning identifiers detected in each image captured at different focal lengths, the relative position relationship between each feature point and the positioning identifier can be obtained, and then based on the relative position relationship and the position information of the positioning identifier, the position of each feature point in the screen can be determined; then, based on the position of each feature point in the screen and the two-dimensional display area of ​​the screen in the established screen model, the UV coordinates of each feature point in the screen model are determined; finally, based on the correspondence between the three-dimensional coordinates and the UV coordinates in the screen coordinate system, the three-dimensional coordinate value of each feature point in the screen coordinate system, that is, the three-dimensional coordinates of each feature point on the screen, is obtained. For example, for the above Figure 3For an image with an Aruco code and a checkerboard grid, as shown in (b), checkerboard corner detection can be performed to obtain the corner points in the image, that is, to detect feature points, and then Aruco code recognition can be performed. After detecting the corner points of the checkerboard grid and any Aruco code, any detected Aruco code is decoded and identified to obtain identification information for any Aruco code. Based on this identification information, the position of any Aruco code on the screen is determined. Then, based on the relative positional relationship between each corner point and any Aruco code, the position of each corner point on the screen is obtained, and then the UV coordinates of each corner point in the screen model are determined. Finally, based on the correspondence between the three-dimensional coordinates in the screen coordinate system and the UV coordinates, the three-dimensional coordinate values ​​of each corner point in the screen coordinate system, that is, the three-dimensional coordinates of each corner point on the screen, are obtained. In this way, for any detected feature point, the two-dimensional coordinates of the feature point in the captured image and the three-dimensional coordinates of the feature point on the screen are determined, thereby constructing a 2D-3D point pair.

[0060] Exemplarily, the image acquisition device is equipped with a tracker for measuring tracking data. The tracker is an instrument capable of determining its own position and posture information. Specifically, it can acquire its own position and rotation information within a preset coordinate system in real time. This position and rotation information constitutes the tracking data. The origin of the preset coordinate system can be any point in space, and the coordinate axis directions of the preset coordinate system can also be customized based on actual circumstances. For example, the location of the image acquisition device base in the virtual shooting location can be determined as the origin of the preset coordinate system. The direction parallel to the screen width in the horizontal plane is set as the x-axis, the direction perpendicular to the x-axis in the horizontal plane is set as the y-axis, and the vertically upward direction is set as the z-axis, where the x-axis, y-axis, and z-axis satisfy the right-hand rule. Alternatively, the center of the screen can be determined as the origin of the preset coordinate system. The screen width is set as the x-axis, the screen length is set as the y-axis, and the z-axis direction is determined using the right-hand rule based on the set x-axis and y-axis. In this way, while the zoom lens is capturing images at multiple focal lengths, the tracker can measure its own position and rotation information (i.e., tracking data) in real time, i.e., the position and rotation information of the zoom lens when capturing images at each focal length.

[0061] Step 202: Obtain a preset internal parameter model, where the internal parameter model represents the values ​​of the internal parameters corresponding to different focal lengths of the zoom lens.

[0062] The specific type of the intrinsic parameter model can be pre-set based on needs and is not limited to this. For example, existing data on the focal lengths of other calibrated zoom lenses and the corresponding internal parameter values ​​can be obtained, and then multiple existing models can be used to verify this existing data. The optimal model from these multiple models is selected as the preset internal parameter model. For example, the internal parameter model can be a polynomial curve. As an example, the internal parameter model includes: a polynomial curve corresponding to each internal parameter; for any internal parameter a, the polynomial curve corresponding to the internal parameter a represents the relationship between the value of the internal parameter a and the focal length. That is, given a focal length A, the value of the internal parameter a at that focal length A can be obtained using the polynomial curve corresponding to the internal parameter a. The polynomial curve is a curve composed of multiple polynomial functions. In this way, the internal parameter calibration of the zoom lens is achieved through curve fitting. During actual virtual shooting, the internal parameters of each focal length can be directly calculated using this polynomial curve, which is more accurate and efficient than methods such as linear interpolation. As an example, the internal parameters may include: fx, fy, cx, cy, k1, k2, k3, p1, p2 and other parameters, where fx and fy are the focal length in the x-direction and the focal length in the y-direction of the image acquisition device coordinate system, respectively, cx and cy are the x-axis coordinates and the y-axis coordinates of the principal point, respectively; k1, k2, k3 are radial distortion parameters, and p1 and p2 are tangential distortion parameters; the internal parameter model may include a polynomial curve corresponding to each parameter in fx, fy, cx, cy, k1, k2, k3, p1, p2.

[0063] In a possible implementation, the identification value of the focal length of the zoom lens can also be normalized to make the result more accurate. Taking into account that in actual applications, the value ranges of the zoom values ​​of different zoom lenses may be different, therefore, in order to improve the universality of the calibration algorithm in the embodiment of the present disclosure, the zoom values ​​of all zoom lenses can be uniformly normalized; in this way, the focal length is mapped to a certain range for processing, making subsequent data processing more convenient and faster. Exemplarily, the internal parameter model represents the value of the internal parameter corresponding to the normalized value of the identification value of different focal lengths. In this way, for a certain zoom value of the zoom lens (i.e., the identification value of the focal length), normalization can be performed, and then the normalized value is substituted into the internal parameter model, for example, by substituting it into a polynomial curve, the internal parameter corresponding to the zoom value can be obtained.

[0064] Step 203: Optimize the values ​​of the parameters in the intrinsic reference model according to the images collected at the multiple focal lengths and the tracking data corresponding to each image collected.

[0065] It is understandable that by optimizing the values ​​of the parameters in the internal parameter model, an optimized internal parameter model can be obtained, thereby completing the calibration of the internal parameters of the zoom lens; in the subsequent actual virtual shooting process, the internal parameter model can be used to obtain the internal parameters corresponding to different focal lengths. As an example, the internal parameter model includes: a polynomial curve corresponding to each internal parameter, and the coefficient values ​​of the polynomial curve corresponding to each internal parameter can be optimized based on images captured at multiple focal lengths and the tracking data corresponding to each image when it is captured, thereby completing the calibration of the internal parameters of the zoom lens based on curve fitting; and then in the subsequent actual virtual shooting process, the polynomial curve can be used to accurately calculate the internal parameter corresponding to any focal length.

[0066] In an embodiment of the present disclosure, images captured by the zoom lens at multiple focal lengths and tracking data corresponding to each image captured are obtained; the tracking data includes position and rotation information; a preset internal parameter model is obtained, and the internal parameter model represents the values ​​of the internal parameters corresponding to different focal lengths of the zoom lens; the values ​​of the parameters in the internal parameter model are optimized according to the images captured at the multiple focal lengths and the tracking data corresponding to each image captured; in this way, the zoom lens installed on the image acquisition device in the virtual shooting scene is calibrated by optimizing the values ​​of the parameters in the internal parameter model; and, considering that each image can be captured at a different position, the corresponding external parameters are different, and the internal parameters are also different, in the process of optimizing the values ​​of the parameters in the internal parameter model, the corresponding tracking data when each image is captured is used to assist, thereby increasing constraints and greatly improving the accuracy and stability of the optimization, so that more accurate values ​​of the parameters in the internal parameter model can be optimized, and there is no need to fix the position of the image acquisition device when capturing images, which greatly improves flexibility. As an example, the intrinsic parameter model can be a polynomial curve, enabling calibration of the zoom lens' intrinsic parameters through curve fitting. Compared to methods like linear interpolation, this polynomial curve can be used to directly calculate the intrinsic parameters for each focal length, which is more accurate and efficient. As another example, calibration can be completed by capturing only a single image at each focal length, significantly reducing the amount of data collected and improving the efficiency of zoom lens calibration.

[0067] The following is a detailed description of possible implementation methods for optimizing the values ​​of the parameters in the internal reference model in step 203:

[0068] Figure 4 A flow chart of a method for optimizing the values ​​of parameters in an internal reference model according to an embodiment of the present disclosure is shown. Figure 4 As shown, the method may include the following steps:

[0069] Step 20301: Obtain the extrinsic parameters corresponding to each image based on the tracking data corresponding to each image acquired and a preset offset transformation matrix.

[0070] The offset transformation matrix represents the deviation between the tracking data corresponding to the zoom lens when capturing images and the extrinsic parameters of the zoom lens when capturing images. For example, the offset transformation matrix may include the deviation component between the rotation information corresponding to each image captured by the tracked zoom lens and the rotation matrix used to capture each image, and may also include the deviation component between the position information corresponding to each image captured by the tracked zoom lens and the translation matrix used to capture each image.

[0071] It can be understood that extrinsic parameters are parameters that describe the position and posture of the image acquisition device in the world coordinate system. The extrinsic parameters may change at different positions or shooting times. For example, moving the image acquisition device or changing the shooting angle will cause the extrinsic parameters to change. Since each image can be captured at a different position, that is, the corresponding extrinsic parameters are different, and considering that there is still a certain deviation between the position and rotation information provided by the tracking data and the actual extrinsic parameters of the image acquisition device, that is, there is an offset transformation, the tracking data corresponding to each image is corrected by the offset transformation matrix to obtain the accurate extrinsic parameters corresponding to each image. As an example, the tracking data corresponding to the image captured by other existing zoom lenses and the extrinsic parameter data when the zoom lens captures the image can be obtained. By analyzing the tracking data and the extrinsic parameter data, the values ​​of the parameters in the preset offset transformation matrix are determined. As another example, the initial value of the preset offset transformation matrix can be set to 0, that is, the initial deviation value of the tracking data corresponding to the zoom lens when capturing images and the external parameters when the zoom lens captures images is 0. In the process of optimizing the values ​​of the parameters in the internal parameter model described below, the values ​​of the parameters in the preset offset transformation matrix are simultaneously optimized, so that the deviation value between the tracking data corresponding to the zoom lens when capturing images represented by the offset transformation matrix and the external parameters when the zoom lens captures images gradually approaches the actual deviation value.

[0072] Step 20302: Determine the projection transformation matrix corresponding to each image based on the external parameters corresponding to each image and the internal parameter model.

[0073] Exemplarily, the projection transformation matrix represents the transformation relationship between the image coordinate system and the screen coordinate system.

[0074] For example, the focal length of the zoom lens when capturing each image can be substituted into the intrinsic parameter model to obtain the intrinsic parameter of the zoom lens when capturing each image, that is, the intrinsic parameter corresponding to each image. For example, the normalized value of the focal length of the zoom lens when capturing each image can be substituted into a polynomial curve to obtain the intrinsic parameter of the zoom lens when capturing each image. Then, based on the extrinsic parameters corresponding to each image and the extrinsic parameters corresponding to each image, the projection transformation matrix corresponding to each image can be obtained.

[0075] Step 20303: Optimize the values ​​of the parameters in the intrinsic reference model according to each image and the projection transformation matrix corresponding to each image.

[0076] In one possible implementation, this step may include optimizing the values ​​of the parameters in the intrinsic reference model and the values ​​of the parameters in the offset transformation matrix based on each image and the projection transformation matrix corresponding to each image. Considering that the preset offset transformation matrix may differ from the actual offset transformation matrix (i.e., the deviation between the actual tracking data and the extrinsic parameters), the values ​​of the parameters in the intrinsic reference model and the values ​​of the parameters in the offset transformation matrix are simultaneously optimized to further improve the accuracy of the values ​​of the parameters in the optimized reference model. In this way, the values ​​of the parameters in the intrinsic reference model and the values ​​of the parameters in the offset transformation matrix can be optimized simultaneously. Since the offset transformation matrix represents the deviation between the tracking data corresponding to the zoom lens when capturing the image and the extrinsic parameters when the zoom lens captures the image, the extrinsic parameters of the zoom lens may change at any time during the subsequent virtual shooting process through the optimized offset transformation matrix. Based on the real-time tracking data and the optimized offset transformation matrix, the accurate real-time extrinsic parameters of the zoom lens can be accurately obtained.

[0077] In one possible implementation, this step may include optimizing the values ​​of the parameters in the intrinsic reference model based on a back-projection error algorithm, utilizing the projection transformation matrix corresponding to each image, the two-dimensional coordinates of the feature points in each image, and the three-dimensional coordinates of the feature points on the screen. Exemplarily, the values ​​of the parameters in the intrinsic reference model and the values ​​of the parameters in the offset transformation matrix may also be optimized simultaneously. In this way, the back-projection error algorithm simplifies the process of estimating the parameters in the intrinsic reference model of the zoom lens, making it easier and faster to converge to accurate values ​​for the parameters in the intrinsic reference model.

[0078] For example, the two-dimensional reference coordinates corresponding to the three-dimensional coordinates of the feature points in each image on the screen can be determined based on the projection transformation matrix corresponding to each image; a back-projection error value can be calculated based on the two-dimensional reference coordinates and the two-dimensional coordinates of the feature points; and the values ​​of the parameters in the intrinsic parameter model can be iteratively optimized based on the back-projection error value until the back-projection error value is less than a preset threshold. The specific value of the preset threshold can be set as needed and is not limited thereto. Specifically, in each iteration, for any image, the three-dimensional coordinates of all feature points in the image detected on the screen can be transformed according to the projection transformation matrix corresponding to the image to obtain the two-dimensional reference coordinates corresponding to the three-dimensional coordinates of all feature points on the screen, and then the back projection error between the two-dimensional reference coordinates of all feature points and the two-dimensional coordinates of all feature points is calculated, which is the back projection error corresponding to the image. After traversing all the collected images, the back projection errors corresponding to all images are summarized, which is the back projection error value of this iteration; for example, for any feature point P in the image, the three-dimensional coordinate P_i of the feature point in the screen coordinate system can be projected to the two-dimensional reference coordinate P_t in the image coordinate system according to the projection transformation matrix corresponding to the image; then the distance between the two-dimensional reference coordinate P_t and the two-dimensional coordinate P'_t of the feature point P in the image coordinate system is calculated, which is the back projection error corresponding to the feature point P. The back projection errors corresponding to all feature points in the image are summarized, which is the back projection error corresponding to the image. Then all images are traversed to obtain the back projection error value of this iteration. Then, determine whether the back projection error of this iteration is less than a preset threshold. If it is less than the preset threshold, use the values ​​of the parameters in the internal reference model in this iteration as the values ​​of the parameters in the final calibrated internal reference model. Otherwise, update the values ​​of the parameters in the internal reference model, and synchronously update the values ​​of the parameters in the internal reference model and the values ​​of the parameters in the offset transformation matrix. Repeat the above calculation of the external parameters corresponding to each image and subsequent steps until the back projection error of a certain iteration is less than the preset threshold, thereby obtaining the values ​​of the parameters in the final calibrated internal reference model.

[0079] In this way, through the above steps 20301-20303, for any captured image, its corresponding tracking data is transformed by a preset offset transformation matrix to obtain the external parameters corresponding to the image; according to the focal length when the image is captured and the internal parameter model, the internal parameters corresponding to the image are determined, and then the projection transformation matrix corresponding to the image is obtained by combining the external parameters corresponding to the image obtained above; and then the back projection error corresponding to the image is calculated according to the image and the projection transformation matrix corresponding to the image; in this way, the above processing is performed on each captured image, and finally the back projection error corresponding to each image is summarized, and the values ​​of the parameters in the internal parameter model are iteratively optimized. The values ​​of the parameters in the internal parameter model and the offset transformation matrix can also be optimized synchronously. The optimization process is the process of continuously reducing the back projection error value calculated in each iteration, thereby obtaining the values ​​of the parameters in the optimized internal parameter model, and realizing the internal parameter calibration of the zoom lens.

[0080] Figure 5 A flow chart of a zoom lens calibration method according to an embodiment of the present disclosure is shown. Figure 5 As shown, the method may include the following steps:

[0081] Step 501: Acquire images captured by the zoom lens at multiple focal lengths and corresponding tracking data when each image is captured; the tracking data includes position and rotation information.

[0082] This step 501 is similar to the above Figure 2 Step 201 is the same as above and will not be described again here.

[0083] Step 502: Obtain a preset internal parameter model, where the internal parameter model represents the values ​​of the internal parameters corresponding to different focal lengths of the zoom lens.

[0084] This step 502 is similar to the above Figure 2 Step 202 is the same as above and will not be described again here.

[0085] Step 503: calibrate the internal parameter corresponding to the main focal length among the multiple focal lengths to obtain a calibration value of the internal parameter corresponding to the main focal length.

[0086] Among them, the main focal length can be arbitrarily selected from the multiple focal lengths; as an example, the main focal length can be the minimum focal length among the multiple focal lengths, that is, the wide-angle end of the zoom lens of the image acquisition device can be used as the main focal length.

[0087] In one possible implementation, calibrating the internal parameter corresponding to the primary focal length may include calibrating the internal parameter corresponding to the primary focal length using a fixed-focus calibration method. For example, the internal parameter corresponding to the primary focal length may be calibrated using an existing fixed-focus lens calibration method. For example, the internal parameter corresponding to the primary focal length may be obtained by searching the corresponding factory manual based on the specific model of the zoom lens. For another example, the internal parameter corresponding to the primary focal length may be obtained by calling a calibration interface of an image acquisition device (such as an OpenCV calibration interface) based on the two-dimensional coordinates of a feature point in an image captured at the primary focal length and the three-dimensional coordinates of the feature point on the screen.

[0088] Step 504: Determine the initial values ​​of the parameters in the intrinsic parameter model according to the calibrated values ​​of the intrinsic parameters corresponding to the main focal length.

[0089] Exemplarily, the internal parameter model includes a polynomial curve corresponding to each internal parameter, and the calibration value of the internal parameter corresponding to the main focal length can be configured as the constant term coefficient in the polynomial curve corresponding to each internal parameter.

[0090] For example, the normalized value of the identification value corresponding to the main focal length is 0. Thus, when the internal parameter model includes a polynomial curve corresponding to each internal parameter, the calibration value of the internal parameter corresponding to the main focal length can be configured as the constant term coefficient in the polynomial curve corresponding to each internal parameter, while the initialization values ​​of the other term coefficients are all 0.

[0091] Step 505: Optimize the values ​​of the parameters in the intrinsic reference model according to the images collected at the multiple focal lengths and the tracking data corresponding to each image collected.

[0092] The possible implementation of step 505 can refer to the above Figure 2 The relevant statement in step 203.

[0093] In a possible implementation, the values ​​of the parameters in the intrinsic reference model can be optimized based on the initial values ​​of the parameters in the intrinsic reference model and according to the images captured at the multiple focal lengths and the tracking data corresponding to each image captured.

[0094] In the disclosed embodiment, the internal parameters corresponding to the main focal length are first calibrated, and then the parameters in the internal parameter model are initialized using the internal parameters corresponding to the main focal length. Then, based on the images captured at multiple focal lengths of the zoom lens and the tracking data corresponding to each image captured, the parameters in the internal parameter model are optimized to the optimal values ​​more quickly, thereby achieving rapid and accurate calibration of the internal parameters of the zoom lens.

[0095] Based on the same inventive concept of the above method embodiment, the embodiment of the present disclosure further provides a calibration device for a zoom lens, which can be used to implement the technical solution described in the above method embodiment. For example, the above Figure 2 、 Figure 4 or Figure 5 The steps of the zoom lens calibration method are shown in FIG.

[0096] Figure 6 FIG1 shows a structural diagram of a zoom lens calibration device according to an embodiment of the present disclosure, wherein the device is used to calibrate a zoom lens installed on an image acquisition device in a virtual shooting scene. Figure 6 As shown, the device may include:

[0097] The acquisition module 601 is used to obtain images captured by the zoom lens at multiple focal lengths and the tracking data corresponding to each image captured; the tracking data includes position and rotation information; the acquisition module 601 is also used to obtain a preset internal parameter model, and the internal parameter model represents the value of the internal parameter corresponding to different focal lengths of the zoom lens; the optimization module 602 is used to optimize the value of the parameters in the internal parameter model based on the images captured at the multiple focal lengths and the tracking data corresponding to each image captured.

[0098] In the disclosed embodiment, the zoom lens installed on the image acquisition device in the virtual shooting scene is calibrated by optimizing the values ​​of the parameters in the internal reference model; and, considering that each image can be acquired at a different position, the corresponding external parameters are different, and the internal parameters are also different, in the process of optimizing the values ​​of the parameters in the internal reference model, the corresponding tracking data of each acquired image is used for assistance, which increases the constraints and greatly improves the accuracy and stability of the optimization, so that more accurate values ​​of the parameters in the internal reference model can be optimized, and there is no need to fix the position of the image acquisition device when acquiring images, which greatly improves the flexibility.

[0099] In a possible implementation, the internal parameter model includes: a polynomial curve corresponding to each internal parameter.

[0100] In one possible implementation, the image acquisition device is equipped with a tracker for measuring the tracking data; the optimization module 602 is further used to: obtain the external parameters corresponding to each image based on the tracking data corresponding to the acquisition of each image and a preset offset transformation matrix; wherein the offset transformation matrix represents the deviation between the tracking data corresponding to the image acquired by the zoom lens and the external parameters when the image is acquired by the zoom lens; determine the projection transformation matrix corresponding to each image based on the external parameters corresponding to each image and the internal parameter model; and optimize the values ​​of the parameters in the internal parameter model based on each image and the projection transformation matrix corresponding to each image.

[0101] In one possible implementation, each of the images includes feature points; the optimization module 602 is further used to: determine the two-dimensional coordinates of the feature points in each image and the three-dimensional coordinates of the feature points on the screen; based on the back projection error algorithm, use the projection transformation matrix corresponding to each image, the two-dimensional coordinates of the feature points in each image and the three-dimensional coordinates of the feature points on the screen to optimize the values ​​of the parameters in the internal parameter model.

[0102] In one possible implementation, the optimization module 602 is further used to: determine the two-dimensional reference coordinates corresponding to the three-dimensional coordinates of the feature points in each image on the screen based on the projection transformation matrix corresponding to each image; and calculate the back projection error value based on the two-dimensional reference coordinates and the two-dimensional coordinates of the feature points; and iteratively optimize the values ​​of the parameters in the internal reference model based on the back projection error value until the back projection error value is less than a preset threshold.

[0103] In a possible implementation, the optimization module 602 is further used to optimize the values ​​of the parameters in the intrinsic parameter model and the values ​​of the parameters in the offset transformation matrix according to each image and the projection transformation matrix corresponding to each image.

[0104] In one possible implementation, the optimization module 602 is further used to: calibrate the internal parameters corresponding to the main focal length among the multiple focal lengths to obtain the calibration values ​​of the internal parameters corresponding to the main focal length; determine the initial values ​​of the parameters in the internal parameter model based on the calibration values ​​of the internal parameters corresponding to the main focal length; and optimize the values ​​of the parameters in the internal parameter model based on the initial values ​​of the parameters in the internal parameter model according to the images captured at the multiple focal lengths and the tracking data corresponding to the capture of each image.

[0105] In a possible implementation, the optimization module 602 is further configured to: perform normalization processing on the identification value of the focal length of the zoom lens, wherein the normalized value of the identification value corresponding to the main focal length is 0.

[0106] In a possible implementation, the images captured at the multiple focal lengths include images obtained by shooting the image on the screen once at each of the multiple focal lengths.

[0107] above Figure 6 The technical effects and specific descriptions of the calibration device shown and its various possible implementation methods can be found in the above-mentioned calibration method, which will not be repeated here.

[0108] It should be understood that the division of the modules in the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or they can be physically separated. In addition, the modules in the device can be implemented in the form of a processor calling software; for example, the device includes a processor, the processor is connected to a memory, and the memory stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the modules of the device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory within the device or a memory outside the device. Alternatively, the modules in the device can be implemented in the form of hardware circuits, and the functions of some or all modules can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), which realizes the functions of some or all of the above modules by designing the logical relationship of the components in the circuit. For another example, in another implementation, the hardware circuit can be implemented by a programmable logic device (PLD). Taking a field programmable gate array (FPGA) as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, thereby realizing the functions of some or all of the above modules. All modules of the above devices can be implemented in the form of software called by the processor, or in the form of hardware circuits, or in part by the form of software called by the processor, and the rest by hardware circuits.

[0109] In the embodiments of the present disclosure, a processor is a circuit capable of processing signals. In one implementation, the processor may be a circuit capable of reading and executing instructions, such as a CPU, a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a neural-network processing unit (NPU), a tensor processing unit (TPU), etc. In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules.

[0110] It can be seen that each module in the above apparatus can be one or more processors (or processing circuits) configured to implement the above embodiment methods, such as: CPU, GPU, NPU, TPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms. In addition, each module in the above apparatus can be fully or partially integrated together, or can be implemented independently, without limitation.

[0111] The present disclosure also provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the method of the above embodiment when executing the instructions. Figure 2 、 Figure 4 or Figure 5 The steps of the zoom lens calibration method are shown in FIG.

[0112] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Figure 7 As shown, the electronic device may include: at least one processor 801 , a communication line 802 , a memory 803 and at least one communication interface 804 .

[0113] The processor 801 can be a general-purpose central processing unit, a microprocessor, a specific application integrated circuit, or one or more integrated circuits for controlling the execution of the program of the disclosed solution; the processor 801 can also include a heterogeneous computing architecture of multiple general-purpose processors, for example, it can be a combination of at least two of a CPU, a GPU, a microprocessor, a DSP, an ASIC, and an FPGA; as an example, the processor 801 can be a CPU+GPU or a CPU+ASIC or a CPU+FPGA.

[0114] Communication link 802 may include a pathway for transmitting information between the aforementioned components.

[0115] The communication interface 804 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc.

[0116] The memory 803 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory can be independent and connected to the processor via a communication line 802. The memory can also be integrated with the processor. The memory provided in the embodiment of the present disclosure can generally have non-volatility. Among them, the memory 803 is used to store computer-executable instructions for executing the disclosed solution, and is controlled by the processor 801 for execution. The processor 801 is used to execute the computer-executable instructions stored in the memory 803, thereby implementing the method provided in the above embodiment of the present disclosure; illustratively, the above Figure 2 、 Figure 4 or Figure 5 The steps of the zoom lens calibration method are shown in FIG.

[0117] Optionally, the computer-executable instructions in the embodiments of the present disclosure may also be referred to as application code, which is not specifically limited in the embodiments of the present disclosure.

[0118] Exemplarily, the processor 801 may include one or more CPUs, for example, Figure 7 The processor 801 may also include a CPU, and any one of a GPU, an ASIC, and an FPGA, for example, Figure 7 CPU0+GPU0 or CPU 0+ASIC0 or CPU0+FPGA0 in.

[0119] For example, an electronic device may include multiple processors, such as Figure 7 801 and processor 807 in FIG. Each of these processors can be a single-core (single-CPU) processor, a multi-core (multi-CPU) processor, or a heterogeneous computing architecture including multiple general-purpose processors. A processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0120] In a specific implementation, as an embodiment, the electronic device may further include an output device 805 and an input device 806. The output device 805 communicates with the processor 801 and can display information in a variety of ways. For example, the output device 805 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. For example, it can be a display device such as a vehicle-mounted HUD, AR-HUD, or a display. The input device 806 communicates with the processor 801 and can receive user input in a variety of ways. For example, the input device 806 can be a mouse, a keyboard, a touch screen device, or a sensing device, etc.

[0121] The embodiments of the present disclosure provide a computer-readable storage medium having computer program instructions stored thereon, which implement the method in the above embodiments when the computer program instructions are executed by a processor. Figure 2 、 Figure 4 or Figure 5 The steps of the zoom lens calibration method are shown in FIG.

[0122] The embodiments of the present disclosure provide a computer program product, which may include computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code; when the computer program product is run on a computer, the computer is caused to execute the method in the above embodiment. Figure 2 、 Figure 4 or Figure 5 The steps of the zoom lens calibration method are shown in FIG.

[0123] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0124] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0125] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0126] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0127] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0128] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0129] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0130] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0131] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A calibration method for a zoom lens, characterized in that: The method is used to calibrate a zoom lens installed on an image acquisition device in a virtual shooting scene, and includes: Acquire images captured by the zoom lens at multiple focal lengths and tracking data corresponding to each image captured; the tracking data includes position and rotation information; Obtaining a preset internal parameter model, where the internal parameter model represents values ​​of internal parameters corresponding to different focal lengths of the zoom lens; The values ​​of the parameters in the internal reference model are optimized according to the images collected at the multiple focal lengths and the tracking data corresponding to each image collected.

2. The method according to claim 1, characterized in that The internal parameter model includes: a polynomial curve corresponding to each internal parameter.

3. The method according to claim 1, characterized in that The image acquisition device is equipped with a tracker for measuring the tracking data; Optimizing the values ​​of the parameters in the intrinsic reference model according to the images captured at the multiple focal lengths and the tracking data corresponding to each image captured, includes: Obtaining an extrinsic parameter corresponding to each image based on the tracking data corresponding to each image captured and a preset offset transformation matrix; wherein the offset transformation matrix represents a deviation between the tracking data corresponding to the image captured by the zoom lens and the extrinsic parameter when the zoom lens captured the image; Determining a projection transformation matrix corresponding to each image according to the external parameters corresponding to each image and the internal parameter model; According to each image and the projection transformation matrix corresponding to each image, the values ​​of the parameters in the intrinsic parameter model are optimized.

4. The method according to claim 3, characterized in that Each of the images includes feature points; The method further comprises: Determining the two-dimensional coordinates of the feature points in each image and the three-dimensional coordinates of the feature points on the screen; Optimizing the values ​​of the parameters in the intrinsic parameter model according to each image and the projection transformation matrix corresponding to each image includes: Based on the back-projection error algorithm, the projection transformation matrix corresponding to each image, the two-dimensional coordinates of the feature points in each image and the three-dimensional coordinates of the feature points on the screen are used to optimize the values ​​of the parameters in the intrinsic parameter model.

5. The method according to claim 4, characterized in that The back-projection error algorithm is based on the projection transformation matrix corresponding to each image, the two-dimensional coordinates of the feature points in each image, and the three-dimensional coordinates of the feature points on the screen to optimize the values ​​of the parameters in the intrinsic parameter model, including: According to the projection transformation matrix corresponding to each image, the two-dimensional reference coordinates corresponding to the three-dimensional coordinates of the feature points in each image on the screen are determined; and based on the two-dimensional reference coordinates and the two-dimensional coordinates of the feature points, the back projection error value is calculated; based on the back projection error value, the values ​​of the parameters in the internal reference model are iteratively optimized until the back projection error value is less than a preset threshold.

6. The method according to claim 3, characterized in that Optimizing the values ​​of the parameters in the intrinsic parameter model according to each image and the projection transformation matrix corresponding to each image includes: According to each image and the projection transformation matrix corresponding to each image, the values ​​of the parameters in the intrinsic parameter model and the values ​​of the parameters in the offset transformation matrix are optimized.

7. The method according to claim 1, characterized in that The method further comprises: Calibrate an internal parameter corresponding to a main focal length among the multiple focal lengths to obtain a calibration value of the internal parameter corresponding to the main focal length; Determining initial values ​​of parameters in the intrinsic parameter model according to the calibrated values ​​of the intrinsic parameters corresponding to the main focal length; Optimizing the values ​​of the parameters in the intrinsic reference model according to the images captured at the multiple focal lengths and the tracking data corresponding to each image captured, includes: On the basis of the initial values ​​of the parameters in the intrinsic reference model, the values ​​of the parameters in the intrinsic reference model are optimized according to the images captured at the multiple focal lengths and the tracking data corresponding to each image captured.

8. The method according to claim 7, characterized in that The method further comprises: Normalization is performed on the identification value of the focal length of the zoom lens, wherein the normalized value of the identification value corresponding to the main focal length is 0.

9. The method according to claim 1, characterized in that The images captured at the multiple focal lengths include images obtained by shooting the screen image once at each of the multiple focal lengths.

10. A calibration device for a zoom lens, characterized in that: Used to calibrate a zoom lens installed on an image acquisition device in a virtual shooting scene, the device comprising: an acquisition module, configured to acquire images captured by the zoom lens at multiple focal lengths and tracking data corresponding to each image captured; the tracking data including position and rotation information; The acquisition module is further configured to acquire a preset internal parameter model, wherein the internal parameter model represents the values ​​of the internal parameters corresponding to different focal lengths of the zoom lens; The optimization module is used to optimize the values ​​of the parameters in the internal reference model according to the images collected at the multiple focal lengths and the tracking data corresponding to each image collected.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 9 when executing the instructions stored in the memory.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images