Image processing methods, apparatus, storage media and computer equipment

Abstract

The present invention discloses an image processing method, device, storage medium and computer equipment. The method comprises: acquiring a preset position image of a camera at a preset position and camera parameters at the preset position; acquiring a current position image of the camera at the current position; and determining the camera parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position. The present invention solves the technical problem of low camera calibration efficiency in the related art.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to an image processing method, device, storage medium and computer equipment. Background Art

[0002] In the process of processing the images acquired by the camera, there are certain differences between the two-dimensional attributes in the two-dimensional image and the three-dimensional attributes in the real three-dimensional space. How to project the two-dimensional semantic information in the existing two-dimensional image, such as people and objects, into the real three-dimensional space involves a camera calibration processing problem. Through camera calibration, the conversion relationship between the two-dimensional coordinates of the image and the three-dimensional coordinates of the space can be realized, thereby obtaining the three-dimensional attributes of the real three-dimensional space.

[0003] In the related art, when calibrating a camera, the method used is to use a three-dimensional calibration object with known real size, establish a correspondence between points on the three-dimensional calibration object with known coordinates and two-dimensional image points, and use a predetermined algorithm to solve the camera parameters of the camera. However, when using the above method, the processing and maintenance of the real three-dimensional calibration object is relatively difficult, so the camera calibration efficiency is low.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide an image processing method, an apparatus, a storage medium and a computer device to at least solve the technical problem of low camera calibration efficiency in the related art.

[0006] According to one aspect of an embodiment of the present invention, there is provided an image processing method, comprising: acquiring a preset position image of a camera at a preset position, and camera parameters at the preset position; acquiring a current position image of the camera at the current position; and determining the camera parameters of the camera at the current position based on the preset position image, the current position image and the camera parameters at the preset position.

[0007] Optionally, the camera parameters of the camera at the current position are determined according to the preset position image, the current position image and the camera parameters at the preset position, including: determining the initial parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; and optimizing the initial parameters according to the pre-position image and the current position image to obtain the camera parameters of the camera at the current position.

[0008] Optionally, determining the initial parameters of the camera at the current position based on the preset position image, the current position image and the camera parameters at the preset position includes: in the case where there are multiple preset position images, selecting a target preset position image from the multiple preset position images, wherein the target pre-position image is a preset position image corresponding to a target preset position closest to the current position; acquiring a scaling factor of the target pre-position image relative to the current position image; and determining the initial parameters of the camera at the current position based on the scaling factor and the camera parameters of the camera at the target pre-position.

[0009] Optionally, selecting the target preset position image from the multiple preset position images includes: determining scaling factors of the multiple preset position images relative to the current position image; correcting the scaling factors corresponding to the multiple preset position images to obtain multiple corrected scaling factors; and determining the target preset position image based on the multiple corrected scaling factors.

[0010] Optionally, determining the scaling factors of the multiple preset-position images relative to the current-position image includes: performing feature matching on the multiple preset-position images with the current-position image respectively to obtain a first feature point set in the current-position image and a second feature point set of the multiple preset-position images respectively; decentering the first feature point set to obtain a decentralized first feature point set, and decentering the second feature point set of the multiple preset-position images respectively to obtain a decentralized second feature point set; determining the homography matrices between the feature points of the current-position image and the multiple preset-position images respectively based on the decentralized first feature point set and the decentralized second feature point sets corresponding to the multiple preset-position images respectively; determining the scaling factors of the multiple preset-position images relative to the current-position image respectively based on the homography matrices between the feature points of the current-position image and the multiple preset-position images respectively.

[0011] Optionally, the scaling factors corresponding to the multiple preset position images are corrected to obtain multiple corrected scaling factors, including: when the scaling factor is less than 1, the scaling factor is used as the corrected scaling factor; when the scaling factor is greater than 1, the inverse of the scaling factor is used as the corrected scaling factor; the target preset position image is determined according to the multiple corrected scaling factors, including: determining the maximum value among the multiple corrected scaling factors; and determining the preset position image corresponding to the maximum value as the target preset position image.

[0012] Optionally, the initial parameters are optimized according to the preset position image and the current position image to obtain the camera parameters of the camera at the current position, including: back-projecting the second feature point in the preset position image into the camera coordinate system to obtain the first coordinate of the second feature point in the preset position image in the camera coordinate system; forward projecting the first coordinate onto the image plane of the current position to obtain the third feature point on the image plane of the current position; and optimizing the initial parameters by minimizing the reprojection error of the third feature point relative to the first feature point in the current position image to obtain the camera parameters of the camera at the current position.

[0013] Optionally, optimizing the initial parameters includes: when the initial parameters include: camera intrinsic parameters, distortion parameters, and camera extrinsic parameters, keeping the camera extrinsic parameters unchanged, and optimizing the camera intrinsic parameters and the distortion parameters.

[0014] Optionally, there are two preset positions, one is the maximum focal length of the camera, and the other is the minimum focal length of the camera.

[0015] Optionally, the camera comprises a variable focal length camera.

[0016] According to another aspect of an embodiment of the present invention, there is also provided an image processing method, comprising: displaying a current position image of a camera at a current position on a display interface; receiving an input operation, wherein the input operation is used to request display of a three-dimensional picture corresponding to the current position image; in response to the input operation, displaying a three-dimensional picture corresponding to the current position image on the display interface, wherein the three-dimensional picture is determined based on a correspondence between coordinates in the current position image and three-dimensional space coordinates, the correspondence is determined based on camera parameters of the camera at the current position, the camera parameters of the camera at the current position are determined based on a preset position image of the camera at a preset position, the camera parameters at the preset position, and the current position image.

[0017] Optionally, the method further includes: identifying a target object in the three-dimensional image, and highlighting the identified target object on the display interface.

[0018] According to one aspect of an embodiment of the present invention, there is also provided an image processing device, comprising: a first acquisition module, used to acquire a preset position image of a camera at a preset position, and camera parameters at the preset position; a first acquisition module, used to acquire a current position image of the camera at the current position; and a first determination module, used to determine the camera parameters of the camera at the current position based on the preset position image, the current position image and the camera parameters at the preset position.

[0019] According to another aspect of an embodiment of the present invention, there is further provided an image processing device, comprising: a first display module, configured to display a current position image of a camera at a current position on a display interface; a first receiving module, configured to receive an input operation, wherein the input operation is configured to request display of a three-dimensional picture corresponding to the current position image; and a second display module, configured to respond to the input operation and display a three-dimensional picture corresponding to the current position image on the display interface, wherein the three-dimensional picture is determined based on a correspondence between coordinates in the current position image and three-dimensional space coordinates, the correspondence being determined based on camera parameters of the camera at the current position, the camera parameters of the camera at the current position being determined based on a preset position image of the camera at a preset position, the camera parameters at the preset position, and the current position image.

[0020] According to one aspect of an embodiment of the present invention, a storage medium is further provided, the storage medium comprising a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute any one of the above-mentioned image processing methods.

[0021] According to another aspect of an embodiment of the present invention, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program; and the processor is used to execute the computer program stored in the memory, wherein when the computer program is executed, the processor executes any one of the above-mentioned image processing methods.

[0022] According to one aspect of an embodiment of the present invention, there is provided an image processing method, comprising: receiving a current position image of a camera at a current position sent by a client device; and feeding back camera parameters of the camera at the current position to the client device, wherein the camera parameters of the camera at the current position are determined based on a preset position image of the camera at a preset position, the camera parameters at the preset position, and the current position image.

[0023] According to another aspect of an embodiment of the present invention, there is provided an image processing method, comprising: determining camera parameters of the camera at the current position based on a preset position image of the camera at the preset position, the camera parameters at the preset position and a first current position image of the camera at the current position; acquiring a second current position image of the camera at the current position; correcting the second current position image using the camera parameters to obtain a corrected image; when the current position is any one of a plurality of focal positions of the camera, obtaining the corrected images respectively corresponding to the plurality of focal positions; generating a three-dimensional graphic based on the corrected images respectively corresponding to the plurality of focal positions, and displaying the generated three-dimensional graphic at a predetermined frame rate to present a virtual reality scene.

[0024] In an embodiment of the present invention, a method of calibrating the camera parameters of a camera at a preset position in advance is adopted, and the camera parameters of the camera at the current position are determined through the camera parameters of the camera at the preset position. Since the current position of the camera can be at any focal length of the camera, the purpose of fully automatically determining the camera parameters of the camera at any focal length is achieved, thereby achieving the technical effect of efficiently calibrating the camera, and further solving the technical problem of low camera calibration efficiency in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 A hardware structure block diagram of a computer terminal for implementing an image processing method is shown;

[0027] Figure 2 is a flowchart of an image processing method 1 according to Embodiment 1 of the present invention;

[0028] Figure 3 is a flowchart of a second image processing method according to embodiment 1 of the present invention;

[0029] Figure 4 is a flowchart of the image processing method 3 according to the first embodiment of the present invention;

[0030] Figure 5 is a flowchart of an image processing method 4 according to embodiment 1 of the present invention;

[0031] Figure 6 is a schematic diagram of a camera calibration method provided according to an optional embodiment of the present invention;

[0032] Figure 7 is a structural block diagram of an image processing device 1 provided according to Embodiment 2 of the present invention;

[0033] Figure 8 is a structural block diagram of an image processing device 2 provided according to Embodiment 2 of the present invention;

[0034] Fig. 9 is a structural block diagram of an image processing device 3 provided according to Embodiment 2 of the present invention;

[0035] Fig.10 is a structural block diagram of an image processing device 4 provided according to embodiment 2 of the present invention;

[0036] Fig.11 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0040] Camera calibration: To determine the relative transformation relationship between the 3D geometric position of a point on the surface of an object and its corresponding point on a 2D image, it is necessary to establish a geometric model of camera imaging. These geometric model parameters include camera intrinsic parameters, camera extrinsic parameters, and distortion parameters. These geometric model parameters are camera parameters, and the process of solving camera parameters is called camera calibration.

[0041] Camera intrinsic parameters: K is generally used to represent the camera intrinsic parameters, which describes the internal parameters of the camera, including the camera focal length, the position of the principal point, etc. These are the inherent properties of the camera.

[0042] Distortion parameters: In geometric optics and cathode ray tube (CRT) displays, distortion is a deviation from a rectilinear projection. Distortion can generally be divided into two categories, including radial distortion and tangential distortion. Generally, radial distortion sometimes also has slight tangential distortion. Distortion parameters are used to describe distortion, so distortion parameters include radial distortion parameters and tangential distortion parameters.

[0043] Camera extrinsic parameters: rotation matrix and translation vector from world coordinate system to camera coordinate system. Camera extrinsic parameters are generally represented by R and t. R is a rotation matrix that can be converted into a three-dimensional rotation vector, which represents the rotation angle around the x, y, and z axes respectively. t is a translation vector, which represents the translation amount in the x, y, and z directions respectively.

[0044] Two-dimensional and three-dimensional corresponding points: two-dimensional image feature points in the camera image (including corner points, scale-invariant feature transform (SIFT) feature points, ORB (Oriented Fast and Rotated BRIEF) feature points, etc.) and the corresponding three-dimensional space feature points in the virtual scene model (including corner points, texture feature points, etc.).

[0045] 2D feature matching: Matching of 2D feature points of different images taken by a variable focal length camera at different focal lengths. 2D feature points include corner points, SIFT feature points, ORB feature points, etc. Feature point matching usually uses the nearest feature vector matching method.

[0046] Example 1

[0047] According to an embodiment of the present invention, a method embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] The method embodiment provided in Embodiment 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (shown as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0049] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0050] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the image processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, to implement the vulnerability detection method of the above-mentioned application program. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] The transmission device is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0052] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0053] Under the above operating environment, this application provides Figure 2 The image processing method shown. Figure 2 is a flowchart of an image processing method 1 according to Embodiment 1 of the present invention. Figure 2 As shown, the method comprises the following steps:

[0054] Step S202, obtaining a preset position image of the camera at the preset position, and camera parameters at the preset position;

[0055] Step S204, collecting a current position image of the camera at the current position;

[0056] Step S206, determining the camera parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position.

[0057] Through the above steps, the camera parameters of the camera at the preset position are calibrated in advance, and the camera parameters of the camera at the current position are determined through the camera parameters of the camera at the preset position. Since the current position of the camera can be any focal length of the camera, the purpose of fully automatically determining the camera parameters of the camera at any focal length is achieved, thereby achieving the technical effect of efficiently calibrating the camera, and further solving the technical problem of low camera calibration efficiency in related technologies.

[0058] As an optional embodiment, the camera may be a variable focal length camera, and the focal length change of the variable focal length camera may be continuous, that is, the variable focal length of the camera may be any focal length between the maximum focal length and the minimum focal length of the camera.

[0059] As an optional embodiment, the preset position mentioned above may be the focal length of the camera, that is, any focal length between the maximum focal length and the minimum focal length of the camera with variable focal length mentioned above. The current position mentioned above may also be the focal length of the camera, that is, any focal length between the maximum focal length and the minimum focal length of the camera with variable focal length mentioned above.

[0060] As an optional embodiment, there can be multiple preset positions, at least two, one for the maximum focal length of the camera, and the other for the minimum focal length of the camera. The more preset positions there are, the more accurate it is to select the preset position closest to the current position of the camera, so that the camera parameters of the camera at the current position can be determined more quickly and efficiently.

[0061] As an optional embodiment, the camera parameters of the camera include camera intrinsic parameters, camera extrinsic parameters and distortion parameters. The camera parameters of the camera at the preset position include: the camera intrinsic parameters of the camera at the preset position, the camera extrinsic parameters of the camera at the preset position and the distortion parameters of the camera at the preset position. The camera parameters of the camera at the current position include: the camera intrinsic parameters of the camera at the current position, the camera extrinsic parameters of the camera at the current position and the distortion parameters of the camera at the current position. Among them, the camera intrinsic parameters may include: the focal length of the camera, the position of the principal point, etc. The camera extrinsic parameters may include: the rotation matrix and the translation vector from the world coordinate system to the camera coordinate system. The distortion parameters may include: radial distortion parameters and tangential distortion parameters.

[0062] As an optional embodiment, when determining the camera parameters of the camera at the current position based on the preset position image, the current position image and the camera parameters at the preset position, a variety of methods can be used. For example, it can be implemented in the following ways: determining the initial parameters of the camera at the current position based on the preset position image, the current position image and the camera parameters at the preset position; optimizing the initial parameters based on the preset position image and the current position image to obtain the camera parameters of the camera at the current position.

[0063] As an optional embodiment, the initial parameters are optimized, including: when the initial parameters include: camera intrinsic parameters, distortion parameters, and camera extrinsic parameters, the camera extrinsic parameters are kept unchanged, and the camera intrinsic parameters and distortion parameters are optimized. Since the zoom of a zoom camera does not change the physical installation position of the camera, it can be assumed that the camera extrinsic parameters remain unchanged, and the camera intrinsic parameters and distortion parameters are optimized. The camera intrinsic parameters may include: focal length, the position of the principal point, etc. Since the position of the principal point can be fixed at the center of the image, the optimization of the camera intrinsic parameters is equivalent to optimizing the focal length. Therefore, optionally, the optimization of the initial parameters may include: optimizing the focal length and distortion parameters.

[0064] As an optional embodiment, the initial parameters of the camera at the current position can be determined in a variety of ways according to the preset position image, the current position image and the camera parameters at the preset position. For example, the following method can be used: when there are multiple preset position images, a target preset position image is selected from the multiple preset position images, wherein the target preset position image is a preset position image corresponding to the target preset position closest to the current position; a scaling factor of the target preset position image relative to the current position image is obtained; and the initial parameters of the camera at the current position are determined according to the scaling factor and the camera parameters of the camera at the target preset position. The preset position image corresponding to the target preset position closest to the current position is selected from the multiple preset position images. Since the camera parameters of the current position are determined according to the camera parameters of the closest preset position, the camera parameters of the camera at the current position obtained in this way can also be more accurate. The scaling factor of the target preset position image relative to the current position image can reflect the deviation between the camera parameters of the preset position and the camera parameters of the current position to a certain extent. Therefore, the camera parameters of the camera at the current position can be determined according to the camera parameters of the preset position and the scaling factor.

[0065] As an optional embodiment, when selecting a target pre-position image from a plurality of pre-position images, a variety of methods may be used, for example, including: determining the scaling factors of the plurality of pre-position images relative to the current position image; correcting the scaling factors corresponding to the plurality of pre-position images to obtain a plurality of corrected scaling factors; and determining the target pre-position image according to the plurality of corrected scaling factors. Since there may be a certain error in the process of determining the scaling factor, in order to ensure the accuracy of the camera parameters of the camera at the current position determined according to the scaling factor, when determining the camera parameters of the camera at the current position according to the scaling factor, the scaling factor may be corrected, and then the camera parameters of the camera at the current position may be determined according to the corrected scaling factor.

[0066] As an optional embodiment, when determining the scaling factors of multiple pre-position images relative to the current position image, it can be determined based on the features of the pre-position images and the features of the current position image. For example, the following processing can be used to implement: feature matching is performed on multiple pre-position images with the current position image to obtain a first feature point set in the current position image and a second feature point set of multiple pre-position images; the first feature point set is decentralized to obtain a decentralized first feature point set, and the second feature point sets of multiple pre-position images are decentralized to obtain a decentralized second feature point set; based on the decentralized first feature point set and the decentralized second feature point set corresponding to the multiple pre-position images, the homography matrix between the current position image and the feature points of the multiple pre-position images is determined; based on the homography matrix between the current position image and the feature points of the multiple pre-position images, the scaling factors of the multiple pre-position images relative to the current position image are determined. It should be noted that the features used for feature matching can include multiple types, such as SIFT feature points, ORB feature points, BRISK feature points, corner points, etc. However, when matching, the pre-position image and the current position image should be the same. In addition, it should be noted that when the pre-position image and the current position image are feature matched, the number of features extracted from the pre-position image and the number of features extracted from the current position image may be different.

[0067] As an optional embodiment, when the zoom factors corresponding to multiple preset position images are corrected and multiple corrected zoom factors are obtained, the following processing can be specifically adopted: when the zoom factor is less than 1, the zoom factor is used as the corrected zoom factor; when the zoom factor is greater than 1, the inverse of the zoom factor is used as the corrected zoom factor; the target preset position image is determined according to the multiple corrected zoom factors, including: determining the maximum value among the multiple corrected zoom factors; determining the preset position image corresponding to the maximum value as the target preset position image. Through the above correction processing, obviously inaccurate zoom factors can be effectively avoided. In addition, the preset position corresponding to the maximum zoom factor among the multiple preset position images is the preset position closest to the current position. Since the camera parameters of the camera at the nearest preset position are closest to the camera parameters of the camera at the current position, the camera parameters of the camera at the current position determined based on the camera parameters of the camera at the nearest preset position are also the most accurate.

[0068] As an optional embodiment, the initial parameters are optimized according to the preset position image and the current position image to obtain the camera parameters of the camera at the current position, including: back-projecting the second feature point in the preset position image into the camera coordinate system to obtain the first coordinate of the second feature point in the preset position image in the camera coordinate system; forward projecting the first coordinate onto the image plane of the current position to obtain the third feature point on the image plane of the current position; optimizing the initial parameters by minimizing the reprojection error of the third feature point relative to the first feature point in the current position image to obtain the camera parameters of the camera at the current position. By comparing the real current position image with the reprojected image obtained by projecting according to the initialized camera parameters, for example, comparing the error between the coordinates (or feature points) in the real current position image and the coordinates (or feature points) in the reprojected image, the initialized camera parameters are optimized to obtain the accurate camera parameters of the camera at the current position.

[0069] According to an embodiment of the present invention, there is also provided an image processing method. Figure 3 is a flowchart of the second image processing method according to the first embodiment of the present invention. Figure 3 As shown, the method comprises the following steps:

[0070] Step S302, displaying a current position image of the camera at the current position on the display interface;

[0071] Step S304, receiving an input operation, wherein the input operation is used to request display of a three-dimensional picture corresponding to the current position image;

[0072] Step S306, in response to the input operation, a three-dimensional picture corresponding to the current position image is displayed on the display interface, wherein the three-dimensional picture is determined based on the correspondence between the coordinates in the current position image and the three-dimensional space coordinates, the correspondence is determined based on the camera parameters of the camera at the current position, the camera parameters of the camera at the current position are determined based on the preset position image of the camera at the preset position, the camera parameters at the preset position, and the current position image.

[0073] When the current position image of the camera at any current focal length position is displayed on the display interface, the camera parameters of the camera at the preset position can be calibrated in advance, and the camera parameters of the camera at the current position can be determined by the camera parameters of the camera at the preset position, so that the three-dimensional picture corresponding to the current arbitrary focal length position can be displayed on the display interface. Since the current position of the camera can be any focal length of the camera, the purpose of fully automatically determining the camera parameters of the camera at any focal length and then fully automatically displaying the corresponding three-dimensional picture is achieved. This not only solves the technical problem of low camera calibration efficiency in related technologies, but also achieves the technical effect of efficiently calibrating the camera and efficiently displaying the corresponding three-dimensional picture.

[0074] As an optional embodiment, in the three-dimensional picture displayed on the display interface, in order to increase the user's additional experience, the target object in the three-dimensional picture can be identified in the displayed three-dimensional picture, and the identified target object can be highlighted on the display interface. By highlighting the identified target object, the picture content of the three-dimensional picture can be effectively displayed. It should be noted that the target object in the above three-dimensional picture can be a person or an object; it can be an active object or a static object; etc.

[0075] According to an embodiment of the present invention, there is also provided an image processing method. Figure 4 is a flowchart of the image processing method 3 according to the first embodiment of the present invention. Figure 4 As shown, the method comprises the following steps:

[0076] Step S402, receiving a current position image of the camera at the current position sent by the client device;

[0077] Step S404, feeding back camera parameters of the camera at the current position to the client device, wherein the camera parameters of the camera at the current position are determined according to the preset position image of the camera at the preset position, the camera parameters at the preset position and the current position image.

[0078] Through the above processing, a method is adopted in which the current position image sent by the client device is received, and the camera parameters of the camera at the current position are fed back to the client device, wherein the camera parameters of the camera at the current position can be determined by the camera parameters of the camera at the preset position. Since the current position of the camera can be at any focal length of the camera, the purpose of fully automatically determining the camera parameters of the camera at any focal length is achieved, and then the camera parameters at the required focal length are fully automatically provided to the client device. This not only solves the technical problem of low camera calibration efficiency in related technologies, but also achieves the technical effect of efficiently calibrating the camera and efficiently responding to the client device.

[0079] According to an embodiment of the present invention, there is also provided an image processing method. Figure 5 is a flowchart of an image processing method 4 according to embodiment 1 of the present invention. Figure 5 As shown, the method comprises the following steps:

[0080] Step S502, determining the camera parameters of the camera at the current position according to the preset position image of the camera at the preset position, the camera parameters at the preset position and the first current position image of the camera at the current position;

[0081] Step S504, capturing a second current position image of the camera at the current position; it should be noted that the second current position image of the camera at the current position may be an image captured again by the camera at the current position relative to the first current position image.

[0082] Step S506, correcting the second current position image using the camera parameters to obtain a corrected image; since the camera parameters include distortion parameters, the image re-captured at the current position can be corrected using the camera parameters to obtain a more accurate corrected image.

[0083] Step S508, when the current position is any one of the multiple focal positions of the camera, obtaining corrected images corresponding to the multiple focal positions respectively;

[0084] Step S510, generating a three-dimensional image according to the corrected images corresponding to the multiple focal positions, and displaying the generated three-dimensional image at a predetermined frame rate to show the virtual reality scene. It should be noted that, since there is a certain conversion relationship between the two-dimensional image and the three-dimensional space, when generating a three-dimensional graphic according to the corrected images corresponding to the multiple focal positions, the conversion relationship can be first determined according to the above-mentioned calibrated camera parameters, and then the corrected images corresponding to the multiple focal positions are generated into a three-dimensional image according to the conversion relationship, and then the generated three-dimensional image is displayed at a predetermined frame rate to show the virtual reality scene. The above-mentioned predetermined frame rate is used to realize the real-time display of the generated three-dimensional image, for example, it can be higher than 30 frames / second, so as to achieve a better virtual reality scene.

[0085] Through the above processing, the second current position image captured again at the current position is corrected by using the preset position image of the camera at the preset position, the camera parameters at the preset position, and the camera parameters at the current position determined by the first current position image of the camera at the current position, so as to facilitate the subsequent generation of a three-dimensional image based on the corrected image, and to display the three-dimensional image in stereo to show the virtual reality scene. Since the current position of the camera can be at any focal length of the camera, the camera parameters at any focal length are fully automatically determined based on the preset position image of the camera at the preset position and the camera parameters at the preset position, and then the three-dimensional image is fully automatically generated, and the three-dimensional image is displayed in stereo to show the virtual reality scene. This not only solves the technical problem of low camera calibration efficiency in the related technology, but also achieves the technical effect of efficient calibration of the camera and efficient response to the client device.

[0086] Based on the above embodiments and optional embodiments, taking the camera as a zoom camera and taking an urban scene as an example, an optional implementation method is provided, which is described in detail below.

[0087] In urban scenes, cameras are widely used in various fields as an efficient video image acquisition device. For example, they can be used in the field of municipal transportation to view pedestrians and vehicles in the target scene. However, there are certain differences between the two-dimensional attributes of the image screen and the three-dimensional attributes of the real scene. How to project the two-dimensional semantic information in the existing two-dimensional image, such as people and vehicles, into the real three-dimensional space to serve the refined management of the city. The core problem is to estimate the camera parameters of the camera (including the camera's intrinsic parameters, the camera's extrinsic parameters, and the distortion parameters). The core technology involved is the camera calibration algorithm, and the conversion relationship between the image's two-dimensional coordinates and the space's three-dimensional coordinates is established. In actual application scenarios, the target scene range is relatively wide, and a variable focal length camera is generally used to obtain a target image at a farther or closer distance by zooming, so as to clearly focus on objects at different distances. Continuous zooming will directly change the camera's intrinsic parameters, but since it is impossible to calibrate the camera in real time only through the camera screen, and then estimate the camera parameters, how to efficiently calibrate the variable focal length camera in the observation scene has practical application value. In addition, unlike ordinary cameras, the variable focal length camera is usually installed at a higher position and has a larger imaging field of view, which causes serious distortion of the captured image, which will directly affect the camera calibration accuracy.

[0088] Based on the above requirements of variable focal length cameras, it is necessary to implement calibration of variable focal length cameras. For ordinary cameras, during calibration, the camera calibration algorithms used include: traditional camera calibration algorithm, camera self-calibration method, and active vision camera calibration algorithm.

[0089] (1) Traditional camera calibration algorithm: It requires the use of a three-dimensional calibration object or a plane calibration object with known real size. By establishing the correspondence between the points on the calibration object with known coordinates and its image points, the optimization algorithm is used to solve the camera parameters of the camera model. A three-dimensional calibration object can be calibrated by a single image. Although the calibration accuracy is high, the processing and maintenance of high-precision three-dimensional calibration objects are difficult. Plane calibration objects are simpler to make than three-dimensional calibration objects, and the accuracy is easy to guarantee, but two or more images are required for calibration.

[0090] (2) Camera self-calibration method: Use some parallel or orthogonal constraints in the scene to calibrate the camera's internal and external parameters. The intersection of spatial parallel lines on the camera image plane is called the vanishing point. The vanishing point-based method usually uses quadratic curves or surface theory to solve the camera's internal and external parameters. However, due to the low accuracy of the vanishing point solution, the camera parameter errors estimated by this method are large. In addition, for distorted cameras, the distortion of spatial parallel lines on the imaging plane will also lead to a decrease in the accuracy of the vanishing point solution, which directly affects the calibration robustness.

[0091] (3) Active visual camera calibration algorithm: Using structure-from-motion technology, the camera’s motion is analyzed to restore the scene’s 3D geometric information while optimizing the camera’s internal and external parameters. This method does not require a known calibration object, but only requires moving the camera to capture images in the same scene. However, it is necessary to ensure that the baseline distance of the camera motion between adjacent images is large.

[0092] None of the above three types of calibration algorithms can solve the external parameters of the camera relative to the geographic coordinate system, because the above three methods are based on local three-dimensional information to achieve camera calibration, and naturally cannot achieve the positioning of objects in the image in real space. From the perspective of data acquisition, the installation position of variable focal length cameras is usually high and far away, and it is impossible to reasonably place calibration objects (such as calibration plates) within the line of sight to achieve camera calibration; in addition, since variable focal length cameras are usually installed outdoors, the unpredictable external environment has a greater impact on the image quality, resulting in heavy image noise, so the accuracy of self-calibration methods such as vanishing point calculation will be difficult to guarantee; in addition, for variable focal length cameras, because the focal length change is a continuous process, it is impossible to sample all continuous focal length ranges through discrete images, so the ordinary monocular camera calibration algorithm is only applicable to the calibration of fixed focal length monocular cameras, and is not applicable to variable focal length cameras.

[0093] In view of this, in this optional implementation, by utilizing the preset position calibration results of a small number of variable focal length cameras (usually, two preset positions are sufficient, namely the maximum focal length and the minimum focal length), an optimization algorithm based on two-dimensional feature matching is used to achieve fully automatic camera calibration under arbitrary zoom conditions, and the camera parameters of the camera are estimated, thereby realizing various applications in three-dimensional scene simulation, such as pedestrian and vehicle positioning, speed estimation, virtual reality, augmented reality, etc.

[0094] Figure 6 is a schematic diagram of a camera calibration method provided according to an optional embodiment of the present invention, such as Figure 6 As shown, the method includes:

[0095] (1) Preset position image selection and calibration: Discretely sample N preset position (N ≥ 2) images (either equal focal length interval sampling or non-equal focal length interval sampling is acceptable, and at least two preset positions, the maximum focal length and the minimum focal length, are included); at each preset position, the traditional algorithm is used to calibrate the camera parameters, including the focal length of the camera at the current preset position, the camera intrinsic parameters such as the principal point position, the camera extrinsic parameters for the conversion from the world coordinate system to the camera coordinate system, and the distortion parameters.

[0096] (2) Sampling an image at any focal length: Adjust the focal length of the variable focal length camera to the current position and obtain the image at the current position.

[0097] (3) Establish association with preset position images: calculate the two-dimensional feature matching relationship between the current position image and each preset position image, and obtain the two-dimensional feature matching pixel points.

[0098] (4) Initial calibration: During the initialization process, it is assumed that the zoom only affects the focal length in the camera intrinsic parameters, and the focal length is initialized based on the two-dimensional feature matching relationship and the camera imaging principle.

[0099] (5) Parameter optimization: In the parameter optimization stage, it is assumed that the camera external parameters remain unchanged, the fixed principal point is located at the center of the image, and the camera internal parameters (including focal length) and distortion parameters are optimized. A new internal parameter matrix is ​​constructed based on the initial focal length, and the two-dimensional feature matching pixels of the current position and the preset position are back-projected to the camera coordinate system. Based on the imaging principle, a reprojection error function is constructed in the camera coordinate system to further optimize the camera internal parameters and distortion parameters.

[0100] The following is a detailed description of the above three parts (3), (4), and (5). Note that unless otherwise specified in the following formulas, all coordinate transformations involving 3D to 2D have a perspective division operation by default, that is, the coordinates of the last dimension are normalized.

[0101] Associate with preset image

[0102] Assume that the preset position image set is l i (i=1, 2, ..., N), the internal parameter matrix of each preset position camera is obtained by calibrating using the traditional solution as K i , the corresponding focal length is expressed as f i , the principal point position p is fixed at the center of the image, and the distortion parameter is dist i ={k i1 , k i2 , p i1 , p i2}, the rotation matrix and translation vector of the camera extrinsic parameters are represented as R i and t i The camera image at the current position after adjusting the focal length is I c Here, SIFT feature matching is used to establish a two-dimensional feature matching relationship between the current position image and the preset position image, and the matching two-dimensional feature point sets are represented as F c and F i , the single feature point in the set is x c and x i . Note that for each set F i The number of feature points in is inconsistent, x c and x i It only refers to a feature point in the feature set.

[0103] Then, all feature points are decentralized to construct the objective function After decentralization, the homography matrix H between the two-dimensional feature points of the current position image and the two-dimensional feature points of the preset position image is calculated. i ∈R 3×3 .

[0104] Initial calibration

[0105] Estimate the scaling factor of the pre-positioned image relative to the current position image based on the homography matrix Among them, H i (m, n) represents the element in the mth row and nth column of the matrix. After that, the scaling factor can be corrected to: Compare all s′ i Take the subscript j corresponding to the maximum value to determine that the current position is closest to the jth preset position. Based on this, the focal length of the current position is initialized to f = s j f j , the distortion parameter is dist = {k1 = k j1 , k2=k j2 , p1=p j1 , p2=p j2}, the rotation matrix and translation vector corresponding to the camera extrinsic parameters are initialized as R = R j and t = t j .

[0106] Parameter Optimization

[0107] In the camera parameter optimization stage, since zooming does not change the physical installation position of the camera, it can be assumed that the camera external parameters remain unchanged, so only the focal length f and the distortion parameter dist are optimized in this stage. The calculation process is as follows:

[0108] (1) At each preset position i, according to the calibrated internal parameter matrix K i and the distortion parameter dist i The image feature point x i Perform distortion correction and back-project to the camera coordinate system to obtain

[0109] (2) For the current position, construct the internal parameter matrix K based on the estimated focal length f and the initialized distortion parameter dist. Forward projection onto the imaging plane at the current position is as follows:

[0110]

[0111]

[0112] in, k1 and k2 represent radial distortion parameters, and p1 and p2 represent tangential distortion parameters. Then project the distorted coordinates onto the imaging plane to obtain the estimated pixel coordinates:

[0113] (3) Based on this, the energy function is constructed: The goal is to minimize the reprojection error and solve the camera intrinsic parameters and distortion parameters;

[0114] (4) Optimization method: K and dist in the above energy function use the results determined by the above initial calibration as initial values. Optionally, the objective function is optimized using the Levenberg-Marquard algorithm. It should be noted that when optimizing the objective function, other optimization methods can also be used, such as the steepest descent method, the Newton method, and the Gauss-Newton method. It should be noted that the above Levenberg-Marquard algorithm is a combination of the steepest descent method (gradient descent method) and Gauss-Newton.

[0115] In addition, it should be noted that the two-dimensional image feature points mentioned above include but are not limited to SIFT feature points, ORB feature points, BRISK feature points, corner points, etc. These feature points can also be used to match the current position image and the preset position image.

[0116] Through the above optional implementation, the following beneficial effects can be achieved:

[0117] (1) Aiming at the digital city scene, a universal variable focal length camera calibration algorithm is proposed to achieve camera parameter estimation at any focal length.

[0118] (2) The focal length of the camera changes continuously, and the continuous focal length change range cannot be obtained through discrete sampling. This scheme can automatically estimate the camera parameters of the variable focal length camera at any focal length by calibrating a small number of preset positions (at least two (camera parameters at the maximum focal length and the minimum focal length)) in advance.

[0119] (3) In addition, it should be noted that the optimization strategy proposed in this optional implementation can be extended to any imaging model and distortion model.

[0120] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0122] Example 2

[0123] According to an embodiment of the present invention, a device for implementing the above-mentioned image processing method 1 is also provided. Figure 7 is a structural block diagram of an image processing device 1 provided according to Embodiment 2 of the present invention, as shown Figure 7 As shown, the device includes: a first acquisition module 72, a first collection module 74 and a first determination module 76. The device is described below.

[0124] The first acquisition module 72 is used to acquire a preset position image of the camera at the preset position and the camera parameters at the preset position; the first acquisition module 74 is connected to the first acquisition module 72 and is used to acquire a current position image of the camera at the current position; the first determination module 76 is connected to the first acquisition module 74 and is used to determine the camera parameters of the camera at the current position based on the preset position image, the current position image and the camera parameters at the preset position.

[0125] According to an embodiment of the present invention, a device for implementing the above-mentioned second image processing method is also provided. Figure 8 is a structural block diagram of an image processing device 2 provided according to Embodiment 2 of the present invention. Figure 8 As shown, the device includes: a first display module 82, a first receiving module 84 and a second display module 86. The device is described below.

[0126] A first display module 82 is used to display a current position image of the camera at the current position on a display interface; a first receiving module 84 is connected to the first display module 82 and is used to receive input operations, wherein the input operations are used to request display of a three-dimensional picture corresponding to the current position image; a second display module 86 is connected to the first receiving module 84 and is used to respond to input operations and display a three-dimensional picture corresponding to the current position image on a display interface, wherein the three-dimensional picture is determined based on a correspondence between coordinates in the current position image and three-dimensional space coordinates, the correspondence is determined based on camera parameters of the camera at the current position, and the camera parameters of the camera at the current position are determined based on a preset position image of the camera at a preset position, the camera parameters at the preset position, and the current position image.

[0127] According to an embodiment of the present invention, a device for implementing the above-mentioned image processing method 3 is also provided. Fig. 9 is a structural block diagram of an image processing device 3 provided according to Embodiment 2 of the present invention. Fig. 9 As shown, the device includes: a second receiving module 92 and a feedback module 94. The device is described below.

[0128] A second receiving module 92 is used to receive a current position image of the camera at the current position sent by a client device; a feedback module 94 is connected to the above-mentioned second receiving module 92, and is used to feedback camera parameters of the camera at the current position to the client device, wherein the camera parameters of the camera at the current position are determined according to a preset position image of the camera at the preset position, the camera parameters at the preset position and the current position image.

[0129] According to an embodiment of the present invention, a device for implementing the above-mentioned image processing method 4 is also provided. Fig.10 is a structural block diagram of an image processing device 4 provided according to Embodiment 2 of the present invention. Fig.10 As shown, the device includes: a second determination module 1002, a second acquisition module 1004, a correction module 1006, a second acquisition module 1008 and a generation module 1010. The device is described below.

[0130] The second determination module 1002 is used to determine the camera parameters of the camera at the current position according to the preset position image of the camera at the preset position, the camera parameters at the preset position and the first current position image of the camera at the current position; the second acquisition module 1004 is connected to the above-mentioned second determination module 1002, and is used to acquire the second current position image of the camera at the current position; the correction module 1006 is connected to the above-mentioned second acquisition module 1004, and uses the camera parameters to correct the second current position image to obtain a corrected image; the second acquisition module 1008 is connected to the above-mentioned correction module 1006, and when the current position is any one of the multiple focal positions of the camera, the corrected images corresponding to the multiple focal positions are obtained; the generation module 1010 is connected to the above-mentioned second acquisition module 1008, and generates a three-dimensional graphic according to the corrected images corresponding to the multiple focal positions, and displays the generated three-dimensional graphic at a predetermined frame rate to show the virtual reality scene.

[0131] It should be noted that the above modules correspond to the steps in Example 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0132] Example 3

[0133] The embodiment of the present invention may provide a computer terminal, which may be any computer terminal device (ie, computer device) in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0134] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0135] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the image processing method of the application: obtaining a preset position image of the camera at the preset position, and the camera parameters at the preset position; acquiring a current position image of the camera at the current position; and determining the camera parameters of the camera at the current position based on the preset position image, the current position image and the camera parameters at the preset position.

[0136] Optionally, Fig.11 is a structural block diagram of a computer terminal according to an embodiment of the present invention. Fig.11 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 112, a memory 114, etc.

[0137] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the image processing method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a preset position image of the camera at the preset position, and the camera parameters at the preset position; collect the current position image of the camera at the current position; determine the camera parameters of the camera at the current position based on the preset position image, the current position image and the camera parameters at the preset position.

[0139] Optionally, the processor may also execute program code for the following steps: determining the camera parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position, including: determining the initial parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; optimizing the initial parameters according to the preset position image and the current position image to obtain the camera parameters of the camera at the current position.

[0140] Optionally, the processor may also execute program code of the following steps: determining initial parameters of the camera at the current position according to a preset position image, a current position image and camera parameters at the preset position, including: in the case where there are multiple preset position images, selecting a target preset position image from the multiple preset position images, wherein the target preset position image is a preset position image corresponding to a target preset position closest to the current position; obtaining a scaling factor of the target preset position image relative to the current position image; and determining the initial parameters of the camera at the current position according to the scaling factor and the camera parameters of the camera at the target preset position.

[0141] Optionally, the processor may also execute program code of the following steps: selecting a target preset position image from a plurality of preset position images, including: determining scaling factors of the plurality of preset position images relative to the current position image; performing correction processing on the scaling factors corresponding to the plurality of preset position images to obtain a plurality of corrected scaling factors; and determining the target preset position image according to the plurality of corrected scaling factors.

[0142] Optionally, the processor may also execute the program code of the following steps: determining the scaling factors of multiple preset-position images relative to the current-position image, including: performing feature matching on the multiple preset-position images with the current-position image respectively to obtain a first feature point set in the current-position image and a second feature point set of the multiple preset-position images respectively; decentralizing the first feature point set to obtain a decentralized first feature point set, and decentralizing the second feature point sets of the multiple preset-position images respectively to obtain a decentralized second feature point set; determining the homography matrix between the current-position image and the feature points of the multiple preset-position images respectively based on the decentralized first feature point set and the decentralized second feature point sets corresponding to the multiple preset-position images respectively; determining the scaling factors of the multiple preset-position images relative to the current-position image respectively based on the homography matrix between the current-position image and the feature points of the multiple preset-position images respectively.

[0143] Optionally, the processor may also execute the program code of the following steps: correcting the scaling factors corresponding to a plurality of preset-position images to obtain a plurality of corrected scaling factors, including: when the scaling factor is less than 1, using the scaling factor as the corrected scaling factor; when the scaling factor is greater than 1, using the inverse of the scaling factor as the corrected scaling factor; determining the target preset-position image according to the plurality of corrected scaling factors, including: determining the maximum value among the plurality of corrected scaling factors; and determining the preset-position image corresponding to the maximum value as the target preset-position image.

[0144] Optionally, the processor may also execute program code of the following steps: optimizing initial parameters according to the preset position image and the current position image to obtain camera parameters of the camera at the current position, including: back-projecting the second feature point in the preset position image into the camera coordinate system to obtain the first coordinate of the second feature point in the preset position image in the camera coordinate system; forward projecting the first coordinate onto the image plane at the current position to obtain the third feature point on the image plane at the current position; optimizing the initial parameters by minimizing the reprojection error of the third feature point relative to the first feature point in the current position image to obtain the camera parameters of the camera at the current position.

[0145] Optionally, the processor may also execute the program code of the following steps: optimizing the initial parameters, including: when the initial parameters include: camera intrinsic parameters, distortion parameters, and camera extrinsic parameters, keeping the camera extrinsic parameters unchanged, and optimizing the camera intrinsic parameters and distortion parameters.

[0146] Optionally, the processor may also execute program code of the following steps: there are two preset positions, one is the maximum focal length of the camera, and the other is the minimum focal length of the camera.

[0147] Optionally, the processor may further execute program code of the following steps: the camera includes a camera with variable focal length.

[0148] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: displaying the current position image of the camera at the current position on the display interface; receiving an input operation, wherein the input operation is used to request display of a three-dimensional picture corresponding to the current position image; in response to the input operation, displaying the three-dimensional picture corresponding to the current position image on the display interface, wherein the three-dimensional picture is determined based on the correspondence between the coordinates in the current position image and the three-dimensional space coordinates, the correspondence is determined based on the camera parameters of the camera at the current position, the camera parameters of the camera at the current position are determined based on the preset position image of the camera at the preset position, the camera parameters at the preset position, and the current position image.

[0149] Optionally, the processor may also execute program code of the following steps: identifying a target object in a three-dimensional image, and highlighting the identified target object on a display interface.

[0150] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive the current position image of the camera at the current position sent by the client device; and feed back the camera parameters of the camera at the current position to the client device, wherein the camera parameters of the camera at the current position are determined according to the preset position image of the camera at the preset position, the camera parameters at the preset position and the current position image.

[0151] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine the camera parameters of the camera at the current position according to the preset position image of the camera at the preset position, the camera parameters at the preset position and the first current position image of the camera at the current position; collect the second current position image of the camera at the current position; use the camera parameters to correct the second current position image to obtain a corrected image; when the current position is any one of the multiple focal positions of the camera, obtain the corrected images corresponding to the multiple focal positions respectively; generate a three-dimensional graphic according to the corrected images corresponding to the multiple focal positions respectively, and display the generated three-dimensional graphic at a predetermined frame rate to show the virtual reality scene.

[0152] By adopting the embodiment of the present invention, the camera parameters of the camera at the preset position are calibrated in advance, and the camera parameters of the camera at the current position are determined through the camera parameters of the camera at the preset position. Since the current position of the camera can be any focal length of the camera, the purpose of fully automatically determining the camera parameters of the camera at any focal length is achieved, thereby achieving the technical effect of efficiently calibrating the camera, and further solving the technical problem of low camera calibration efficiency in the related art.

[0153] It can be understood by those skilled in the art that Fig.11 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.11 The structure of the electronic device is not limited. For example, the computer terminal 11 may also include Fig.11 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.11 Different configurations are shown.

[0154] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0155] Example 4

[0156] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image processing method provided in the embodiment 1 above.

[0157] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0158] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: acquiring a preset position image of the camera at the preset position, and camera parameters at the preset position; acquiring a current position image of the camera at the current position; and determining the camera parameters of the camera at the current position based on the preset position image, the current position image and the camera parameters at the preset position.

[0159] Optionally, in this embodiment, the storage medium is also configured to store program codes for executing the following steps: determining the camera parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position, including: determining the initial parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; optimizing the initial parameters according to the preset position image and the current position image to obtain the camera parameters of the camera at the current position.

[0160] Optionally, in this embodiment, the storage medium is also configured to store a program code for executing the following steps: determining the initial parameters of the camera at the current position based on the preset position image, the current position image and the camera parameters at the preset position, including: in the case where there are multiple preset position images, selecting a target preset position image from the multiple preset position images, wherein the target preset position image is a preset position image corresponding to a target preset position closest to the current position; obtaining a scaling factor of the target preset position image relative to the current position image; and determining the initial parameters of the camera at the current position based on the scaling factor and the camera parameters of the camera at the target preset position.

[0161] Optionally, in this embodiment, the storage medium is also configured to store program codes for executing the following steps: selecting a target preset position image from a plurality of preset position images, including: determining scaling factors of the plurality of preset position images relative to the current position image; performing correction processing on the scaling factors corresponding to the plurality of preset position images to obtain a plurality of corrected scaling factors; and determining the target preset position image based on the plurality of corrected scaling factors.

[0162] Optionally, in this embodiment, the storage medium is also configured to store program codes for executing the following steps: determining scaling factors of multiple preset-position images relative to the current-position image, including: performing feature matching on the multiple preset-position images with the current-position image respectively to obtain a first feature point set in the current-position image and a second feature point set of the multiple preset-position images respectively; decentralizing the first feature point set to obtain a decentralized first feature point set, and decentralizing the second feature point sets of the multiple preset-position images respectively to obtain a decentralized second feature point set; determining the homography matrices between the current-position image and the feature points of the multiple preset-position images respectively based on the decentralized first feature point set and the decentralized second feature point sets corresponding to the multiple preset-position images respectively; determining the scaling factors of the multiple preset-position images relative to the current-position image respectively based on the homography matrices between the current-position image and the feature points of the multiple preset-position images respectively.

[0163] Optionally, in this embodiment, the storage medium is also configured to store program codes for executing the following steps: correcting the scaling factors corresponding to multiple preset position images to obtain multiple corrected scaling factors, including: when the scaling factor is less than 1, using the scaling factor as the corrected scaling factor; when the scaling factor is greater than 1, using the reciprocal of the scaling factor as the corrected scaling factor; determining the target preset position image according to the multiple corrected scaling factors, including: determining the maximum value among the multiple corrected scaling factors; determining the preset position image corresponding to the maximum value as the target preset position image.

[0164] Optionally, in this embodiment, the storage medium is also configured to store program codes for executing the following steps: optimizing initial parameters according to the preset position image and the current position image to obtain camera parameters of the camera at the current position, including: back-projecting the second feature point in the preset position image into the camera coordinate system to obtain the first coordinate of the second feature point in the preset position image in the camera coordinate system; forward projecting the first coordinate onto the image plane at the current position to obtain a third feature point on the image plane at the current position; optimizing the initial parameters by minimizing the reprojection error of the third feature point relative to the first feature point in the current position image to obtain the camera parameters of the camera at the current position.

[0165] Optionally, in this embodiment, the storage medium is also configured to store program codes for performing the following steps: optimizing the initial parameters, including: when the initial parameters include: camera intrinsic parameters, distortion parameters, and camera extrinsic parameters, keeping the camera extrinsic parameters unchanged, and optimizing the camera intrinsic parameters and distortion parameters.

[0166] Optionally, in this embodiment, the storage medium is further configured to store program codes for executing the following steps: there are two preset positions, one is the maximum focal length of the camera, and the other is the minimum focal length of the camera.

[0167] Optionally, in this embodiment, the storage medium is further configured to store program codes for executing the following steps: the camera includes a camera with a variable focal length.

[0168] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: displaying a current position image of the camera at the current position on a display interface; receiving an input operation, wherein the input operation is used to request display of a three-dimensional picture corresponding to the current position image; in response to the input operation, displaying a three-dimensional picture corresponding to the current position image on the display interface, wherein the three-dimensional picture is determined based on a correspondence between coordinates in the current position image and three-dimensional space coordinates, the correspondence is determined based on camera parameters of the camera at the current position, the camera parameters of the camera at the current position are determined based on a preset position image of the camera at a preset position, the camera parameters at the preset position, and the current position image.

[0169] Optionally, in this embodiment, the storage medium is further configured to store program codes for executing the following steps: identifying a target object in a three-dimensional image, and highlighting the identified target object on a display interface.

[0170] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: receiving a current position image of the camera at the current position sent by a client device; and feeding back camera parameters of the camera at the current position to the client device, wherein the camera parameters of the camera at the current position are determined based on a preset position image of the camera at a preset position, the camera parameters at the preset position, and the current position image.

[0171] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: determining camera parameters of the camera at the current position based on a preset position image of the camera at the preset position, camera parameters at the preset position, and a first current position image of the camera at the current position; acquiring a second current position image of the camera at the current position; correcting the second current position image using the camera parameters to obtain a corrected image; when the current position is any one of a plurality of focal positions of the camera, obtaining corrected images corresponding to the plurality of focal positions respectively; generating a three-dimensional graphic based on the corrected images corresponding to the plurality of focal positions respectively, and displaying the generated three-dimensional graphic at a predetermined frame rate to display a virtual reality scene.

[0172] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0173] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0175] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0178] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: include: Acquire a preset position image of the camera at the preset position, and camera parameters at the preset position; Acquire a current position image of the camera at the current position; Determining the camera parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; Wherein, determining the camera parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position includes: determining the initial parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; back-projecting the second feature point in the preset position image into the camera coordinate system to obtain the first coordinate of the second feature point in the preset position image in the camera coordinate system; forward projecting the first coordinate onto the image plane of the current position to obtain the third feature point on the image plane of the current position; optimizing the initial parameters by minimizing the reprojection error of the third feature point relative to the first feature point in the current position image to obtain the camera parameters of the camera at the current position.

2. The method according to claim 1, characterized in that Determining initial parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position includes: In the case where there are multiple preset position images, selecting a target preset position image from the multiple preset position images, wherein the target preset position image is a preset position image corresponding to a target preset position closest to the current position; Obtaining a scaling factor of the target preset position image relative to the current position image; The initial parameters of the camera at the current position are determined according to the zoom factor and the camera parameters of the camera at the target preset position.

3. The method according to claim 2, characterized in that Selecting the target preset position image from the plurality of preset position images comprises: Determining scaling factors of the plurality of preset position images relative to the current position image; Correcting the scaling factors corresponding to the plurality of preset position images to obtain a plurality of corrected scaling factors; The target preset position image is determined according to the multiple modified scaling factors.

4. The method according to claim 3, characterized in that Determining the scaling factors of the plurality of preset position images relative to the current position image comprises: Perform feature matching on the multiple pre-position images and the current position image respectively to obtain a first feature point set in the current position image and a second feature point set of the multiple pre-position images respectively; Decentralizing the first feature point set to obtain a decentralized first feature point set, and decentralizing the second feature point sets of the plurality of preset position images to obtain decentralized second feature point sets; Determine, according to the decentralized first feature point set and the decentralized second feature point set corresponding to the plurality of preset position images respectively, a homography matrix between the current position image and the feature points of the plurality of preset position images respectively; The scaling factors of the plurality of preset-position images relative to the current-position image are determined respectively according to the homography matrices between the feature points of the current-position image and the plurality of preset-position images.

5. The method according to claim 3, characterized in that: Correcting the scaling factors corresponding to the plurality of preset position images to obtain a plurality of corrected scaling factors, including: when the scaling factor is less than 1, using the scaling factor as the corrected scaling factor; when the scaling factor is greater than 1, using the reciprocal of the scaling factor as the corrected scaling factor; Determining the target pre-position image according to the multiple corrected scaling factors includes: determining a maximum value among the multiple corrected scaling factors; and determining a pre-position image corresponding to the maximum value as the target pre-position image.

6. The method according to claim 1, characterized in that Optimizing the initial parameters includes: When the initial parameters include: camera intrinsic parameters, distortion parameters, and camera extrinsic parameters, the camera extrinsic parameters are kept unchanged, and the camera intrinsic parameters and the distortion parameters are optimized.

7. The method according to any one of claims 1 to 6, characterized in that There are two preset positions, one is the maximum focal length of the camera, and the other is the minimum focal length of the camera.

8. The method according to claim 7, characterized in that The camera comprises a variable focal length camera.

9. An image processing method, characterized in that: include: Display the current position image of the camera at the current position on the display interface; receiving an input operation, wherein the input operation is used to request display of a three-dimensional picture corresponding to the current position image; In response to the input operation, a three-dimensional picture corresponding to the current position image is displayed on the display interface, wherein the three-dimensional picture is determined according to a correspondence between coordinates in the current position image and three-dimensional space coordinates, the correspondence is determined according to camera parameters of the camera at the current position, the camera parameters of the camera at the current position are determined according to a preset position image of the camera at a preset position, the camera parameters at the preset position, and the current position image; Among them, the camera parameters of the camera at the current position are determined based on the preset position image of the camera at the preset position, the camera parameters at the preset position, and the current position image, including: determining the initial parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; back-projecting the second feature point in the preset position image into the camera coordinate system to obtain the first coordinate of the second feature point in the preset position image in the camera coordinate system; forward projecting the first coordinate onto the image plane of the current position to obtain the third feature point on the image plane of the current position; optimizing the initial parameters by minimizing the reprojection error of the third feature point relative to the first feature point in the current position image to obtain the camera parameters of the camera at the current position.

10. The method according to claim 9, characterized in that Also includes: A target object in the three-dimensional image is identified, and the identified target object is highlighted on the display interface.

11. An image processing device, characterized in that: include: A first acquisition module, used to acquire a preset position image of the camera at the preset position, and camera parameters at the preset position; A first acquisition module, used for acquiring a current position image of the camera at a current position; A first determination module, configured to determine the camera parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; Among them, the first determination module is also used to determine the initial parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; back-project the second feature point in the preset position image into the camera coordinate system to obtain the first coordinate of the second feature point in the preset position image in the camera coordinate system; forward-project the first coordinate onto the image plane of the current position to obtain the third feature point on the image plane of the current position; optimize the initial parameters by minimizing the reprojection error of the third feature point relative to the first feature point in the current position image to obtain the camera parameters of the camera at the current position.

12. An image processing device, characterized in that: include: A first display module, used to display a current position image of the camera at a current position on a display interface; A first receiving module, configured to receive an input operation, wherein the input operation is used to request display of a three-dimensional image corresponding to the current position image; A second display module, configured to respond to the input operation and display a three-dimensional picture corresponding to the current position image on the display interface, wherein the three-dimensional picture is determined based on a correspondence between coordinates in the current position image and three-dimensional space coordinates, the correspondence is determined based on camera parameters of the camera at the current position, the camera parameters of the camera at the current position are determined based on a preset position image of the camera at a preset position, the camera parameters at the preset position, and the current position image; Among them, the second display module is also used to determine the initial parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; back-project the second feature point in the preset position image into the camera coordinate system to obtain the first coordinate of the second feature point in the preset position image in the camera coordinate system; forward-project the first coordinate onto the image plane of the current position to obtain the third feature point on the image plane of the current position; optimize the initial parameters by minimizing the reprojection error of the third feature point relative to the first feature point in the current position image to obtain the camera parameters of the camera at the current position.

13. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the image processing method according to any one of claims 1 to 10.

14. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the image processing method according to any one of claims 1 to 10.

15. An image processing method, characterized in that: include: receiving a current position image of a camera at a current position sent by a client device; Feedback the camera parameters of the camera at the current position to the client device, wherein the camera parameters of the camera at the current position are determined according to a preset position image of the camera at the preset position, the camera parameters at the preset position, and the current position image; Wherein, the camera parameters of the camera at the current position are determined according to a preset position image of the camera at the preset position, the camera parameters at the preset position and the current position image, including: determining the initial parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; back-projecting the second feature point in the preset position image into the camera coordinate system to obtain the first coordinate of the second feature point in the preset position image in the camera coordinate system; forward projecting the first coordinate onto the image plane of the current position to obtain the third feature point on the image plane of the current position; optimizing the initial parameters by minimizing the reprojection error of the third feature point relative to the first feature point in the current position image to obtain the camera parameters of the camera at the current position.

16. An image processing method, characterized in that: include: Determining the camera parameters of the camera at the current position according to a preset position image of the camera at the preset position, the camera parameters at the preset position, and a first current position image of the camera at the current position; Wherein, determining the camera parameters of the camera at the current position according to the preset position image of the camera at the preset position, the camera parameters at the preset position and the first current position image of the camera at the current position includes: determining the initial parameters of the camera at the current position according to the preset position image, the current position image and the camera parameters at the preset position; back-projecting the second feature point in the preset position image into the camera coordinate system to obtain the first coordinate of the second feature point in the preset position image in the camera coordinate system; forward-projecting the first coordinate onto the image plane of the current position to obtain the third feature point on the image plane of the current position; optimizing the initial parameters by minimizing the reprojection error of the third feature point relative to the first feature point in the current position image to obtain the camera parameters of the camera at the current position; Acquire a second current position image of the camera at the current position; Correcting the second current position image using the camera parameters to obtain a corrected image; When the current position is any one of a plurality of focal positions of the camera, obtaining corrected images respectively corresponding to the plurality of focal positions; A three-dimensional graphic is generated according to the corrected images respectively corresponding to the multiple focal length positions, and the generated three-dimensional graphic is displayed at a predetermined frame rate to show a virtual reality scene.

Images