Image conversion method, device, computer readable storage medium and electronic device

Abstract

The disclosed embodiments disclose an image conversion method, device, computer-readable storage medium and electronic device, wherein the method includes: determining the intrinsic parameters and distortion parameters of a camera used to capture a first image; based on the intrinsic parameters and distortion parameters, performing distortion correction processing on the first image to obtain a second image; in the three-dimensional local coordinate system of the camera, based on the intrinsic parameters of the camera and a preset camera rotation angle, converting the second image into a third image; based on the intrinsic parameters and distortion parameters, performing distortion restoration processing on the third image to obtain a fourth image. The disclosed embodiments can perform distortion operations on an image after the camera shooting angle is transformed, so that the image quality of the image after the camera shooting angle is transformed is consistent with that of the original image, the implementation method is simple, the efficiency of generating images is high, and the generated images are closer to real scenes.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image conversion method, device, computer-readable storage medium, and electronic device. Background Art

[0002] In the field of computer vision, many application scenarios require images from multiple human eye angles. However, due to practical conditions, it is sometimes impossible to use a camera to shoot at various angles one by one. Therefore, it is necessary to convert the camera shooting angle based on the existing images so that images with different angles can be obtained.

[0003] For example, the line of sight estimation algorithm based on deep learning requires a large amount of line of sight data for training. Line of sight data usually consists of two parts: the image and the line of sight angle corresponding to the image. In order to obtain relatively good prediction results for different line of sight angle (pitch / yaw) data, the line of sight data involved in model training must be able to cover most line of sight angle areas. However, in the actual line of sight data collection process, the collection cost (including manpower / time / material resources, etc.) is relatively expensive, and it is difficult to cover most line of sight areas, so there will be empty areas. How to effectively use existing data to generate more realistic line of sight data to supplement the line of sight empty areas is a very worthwhile issue to explore. Summary of the invention

[0004] Embodiments of the present disclosure provide an image conversion method, an apparatus, a computer-readable storage medium, and an electronic device.

[0005] An embodiment of the present disclosure provides an image conversion method, which includes: determining the intrinsic parameters and distortion parameters of a camera used to acquire a first image; performing distortion correction processing on the first image based on the intrinsic parameters and the distortion parameters to obtain a second image; in a three-dimensional local coordinate system of the camera, based on the intrinsic parameters of the camera and a preset camera rotation angle, converting the second image into a third image; based on the intrinsic parameters and the distortion parameters, performing distortion restoration processing on the third image to obtain a fourth image.

[0006] According to another aspect of an embodiment of the present disclosure, an image conversion device is provided, which includes: a first determination module, used to determine the intrinsic parameters and distortion parameters of a camera used to acquire a first image; a correction module, used to perform distortion correction processing on the first image based on the intrinsic parameters and the distortion parameters to obtain a second image; a first conversion module, used to convert the second image into a third image based on the intrinsic parameters of the camera and a preset camera rotation angle in a three-dimensional local coordinate system of the camera; and a restoration module, used to perform distortion restoration processing on the third image based on the intrinsic parameters and the distortion parameters to obtain a fourth image.

[0007] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to execute the above-mentioned image conversion method.

[0008] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing processor executable instructions; the processor, configured to read the executable instructions from the memory and execute the instructions to implement the above-mentioned image conversion method.

[0009] Based on the image conversion method, device, computer-readable storage medium and electronic device provided by the above embodiments of the present disclosure, by determining the intrinsic parameters and distortion parameters of the camera used to collect the first image, based on the intrinsic parameters and distortion parameters, the first image is subjected to distortion correction processing to obtain a second image, and in the three-dimensional local coordinate system of the camera, the second image is converted into a third image based on the intrinsic parameters of the camera and the preset camera rotation angle; based on the intrinsic parameters and distortion parameters, the third image is subjected to distortion restoration processing to obtain a fourth image, so that the image is subjected to distortion correction before the camera shooting angle is transformed, and the influence of image distortion on the camera shooting angle transformation is reduced, and the image is subjected to distortion operation again after the camera shooting angle is transformed, so that the imaging quality of the image after the camera shooting angle transformation is consistent with that of the original image. The implementation method of the embodiments of the present disclosure is simple, and only the intrinsic parameters and distortion parameters of the camera are needed to generate images simulated by the same camera at different camera shooting angles, and the efficiency of generating images is high. Since the pixels of the generated image are transformed from the pixels of the original image, the generated image is closer to the real scene.

[0010] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 is a system diagram to which the present disclosure is applicable.

[0013] Figure 2 It is a flowchart of an image conversion method provided by an exemplary embodiment of the present disclosure.

[0014] Figure 3 It is a flowchart of an image conversion method provided by another exemplary embodiment of the present disclosure.

[0015] Figure 4 It is a flowchart of an image conversion method provided by another exemplary embodiment of the present disclosure.

[0016] Figure 5 It is a schematic diagram of the structure of an image conversion device provided by an exemplary embodiment of the present disclosure.

[0017] Figure 6 It is a structural schematic diagram of an image conversion device provided by another exemplary embodiment of the present disclosure.

[0018] Figure 7 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0020] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0021] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0022] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0023] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0024] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0025] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0026] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0027] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0028] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0029] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0030] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0031] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0032] Application Overview

[0033] In order to generate images at different sight angles or camera shooting angles, the existing technology generally adopts the following two solutions:

[0034] (1) Data augmentation is performed in 2D space to fill the hole area, such as rotation, mirroring, etc.

[0035] (2) Synthesize the line of sight data of the hole area based on deep learning methods.

[0036] Solution (1) is based on 2D information only. The transformation process may not conform to the optical imaging principle and may cause image distortion. The hole filling is not targeted. It may cause a large difference from the actual application scene (large angle rotation will cause a large difference from the actual scene).

[0037] Solution (2) is greatly affected by the accuracy of the model, and the pixel values ​​in the image are synthesized through the model, so the image is easily distorted.

[0038] Exemplary Systems

[0039] Figure 1 An exemplary system architecture 100 is shown to which the image conversion method or the image conversion apparatus according to the embodiments of the present disclosure can be applied.

[0040] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0041] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as image processing applications, human-computer interaction applications, search applications, web browser applications, instant messaging tools, etc.

[0042] The terminal device 101 can be various electronic devices, including but not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0043] The server 103 may be a server that provides various services, such as a background image processing server that performs camera angle conversion on the image uploaded by the terminal device 101. The background image processing server may process the received image to obtain a processing result (such as an image after camera angle conversion).

[0044] It should be noted that the image conversion method provided in the embodiment of the present disclosure can be executed by the server 103 or by the terminal device 101. Accordingly, the image conversion device can be set in the server 103 or in the terminal device 101.

[0045] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided as required. In the case where the image used for camera shooting angle conversion does not need to be acquired remotely, the above system architecture may not include a network, but only include a server or a terminal device.

[0046] Exemplary Methods

[0047] Figure 2 FIG. 1 is a flow chart of an image conversion method provided by an exemplary embodiment of the present disclosure. Figure 1 On the terminal device 101 or server 103 shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0048] Step 201: Determine the intrinsic parameters and distortion parameters of a camera used to capture a first image.

[0049] In this embodiment, the electronic device may determine the intrinsic parameters and distortion parameters of the camera used to acquire the first image. The method for determining the intrinsic parameters of the camera may be an existing method, such as the Zhang Zhengyou calibration method.

[0050] In general, the camera imaging formula is:

[0051]

[0052] in is the coordinate of the camera in the 3D local coordinate system, Represents the coordinates of the image in the two-dimensional coordinate system, and K represents the camera intrinsic parameter.

[0053] Camera distortion includes radial distortion and tangential distortion. The integrated distortion formula is:

[0054] u d =u * (1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 1 u * v * +p 2 (r 2 +2u *2 ) (2)

[0055] v d =v * (1+k 1 r 2 +k 2 r 4 +k3 r 6 )+p 1 (r 2 +2v *2 )+2p 2 u * v * (3)

[0056] in, r 2 =u *2 +v *2 ,(k 1 ,k 2 ,p 1 ,p 2 ,k 3 ) is the distortion parameter of the camera. Represents the coordinates of the distorted image in a two-dimensional coordinate system, Represents the coordinates of the undistorted image in the two-dimensional coordinate system. Equations (2) and (3) are the coordinate transformation formulas for the image from distorted to undistorted.

[0057] Step 202: Based on the intrinsic parameters and the distortion parameters, distortion correction processing is performed on the first image to obtain a second image.

[0058] In this embodiment, the electronic device can perform distortion correction processing on the first image based on the intrinsic parameter and the distortion parameter to obtain the second image.

[0059] Specifically, the electronic device can use the above equations (2) and (3) to perform inverse transformation to perform distortion correction processing on the first image.

[0060] Step 203 : In the three-dimensional local coordinate system of the camera, based on the intrinsic parameters of the camera and a preset camera rotation angle, the second image is converted into a third image.

[0061] In this embodiment, the electronic device can convert the second image into a third image in the three-dimensional local coordinate system of the camera based on the intrinsic parameters of the camera and the preset camera rotation angle. The camera rotation angle can be a pre-specified angle. Generally, the camera rotation angle can include a pitch angle, a yaw angle, and a roll angle. The camera rotation angle is a change in the camera shooting angle based on the camera shooting angle of the second image.

[0062] Usually, based on the intrinsic parameters of the camera, the pixels in the second image can be converted to the three-dimensional local coordinate system of the camera, and an affine transformation is performed in the three-dimensional local coordinate system to complete the rotation of the pixels, and then mapped back to the two-dimensional coordinate system to obtain the third image.

[0063] Step 204: Based on the internal parameters and the distortion parameters, the third image is subjected to distortion restoration processing to obtain a fourth image.

[0064] In this embodiment, the electronic device can perform distortion restoration processing on the third image based on the intrinsic parameter and the distortion parameter to obtain the fourth image. Specifically, the electronic device can perform distortion restoration processing on the third image based on the above formula (2) and formula (3).

[0065] The method provided by the above-mentioned embodiment of the present disclosure determines the intrinsic parameters and distortion parameters of the camera used to capture the first image, performs distortion correction processing on the first image based on the intrinsic parameters and distortion parameters to obtain a second image, and converts the second image into a third image based on the intrinsic parameters of the camera and a preset camera rotation angle in the three-dimensional local coordinate system of the camera; based on the intrinsic parameters and distortion parameters, performs distortion restoration processing on the third image to obtain a fourth image, thereby achieving distortion correction of the image before the camera shooting angle is changed, reducing the influence of image distortion on the camera shooting angle change, and performing distortion operation on the image after the camera shooting angle is changed, so that the image quality of the image after the camera shooting angle is changed is consistent with that of the original image. The implementation method of the embodiment of the present disclosure is simple, and only the intrinsic parameters and distortion parameters of the camera are needed to generate images simulated by the same camera at different camera shooting angles, and the efficiency of generating images is high. Since the pixels of the generated image are converted from the pixels of the original image, the generated image is closer to the real scene.

[0066] In some optional implementations, the electronic device may further perform the following steps:

[0067] Step 1: determine the sight angle of the first image in the three-dimensional local coordinate system of the camera.

[0068] The sight line angle is used to characterize the rotation angle of the sight line of the human eye (i.e., the line between the human eye and the viewpoint) relative to the origin of the three-dimensional local coordinate system in the three-dimensional local coordinate system of the camera. The sight line angle of the first image can be a label corresponding to the first image. Usually, the sight line angle of the first image is known, and the viewpoint of the human eye can be set in advance when shooting the face to determine the sight line angle, or the sight line angle can be estimated by using the existing sight line angle estimation method for the image taken by the camera.

[0069] Step 2: based on the camera rotation angle, convert the sight angle of the first image to the target sight angle.

[0070] As an example, the sight angle of the first image is (pitch0, yaw0, roll0). According to the camera rotation angle, the sight angle in the three-dimensional local coordinate system of the rotated camera can be calculated to be (pitch1, yaw1, roll1), that is, the target sight angle is (pitch1, yaw1, roll1).

[0071] Step three, determining the target sight angle as the sight angle of the fourth image.

[0072] Here, the target sight angle is the label corresponding to the fourth image.

[0073] This implementation method sets a sight angle label for the fourth image by determining the sight angle of the fourth image, so that the fourth image and the corresponding sight angle label can be applied to the training of the sight estimation model. It realizes the generation of training sample images at different camera shooting angles based on existing images, without using an actual camera to collect images at the corresponding camera shooting angles, thereby improving the efficiency of model training.

[0074] In some optional implementations, the above step 2 may be performed as follows:

[0075] First, the sight line angle of the first image is vectorized in a three-dimensional local coordinate system to obtain a first vector.

[0076] The vectorization process is to convert the camera shooting angle into a vector in the above three-dimensional local coordinate system. As an example, assuming that the camera shooting angle of the first image is (pitch, yaw), the first vector (x, y, z) is obtained based on the following formula:

[0077] x=cos(pitch)*sin(yaw)

[0078] y=sin(pitch) (4)

[0079] z=-cos(pitch)*cos(yaw)

[0080] It should be noted that when performing vectorization processing here, the line of sight angle may not include roll, because on the basis of determining the coordinate system and the sequence relationship, the spatial vector (x, y, z) can be obtained through pitch and yaw, and the roll angle can be deduced through the spatial vector.

[0081] Then, based on the camera rotation angle, an affine transformation is performed on the first vector to obtain a second vector.

[0082] Specifically, the affine transformation formula is as follows:

[0083]

[0084] in,

[0085]

[0086]

[0087]

[0088] R x represents the rotation matrix around the x-axis, R y Represents the rotation matrix around the y-axis, R z Represents the rotation matrix around the z-axis. The pitch, yaw, and roll in the rotation matrix are the camera rotation angles.

[0089] Based on the above formula (5), we can get the second vector (x r ,y r , z r ).

[0090] Finally, the second vector is converted to the target line of sight angle.

[0091] Specifically, the second vector may be converted into the target sight angle using a vector-to-Euler angle formula.

[0092] Continuing with the above example, the second vector can be converted to the target sight angle (pitch', yaw') based on the following formula:

[0093] pitch'=arcsin(y)

[0094]

[0095] It should be noted that the above formula only includes pitch and yaw because in general, the sight angle can be expressed by pitch and yaw, and roll is generally not used. However, roll can be calculated based on pitch and yaw. The conversion process between vectors and Euler angles is a well-known technology and will not be repeated here.

[0096] This implementation method vectorizes the sight angle of the first image, and then performs affine transformation on the vector to finally obtain the target sight angle, so that the sight angle can be accurately rotated in the three-dimensional local coordinate system of the above-mentioned camera to obtain the accurate target sight angle.

[0097] Further references Figure 3 , shows a flow chart of another embodiment of the image conversion method. Figure 3 As shown in the above Figure 2 Based on the illustrated embodiment, step 203 may include the following steps:

[0098] Step 2031: extract a first pixel point set including a preset number of points from a first image.

[0099] The preset number can be set arbitrarily, for example, the preset number is 4. The electronic device can extract the first pixel point set based on various methods, such as random extraction, specified extraction, etc.

[0100] Optionally, the electronic device may extract a first pixel point set including a preset number of non-collinear points from the first image. The first non-collinear pixel point set may be the first pixel point set that fully characterizes the two-dimensional features of the image, thereby improving the accuracy of subsequent affine transformation.

[0101] Step 2032: based on the intrinsic parameters and the distortion parameters, convert the first pixel point set into a second pixel point set.

[0102] Specifically, the electronic device can transform the coordinates of each point in the first pixel point set based on the above formula (2) and formula (3) to obtain the second pixel point set.

[0103] Step 2033: In the two-dimensional image plane coordinate system of the first image, based on the intrinsic parameters and the camera rotation angle, the second pixel point set is converted into a third pixel point set.

[0104] Typically, the electronic device can convert the second pixel point set to the three-dimensional local coordinate system of the camera based on the camera's intrinsic parameters, perform affine transformation in the three-dimensional local coordinate system to complete the rotation of the pixel points, and then map it back to the two-dimensional coordinate system to obtain a third pixel point set.

[0105] Step 2034, determining the coordinate mapping relationship between the second pixel point set and the third pixel point set in the two-dimensional image plane coordinate system.

[0106] Specifically, the above mapping relationship can be obtained based on the following formula:

[0107]

[0108] in, represents the two-dimensional coordinates of the third pixel point set, represents the two-dimensional coordinates of the second pixel set, K is the camera intrinsic parameter, R z , R y , R x is the rotation matrix shown in equation (5). Let f = K·R z ·R y ·R x ·K -1 , then f represents the coordinate mapping relationship.

[0109] Step 2035: convert the second image into a third image according to the coordinate mapping relationship.

[0110] Specifically, based on the above f, the coordinates of each pixel in the second image can be transformed to finally obtain the third image.

[0111] Above Figure 3 The method provided in the corresponding embodiment extracts a first pixel point set from the first image, obtains a third pixel point set based on the intrinsic parameters and the camera rotation angle, and thus obtains the coordinate mapping relationship of the pixel point set, thereby realizing the use of fewer pixel points to determine the coordinate mapping relationship, which helps to reduce the amount of calculation and improve the efficiency of image conversion.

[0112] In some optional implementations, such as Figure 4 As shown, step 2033 may include the following steps:

[0113] Step 20331: Based on the internal parameters, convert the second pixel point set into a first spatial point set in the three-dimensional local coordinate system.

[0114] Specifically, from the above formula (1), the conversion formula from the two-dimensional coordinate system of the image to the three-dimensional local coordinate system of the camera can be obtained as follows:

[0115]

[0116] Based on formula (8), the second pixel point set can be converted into the first spatial point set in the three-dimensional local coordinate system.

[0117] Step 20332: Based on the camera rotation angle, perform affine transformation on the coordinates of the first spatial point set to obtain a second spatial point set.

[0118] Specifically, the coordinates of the first space point set may be affine transformed according to the above formula (5).

[0119] Step 20333: Based on the internal parameters, the second spatial point set is converted into a third pixel point set in the two-dimensional image plane coordinate system.

[0120] Specifically, based on formula (1), the second spatial point set in the three-dimensional local coordinate system of the camera can be converted into a third pixel point set in the two-dimensional image plane coordinate system.

[0121] This implementation method converts the second pixel point set into a three-dimensional local coordinate system for affine transformation, and then converts it back to a two-dimensional coordinate system, so that the camera shooting angle of the second pixel point set can be accurately converted, which helps to improve the accuracy of determining the coordinate mapping relationship between the second image and the third image.

[0122] Exemplary Devices

[0123] Figure 5 FIG. 1 is a schematic diagram of the structure of an image conversion device provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 5 As shown, the image conversion device includes: a first determination module 501, used to determine the intrinsic parameters and distortion parameters of the camera used to capture the first image; a correction module 502, used to perform distortion correction processing on the first image based on the intrinsic parameters and the distortion parameters to obtain a second image; a first conversion module 503, used to convert the second image into a third image based on the intrinsic parameters of the camera and a preset camera rotation angle in a three-dimensional local coordinate system of the camera; and a restoration module 504, used to perform distortion restoration processing on the third image based on the intrinsic parameters and the distortion parameters to obtain a fourth image.

[0124] In this embodiment, the first determination module 501 may determine the intrinsic parameters and distortion parameters of the camera used to acquire the first image. The method for determining the intrinsic parameters of the camera may be an existing method, such as the Zhang Zhengyou calibration method.

[0125] In this embodiment, the correction module 502 may perform distortion correction processing on the first image based on the intrinsic parameter and the distortion parameter to obtain the second image.

[0126] Specifically, the electronic device can use the above equations (2) and (3) to perform inverse transformation to perform distortion correction processing on the first image.

[0127] In this embodiment, the first conversion module 503 can convert the second image into a third image based on the camera's internal parameters and a preset camera rotation angle in the camera's three-dimensional local coordinate system. The camera rotation angle can be a pre-specified angle. Generally, the camera rotation angle can include a pitch angle, a yaw angle, and a roll angle. The camera rotation angle is a change in the camera shooting angle based on the camera shooting angle of the second image.

[0128] Usually, based on the intrinsic parameters of the camera, the pixels in the second image can be converted to the three-dimensional local coordinate system of the camera, and an affine transformation is performed in the three-dimensional local coordinate system to complete the rotation of the pixels, and then mapped back to the two-dimensional coordinate system to obtain the third image.

[0129] In this embodiment, the restoration module 504 can perform distortion restoration processing on the third image based on the internal parameters and the distortion parameters to obtain the fourth image. Specifically, the electronic device can implement distortion restoration processing on the third image based on the above formula (2) and formula (3).

[0130] Reference Figure 6 , Figure 6 It is a structural schematic diagram of an image conversion device provided by another exemplary embodiment of the present disclosure.

[0131] In some optional implementations, the first conversion module 503 includes: an extraction unit 5031, used to extract a first pixel point set containing a preset number of points from the first image; a first conversion unit 5032, used to convert the first pixel point set into a second pixel point set based on a distortion parameter; a second conversion unit 5033, used to convert the second pixel point set into a third pixel point set based on an intrinsic parameter and a camera rotation angle in a two-dimensional image plane coordinate system of the first image; a determination unit 5034, used to determine a coordinate mapping relationship between the second pixel point set and the third pixel point set in the two-dimensional image plane coordinate system; and a third conversion unit 5035, used to convert the second image into a third image based on the coordinate mapping relationship.

[0132] In some optional implementations, the second conversion unit 5033 includes: a first conversion subunit 50331, used to convert the second pixel point set into a first space point set in a three-dimensional local coordinate system based on an intrinsic parameter; a second conversion subunit 50332, used to perform an affine transformation on the coordinates of the first space point set based on the camera rotation angle to obtain a second space point set; and a third conversion subunit 50333, used to convert the second space point set into a third pixel point set in a two-dimensional image plane coordinate system based on the intrinsic parameter.

[0133] In some optional implementations, the extraction unit 5031 is further used to: extract a first pixel point set including a preset number of non-collinear points from the first image.

[0134] In some optional implementations, the device also includes: a second determination module 505, used to determine the line of sight angle of the first image in the three-dimensional local coordinate system of the camera; a second conversion module 506, used to convert the line of sight angle of the first image to a target line of sight angle based on the camera rotation angle; and a third determination module 507, used to determine the target line of sight angle as the line of sight angle of the fourth image.

[0135] In some optional implementations, the second conversion module 506 includes: a processing unit 5061, used to vectorize the line of sight angle of the first image in a three-dimensional local coordinate system to obtain a first vector; a fourth conversion unit 5062, used to perform affine transformation on the first vector based on the camera rotation angle to obtain a second vector; and a fifth conversion unit 5063, used to convert the second vector into a target line of sight angle.

[0136] The image conversion device provided by the above-mentioned embodiment of the present disclosure determines the intrinsic parameters and distortion parameters of the camera used to collect the first image, performs distortion correction processing on the first image based on the intrinsic parameters and distortion parameters to obtain a second image, and converts the second image into a third image based on the intrinsic parameters of the camera and a preset camera rotation angle in the three-dimensional local coordinate system of the camera; based on the intrinsic parameters and distortion parameters, performs distortion restoration processing on the third image to obtain a fourth image, thereby realizing distortion correction of the image before the camera shooting angle is changed, reducing the influence of image distortion on the camera shooting angle change, and performing distortion operation on the image after the camera shooting angle is changed, so that the image quality of the image after the camera shooting angle is changed is consistent with that of the original image. The implementation method of the embodiment of the present disclosure is simple, and only the intrinsic parameters and distortion parameters of the camera are needed to generate images simulated by the same camera at different camera shooting angles, and the efficiency of generating images is high. Since the pixels of the generated image are converted from the pixels of the original image, the generated image is closer to the real scene.

[0137] Exemplary Electronic Devices

[0138] Below, reference Figure 7 To describe an electronic device according to an embodiment of the present disclosure. The electronic device may be as follows Figure 1 Any one or both of the terminal device 101 and the server 103 shown, or a stand-alone device independent of them, the stand-alone device can communicate with the terminal device 101 and the server 103 to receive the collected input signals from them.

[0139] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0140] like Figure 7 As shown, the electronic device 700 includes one or more processors 701 and a memory 702 .

[0141] The processor 701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 700 to perform desired functions.

[0142] The memory 702 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 701 may run the program instructions to implement the image conversion method of each embodiment of the present disclosure above and / or other desired functions. Various contents such as the first image, the fourth image, etc. may also be stored in the computer-readable storage medium.

[0143] In one example, the electronic device 700 may further include: an input device 703 and an output device 704, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0144] For example, when the electronic device is a terminal device 101 or a server 103, the input device 703 may be a camera, a mouse, a keyboard, etc., for inputting images. When the electronic device is a stand-alone device, the input device 703 may be a communication network connector for receiving input images from the terminal device 101 and the server 103.

[0145] The output device 704 can output various information to the outside, including the fourth image after the camera shooting angle is converted. The output device 704 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0146] Of course, to simplify, Figure 7 Only some of the components related to the present disclosure in the electronic device 700 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 700 may also include any other appropriate components.

[0147] Exemplary computer program products and computer-readable storage media

[0148] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the image conversion method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0149] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0150] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the image conversion method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0151] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0152] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0153] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0154] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0155] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0156] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0157] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0158] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An image conversion method, comprising: Determining intrinsic parameters and distortion parameters of a camera used to acquire the first image; Based on the intrinsic reference and the distortion parameter, performing distortion correction processing on the first image to obtain a second image; In the three-dimensional local coordinate system of the camera, based on the intrinsic parameters of the camera and a preset camera rotation angle, converting the second image into a third image; Based on the intrinsic reference and the distortion parameter, performing distortion restoration processing on the third image to obtain a fourth image; Determine a sight angle of the first image in a three-dimensional local coordinate system of the camera; Based on the camera rotation angle, converting the sight angle of the first image to a target sight angle; The target sight line angle is determined as the sight line angle of the fourth image.

2. The method according to claim 1, wherein: The converting the second image into a third image based on the intrinsic parameter of the camera and a preset camera rotation angle includes: Extracting a first pixel point set including a preset number of points from the first image; Based on the intrinsic parameter and the distortion parameter, converting the first pixel point set into a second pixel point set; In the two-dimensional image plane coordinate system of the first image, based on the intrinsic parameter and the camera rotation angle, the second pixel point set is converted into a third pixel point set; Determine a coordinate mapping relationship between the second pixel point set and the third pixel point set in the two-dimensional image plane coordinate system; The second image is converted into the third image according to the coordinate mapping relationship.

3. The method according to claim 2, wherein: The converting the second pixel point set into a third pixel point set based on the intrinsic parameter and the camera rotation angle includes: Based on the intrinsic parameter, converting the second pixel point set into a first spatial point set in the three-dimensional local coordinate system; Based on the camera rotation angle, performing an affine transformation on the coordinates of the first spatial point set to obtain a second spatial point set; Based on the intrinsic parameter, the second spatial point set is converted into a third pixel point set in the two-dimensional image plane coordinate system.

4. The method according to claim 2, wherein: The step of extracting a first pixel point set including a preset number of points from the first image comprises: A first pixel point set including a preset number of non-collinear points is extracted from the first image.

5. The method according to claim 1, wherein: The converting the sight angle of the first image to the target sight angle based on the camera rotation angle includes: Performing vector processing on the sight line angle of the first image in the three-dimensional local coordinate system to obtain a first vector; Based on the camera rotation angle, perform an affine transformation on the first vector to obtain a second vector; The second vector is converted to the target sight angle.

6. An image conversion device, comprising: A first determination module, used to determine an intrinsic parameter and a distortion parameter of a camera used to acquire a first image; A correction module, configured to perform distortion correction processing on the first image based on the intrinsic reference and the distortion parameter to obtain a second image; A first conversion module, configured to convert the second image into a third image based on an intrinsic parameter of the camera and a preset camera rotation angle in a three-dimensional local coordinate system of the camera; A restoration module, configured to perform distortion restoration processing on the third image based on the internal reference and the distortion parameter to obtain a fourth image; A second determination module is used to determine the sight angle of the first image in the three-dimensional local coordinate system of the camera; A second conversion module, configured to convert the sight angle of the first image to a target sight angle based on the camera rotation angle; The third determining module is used to determine the target sight line angle as the sight line angle of the fourth image.

7. The device according to claim 6, wherein: The first conversion module comprises: An extraction unit, configured to extract a first pixel point set including a preset number of points from the first image; A first conversion unit, configured to convert the first pixel point set into a second pixel point set based on the distortion parameter; A second conversion unit, configured to convert the second pixel point set into a third pixel point set based on the intrinsic parameter and the camera rotation angle in a two-dimensional image plane coordinate system of the first image; A determination unit, configured to determine a coordinate mapping relationship between the second pixel point set and the third pixel point set in the two-dimensional image plane coordinate system; A third conversion unit is used to convert the second image into the third image according to the coordinate mapping relationship.

8. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 5.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 5.

Images