Image processing method and apparatus for free-viewpoint camera

Abstract

Disclosed in the present invention is an image processing method for a free-viewpoint camera. The method comprises: acquiring camera intrinsic parameters, camera extrinsic parameters and ideal camera pose parameters of different cameras, wherein the camera intrinsic parameters at least comprise the focal lengths f of the cameras; on the basis of the camera extrinsic parameters and the ideal camera pose parameters, determining camera pose adjustment parameters, and on the basis of the focal lengths f of the different cameras and an average focal length formula (1), determining scaling adjustment parameters; and on the basis of the scaling adjustment parameters of the cameras, performing scaling correction on corresponding images, and on the basis of the pose adjustment parameters of the cameras, performing pose correction on the corresponding images.

Description

Image processing method and device for free-viewing-angle camera Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and device for a free-viewing-angle camera. Background Art

[0002] A free-viewpoint camera system consists of several cameras with synchronized recording start and end times, as well as the moment of capture. Viewers can switch camera positions at any time while watching free-viewpoint videos, achieving a seamless viewing experience. Because free-viewpoint system cameras cannot be positioned strictly according to a standard, the relative position or angle of the same object in the images of adjacent cameras can vary significantly. This can cause image jitter when switching camera positions, affecting the imaging quality and reducing the viewing experience.

[0003] Summary of the Invention

[0004] The main purpose of the present invention is to provide an image processing method and device for a free-viewpoint camera, aiming to solve the problem that the existing free-viewpoint camera system generates jitter when switching viewing positions, affecting the viewing experience.

[0005] To achieve the above object, the present invention provides an image processing method, comprising:

[0006] Obtain camera intrinsic parameters, camera extrinsic parameters, and ideal camera posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera;

[0007] The camera posture adjustment parameters are determined based on the ideal posture parameters of the camera outside the camera, and the focal length f of different cameras and the average focal length are determined based on the ideal posture parameters of the camera outside the camera. Determine scaling adjustment parameters;

[0008] The corresponding image is scaled and corrected based on the scale adjustment parameters of each camera, and the corresponding image is posture-corrected based on the posture adjustment parameters of each camera.

[0009] The image processing method and device provided by the present invention obtain camera intrinsic parameters, camera extrinsic parameters and ideal camera posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera; determine the camera posture adjustment parameters based on the camera extrinsic parameters and the ideal camera posture parameters, and calculate the camera posture adjustment parameters based on the focal length f of different cameras and the average focal length. Determine the zoom adjustment parameters; perform zoom correction on the corresponding image based on the zoom adjustment parameters of each camera, and perform image posture correction on the corresponding image based on the posture adjustment parameters of each camera. Compared with the existing technology, each camera is first calibrated to obtain camera intrinsic parameters and camera extrinsic parameters, and then the ideal posture of each camera is calculated through mathematical methods. Finally, the image correction algorithm is used to perform image processing such as rotation, translation, and cropping to obtain the image of each camera in the ideal posture. In the ideal posture, all cameras are looking at the same point in space, and the longitudinal axis Y axis of the camera is parallel to ensure the viewing effect when switching camera positions. This method ensures that the relative position of the object in the center of the picture remains unchanged in the pictures of different camera positions, thereby achieving a "shake-free" viewing experience when switching camera positions. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG1 is a schematic diagram of the system structure of the hardware operating environment involved in the embodiment of the present invention;

[0011] FIG2 is a flow chart of an embodiment of an image processing method according to the present invention;

[0012] FIG3 is a diagram illustrating an example of image comparison before image processing by different cameras in an embodiment of the image processing method of the present invention;

[0013] FIG4 is a diagram illustrating an example of image comparison after image processing by different cameras in an embodiment of the image processing method of the present invention. DETAILED DESCRIPTION

[0014] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0016] In the prior art, multi-terminal display systems are difficult to implement.

[0017] In order to solve the above technical problems, the present invention provides an image processing method. First, each camera is calibrated to obtain camera intrinsic parameters and camera extrinsic parameters. Then, the ideal posture of each camera is calculated by mathematical methods. Finally, image correction algorithms are used to perform rotation, translation, cropping and other image processing to obtain the picture of each camera in the ideal posture. In the ideal posture, all cameras look at the same point in space, and the longitudinal axis Y axis of the camera is parallel to ensure the viewing effect when switching camera positions. This method ensures that the relative position of the object in the center of the picture remains unchanged in the pictures of different camera positions, thereby achieving a "shake-free" viewing experience when switching camera positions.

[0018] As shown in FIG1 , FIG1 is a schematic diagram of the system structure of the hardware operating environment involved in the embodiment of the present invention.

[0019] The terminal of the embodiment of the present invention can be a terminal device with computing capabilities, or it can be a PC, or it can be a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) player, a portable computer, and other mobile terminal devices with display capabilities.

[0020] As shown in Figure 1, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0021] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal can also be configured with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0022] Those skilled in the art will understand that the terminal structure shown in FIG1 does not constitute a limitation on the terminal, and may include more or fewer components than shown, or a combination of certain components, or a different arrangement of components.

[0023] As shown in FIG1 , the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an image processing program.

[0024] In the terminal shown in FIG1 , the network interface 1004 is primarily used to connect to a backend server and perform image processing with the backend server; the user interface 1003 is primarily used to connect to a client (user end) and perform image processing with the client; and the processor 1001 can be used to call an image processing program stored in the memory 1005 and perform the following operations:

[0025] Obtain camera intrinsic parameters, camera extrinsic parameters, and ideal camera posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera;

[0026] The camera posture adjustment parameters are determined based on the ideal posture parameters of the camera outside the camera, and the focal length f of different cameras and the average focal length are determined based on the ideal posture parameters of the camera outside the camera. Determine scaling adjustment parameters;

[0027] The corresponding image is scaled and corrected based on the scale adjustment parameters of each camera, and the corresponding image is posture-corrected based on the posture adjustment parameters of each camera.

[0028] Furthermore, the processor 1001 may call the image processing program stored in the memory 1005 and perform the following operations:

[0029] Calculate the fundamental matrix F based on the matching feature points of different camera images;

[0030] The camera calibration parameters are calculated based on the basic matrix F and the spatial geometric relationship; wherein the camera calibration parameters include at least camera intrinsic parameters and camera extrinsic parameters, and the camera extrinsic parameters include at least rotation matrix R, camera center A n and the translation matrix t;

[0031] Based on the rotation matrix R and the camera center A n Determine the ideal camera orientation;

[0032] Based on the center coordinate A n Determine the baseline direction;

[0033] Determine ideal pose parameters of the camera based on the ideal camera orientation and the baseline direction.

[0034] Furthermore, the processor 1001 may call the image processing program stored in the memory 1005 and perform the following operations:

[0035] The real-time principal optical axis parameters of the camera are obtained based on the rotation matrix R

[0036] Determine the line connecting any spatial point and the camera center based on the preset angle formula The angle between the camera's real-time principal optical axis is θ n ; Wherein, the preset angle formula is:

[0037] The average value of the sum of all angles is calculated based on a preset average value calculation formula, wherein the preset average value calculation formula is:

[0038] based on The coordinates of the ideal alignment point X′ are determined by minimizing the value of

[0039] Based on the camera center point coordinates A of each camera n The direction of the line connecting the ideal alignment point X' Determine the ideal orientation for each camera.

[0040] Furthermore, the processor 1001 may call the image processing program stored in the memory 1005 and perform the following operations:

[0041] Get the camera center point coordinates A1, A2, A3, ..., A of all cameras n ;

[0042] The steps of obtaining the center point coordinates are executed cyclically until the coordinates of the last two points are obtained; wherein the cyclic steps are:

[0043] Calculate the coordinates of the camera center points of adjacent serial numbers in sequence: A n =(A n +A n+1 ) / 2, as the new spatial point coordinates;

[0044] The baseline direction is determined based on the obtained coordinates of the last two points.

[0045] Furthermore, the processor 1001 may call the image processing program stored in the memory 1005 and perform the following operations:

[0046] Determining calibration parameters of each camera based on the camera extrinsic parameters;

[0047] The attitude adjustment parameters of each camera are determined based on the difference between the calibrated attitude parameters of each camera and the ideal attitude parameters.

[0048] Furthermore, the processor 1001 may call the image processing program stored in the memory 1005 and perform the following operations:

[0049] Calculating the cross product of the ideal camera orientation and the baseline direction to determine the Y-axis parameter corresponding to the camera;

[0050] Determining an X-axis parameter of each camera based on the ideal camera orientation and the Y-axis parameter;

[0051] The ideal posture parameters of each camera are determined based on the X-axis parameters, the Y-axis parameters, and the ideal camera orientation.

[0052] Furthermore, the processor 1001 may call the image processing program stored in the memory 1005 and perform the following operations:

[0053] Calculate the average focal length based on the focal length f of each camera

[0054] Based on the focal length f of each camera and the average focal length The difference in values ​​determines the scaling adjustment parameter.

[0055] Furthermore, the processor 1001 may call the image processing program stored in the memory 1005 and perform the following operations:

[0056] Obtain real-time images taken by each camera at the same time;

[0057] Extracting feature points from each of the real-time images;

[0058] Feature point matching is performed based on the extracted feature points to obtain multiple groups of matching feature points.

[0059] Referring to FIG2 , FIG2 is a flow chart of an embodiment of an image processing method for a free-viewpoint camera of the present invention. Based on the algorithm of the present invention, each camera is first calibrated to obtain camera intrinsic parameters and camera extrinsic parameters. Then, the ideal posture of each camera is calculated by mathematical methods to ensure the viewing effect when switching camera positions. Finally, the image is rotated, translated, cropped, and other processes are performed on the image through the image correction algorithm to obtain the image of each camera in the ideal posture. This method ensures that the relative position of the object in the center of the image remains unchanged in the images of different camera positions, thereby achieving a "shake-free" viewing experience when switching camera positions. In some embodiments, the image processing method includes:

[0060] Step S10: Obtain camera intrinsic parameters, camera extrinsic parameters, and ideal camera posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera.

[0061] The image processing method of the present invention is primarily applicable to a free-viewpoint camera system, in which the number of cameras is two or more, with no specific limit. After the cameras are positioned, all cameras are first used to capture a single image simultaneously, obtaining real-time images captured by each camera at the same time. Feature points are then extracted from each real-time image, and the feature points of each real-time image are matched based on their characteristics. Correspondence between the feature points of each real-time image is established, resulting in multiple sets of feature points.

[0062] After obtaining multiple sets of feature points, the camera's intrinsic and extrinsic parameters can be obtained based on the multiple sets of feature points, thereby obtaining the camera's focal length f and the rotation matrix R in the extrinsic parameters. The camera's real-time principal optical axis parameters are then determined based on the rotation matrix. The camera's ideal orientation is then determined based on the camera's real-time principal optical axis parameters. The camera's ideal posture parameters are then determined based on the camera's ideal orientation. The camera's ideal posture parameters include X-axis parameters, Y-axis parameters, and Z-axis parameters.

[0063] Step S20, determining the camera posture adjustment parameters based on the ideal posture parameters of the camera outside the camera, and based on the focal length f of different cameras and the average focal length Determines the scaling parameters.

[0064] The camera's actual pose parameters can be determined based on its extrinsic parameters. The pose adjustment parameters for each camera can be determined based on the difference between the camera's actual pose parameters and its ideal pose parameters. The zoom adjustment parameters for each camera can be determined based on the difference between the focal length f of each camera and the average f of all cameras.

[0065] Step S30 : performing scaling correction on the corresponding image based on the scaling adjustment parameters of each camera, and performing image posture correction on the corresponding image based on the posture adjustment parameters of each camera.

[0066] Based on the posture adjustment parameters of each camera, the images captured by each camera are subjected to corresponding scaling correction. Based on the posture adjustment parameters of each camera, the images captured by each camera are subjected to translation, rotation and other processing (achieved by changing the projection relationship of the image) to achieve posture correction of the image. The image processing effect is shown in Figures 3 and 4, wherein Figure 3 is an example diagram of image comparison before image processing of different cameras in an embodiment of the image processing method of the present invention, and Figure 4 is an example diagram of image comparison after image processing of different cameras in an embodiment of the image processing method of the present invention. In some embodiments, the corrected image can also be cropped to obtain a corrected image without black edges.

[0067] In the above embodiment, the camera intrinsic parameters, camera extrinsic parameters and ideal camera posture parameters of different cameras are obtained, wherein the camera intrinsic parameters at least include the focal length f of the camera; the camera posture adjustment parameters are determined based on the camera extrinsic parameters and the ideal camera posture parameters, and the focal length f of different cameras is compared with the average focal length. Determine the zoom adjustment parameters; perform zoom correction on the corresponding images based on the zoom adjustment parameters of each camera, and perform posture correction on the corresponding images based on the posture adjustment parameters of each camera. Through the above method, each camera is first calibrated to obtain camera intrinsic parameters and camera extrinsic parameters, and then the ideal posture of each camera is calculated by mathematical methods. Finally, the image correction algorithm is used to perform image processing such as rotation, translation, and cropping to obtain the image of each camera in the ideal posture. In the ideal posture, all cameras look at the same point in space, and the longitudinal axis Y axis of the camera is parallel to ensure the viewing effect when switching camera positions. This method ensures that the relative position of the object in the center of the picture remains unchanged in the pictures of different camera positions, thereby achieving a "shake-free" viewing experience when switching camera positions.

[0068] In some embodiments, step S10 includes:

[0069] Step S11 : calculating a fundamental matrix F based on matching feature points of different camera images.

[0070] In some embodiments, before step S11, the method further includes:

[0071] Step S40, obtaining real-time images captured by each camera at the same time;

[0072] Step S50, extracting feature points from each of the real-time images;

[0073] Step S60: performing feature point matching based on the extracted feature points to obtain multiple groups of matching feature points.

[0074] After the cameras are positioned, all cameras are used to capture a single image at the same time, generating a real-time image. Feature points are then extracted from each real-time image. The feature points of each real-time image are then matched based on their characteristics, establishing a correspondence between the feature points of each real-time image to generate multiple sets of feature points.

[0075] For a set of matching feature points in two real-time images, the homogenized image coordinates are x and x′ respectively. According to the spatial geometric relationship, x′ can be obtained T Fx=0, the basic matrix F can be determined based on more than eight sets of matching feature points.

[0076] Step S12, calculate the camera calibration parameters based on the basic matrix F and the spatial geometric relationship; wherein the camera calibration parameters include at least the camera intrinsics and camera extrinsics, and the camera extrinsics include at least the rotation matrix R, the camera center An and the translation matrix t (where An = -R't, R' is the inverse matrix of R), wherein each column of the rotation matrix R can represent the orientation of the x, y, and z axes of the camera coordinate system in the world coordinate system. The translation matrix t is a 3x1 matrix, which represents the displacement of the camera center relative to the center of the world coordinate system. The method for calculating the camera intrinsics and camera extrinsics based on the basic matrix and the spatial geometric relationship can refer to the existing calculation method and will not be repeated here. The rotation matrix R is a 3x3 matrix. Assume that the rotation angle θ around the x, y, and z axes in space can be represented by the following matrix:

[0077] The rotation matrix R is R x , R y With R z The product of the three, that is, if you rotate around the x, y, z axis by angles Ψ, θ, Φ respectively, the rotation matrix is:

[0078] It represents the rotation of the camera coordinate system relative to the world coordinate system. Each column of the rotation matrix R represents the orientation of the x, y, and z axes of the camera coordinate system in the world coordinate system. The translation matrix t is a 3x1 matrix that represents the displacement of the camera center relative to the center of the world coordinate system. The above formulas yield the final calibration results: the camera intrinsic parameter matrix K, the camera extrinsic parameter matrix R, and t.

[0079] Step S13: Based on the rotation matrix R and the camera center A n Determine the ideal orientation of the camera.

[0080] Specifically, in some embodiments, step S13 includes:

[0081] Step S131, obtaining the real-time principal optical axis parameters of the camera based on the rotation matrix R

[0082] The main optical axis of the camera is a straight line passing through the centers of the two spherical surfaces of the thin lens. The real-time main optical axis parameters are Refers to the matrix parameters corresponding to the main optical axis of the camera in real-time placement state. Among them, the real-time main optical axis parameters of the camera are It can be obtained by converting the rotation matrix R, where the third column of the rotation matrix R corresponds to the direction of the main optical axis.

[0083] Step S132: Determine the line connecting any spatial point and the camera center based on the preset angle formula The angle between the camera's real-time principal optical axis is θ n ; Wherein, the preset angle formula is:

[0084] Step S133, based on the angle θ n The minimum average value of determines the coordinates of the ideal alignment point X′.

[0085] Specifically, suppose there is a point X in space, and the center of camera n is point A in space n (Among them, A n The camera position in the calibration information is obtained as described above, and the main optical axis is (obtained by converting the rotation matrix in the camera external parameters). Ideally, if all cameras align the imaging center point with X, then the camera's main optical axis and direction vector The angle between them is 0 (i.e., they coincide). In reality, suppose there is a point X′ in space, and the line connecting it and the camera center is With the camera's main optical axis The angle between them is θ n , according to the properties of vector product:

[0086] set up minimize The coordinate of X' can be obtained. This is the ideal orientation of each camera (ideal Z axis).

[0087] Step S134: based on the camera center point coordinates A of each camera n The direction of the line connecting the ideal alignment point X' Determine the ideal orientation for each camera.

[0088] Based on the above steps, after determining the coordinates of the ideal alignment point X′, the camera center coordinates A of each camera aren Calculate the coordinates A of each camera center point n The direction of the line connecting the ideal alignment point X' This determines the ideal orientation of each camera.

[0089] In the above image processing method, the real-time principal optical axis parameters of the camera are obtained based on the rotation matrix R.

[0090] Determine the line connecting any spatial point and the camera center based on the preset angle formula The angle between the camera's real-time principal optical axis is θ n , based on the angle θ n The minimum average value of the ideal alignment point X' is determined; based on the camera center point coordinates A of each camera n The direction of the line connecting the ideal alignment point X' Determine the ideal orientation of each camera. Through the above method, the ideal orientation of all cameras towards the spatial point can be accurately determined, so that subsequent image processing can be based on the ideal orientation to obtain better image viewing effects.

[0091] Step S14, based on the center coordinate A n Determine the baseline orientation.

[0092] Specifically, in some embodiments, step S40 includes:

[0093] Step S141, obtaining the camera center point coordinates A1, A2, A3, ..., A of all cameras n ;

[0094] Step S142, looping steps are executed repeatedly until the coordinates of the last two points are obtained; wherein the looping steps are:

[0095] Calculate the coordinates of the camera center points of adjacent serial numbers in sequence: A n =(A n +A n+1 ) / 2, as the new spatial point coordinates;

[0096] Step S143: determining the baseline direction based on the obtained coordinates of the last two points.

[0097] The spatial coordinates A1, A2, A3, ..., A of the center of each camera can be obtained by the calibration step. n , calculate in sequence A new set of spatial points can be obtained. Repeat the above steps several times until the last two points are obtained. The direction vector of the two points is calculated. The direction vector composed of the two points is the free view baseline direction.

[0098] Step S15: determining ideal posture parameters of the camera based on the ideal camera orientation and the baseline direction.

[0099] Specifically, in some embodiments, step S15 includes:

[0100] Step S151, calculating the cross product of the ideal camera orientation and the baseline direction to determine the Y-axis parameter corresponding to the camera;

[0101] Step S152, determining the X-axis parameters of each camera based on the ideal camera orientation and the Y-axis parameters;

[0102] Step S153: determining the ideal posture parameters of each camera based on the X-axis parameters, the Y-axis parameters, and the ideal camera orientation.

[0103] Based on the above steps, the ideal main orientation of the camera (the direction of the camera coordinate axis Z axis) is calculated, and then the baseline direction of the free perspective system is calculated. (Detailed explanation later), perform a vector product (cross product) on these two vectors You can get a perpendicular and The direction of the camera is the Y-axis parameter. Since the camera Z-axis and Y-axis parameters have been determined, the X-axis parameter is also determined, and the ideal camera posture is determined. Among them, the above X-axis parameters, Y-axis parameters, and Z-axis parameters are the direction vector parameters corresponding to each axis. The determined camera X-axis parameters, Y-axis parameters, and Z-axis parameters are used as the ideal posture parameters of each camera. In this ideal posture, all cameras look at the same point in space, and the camera's longitudinal axis (Y axis) is parallel, ensuring a good viewing experience.

[0104] In some embodiments, the step of determining the camera posture adjustment parameters based on the ideal posture parameters of the external participating camera includes:

[0105] Step S21, determining the calibration posture parameters of each camera based on the camera extrinsic parameters;

[0106] Step S22 : determining the attitude adjustment parameters of each camera based on the difference between the calibrated attitude parameters of each camera and the ideal attitude parameters.

[0107] Based on the embodiment, the calibration state of the camera refers to the actual real-time state of the camera. After obtaining the camera extrinsic parameters, the calibration X-axis parameters, calibration Y-axis parameters and calibration Z-axis parameters corresponding to the camera calibration state can be determined based on the camera extrinsic parameters. Then, based on the ideal posture parameters obtained in the above embodiment, the difference between the two is calculated to determine the posture adjustment parameters of each camera.

[0108] In some embodiments, the focal length f of different cameras is compared with the average focal length The steps for determining the scaling adjustment parameters include:

[0109] Step S23, calculating the average focal length based on the focal length f of each camera

[0110] Specifically, the average focal length is calculated based on the focal length f of each camera. This average focal length is used during calibration to obtain a calibrated image with consistent focal length, avoiding the feeling of zooming when switching camera positions.

[0111] Step S24, based on the focal length f of each camera and the average focal length The difference in values ​​determines the scaling adjustment parameter.

[0112] Specifically, the average focal length For each camera's corresponding image, a zoom adjustment parameter can be determined based on the difference between the camera's focal length and the average focal length, and the corresponding image can be processed based on the zoom adjustment parameter.

[0113] In addition, the present invention also provides an image processing device.

[0114] The image processing device of the present invention includes:

[0115] The basic matrix calculation module calculates the basic matrix F based on the matching feature points of different camera images;

[0116] The calibration parameter calculation module is used to calculate the camera calibration parameters based on the basic matrix F and the spatial geometric relationship; wherein the camera calibration parameters at least include camera intrinsic parameters and camera extrinsic parameters, and the camera extrinsic parameters at least include the rotation matrix R, the camera center A n and the translation matrix t;

[0117] The ideal orientation determination module is used to determine the direction of the camera based on the rotation matrix R and the camera center A. n Determine the ideal camera orientation;

[0118] A baseline direction determination module is used to determine the direction of the baseline based on the center coordinate A n Determine the baseline direction;

[0119] An image correction module is used to perform image correction on the camera based on the camera intrinsic parameters, the camera extrinsic parameters, the camera ideal orientation and the baseline direction.

[0120] The specific implementation of the image processing device of the present invention can refer to the various embodiments of the image processing method of the present invention, and will not be repeated here.

[0121] In addition, an embodiment of the present invention further provides an electronic device.

[0122] The electronic device of the present invention includes a memory and a processor. The memory stores an image processing program. When the image processing program is executed by the processor, the steps of the image processing method described above are implemented.

[0123] The method implemented when the image processing program running on the processor is executed can refer to the various embodiments of the image processing method of the present invention, and will not be repeated here.

[0124] In addition, an embodiment of the present invention further provides a computer-readable storage medium.

[0125] The computer-readable storage medium of the present invention stores an image processing program, and when the image processing program is executed by a processor, the steps of the image processing method described above are implemented.

[0126] The method implemented when the image processing program running on the processor is executed can refer to the various embodiments of the image processing method of the present invention, and will not be described in detail here.

[0127] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0128] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0129] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0130] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image processing method for a free-viewpoint camera, comprising: Obtain camera intrinsic parameters, camera extrinsic parameters, and ideal camera posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera; The camera posture adjustment parameters are determined based on the ideal posture parameters of the camera outside the camera, and the focal length f of different cameras and the average focal length are determined based on the ideal posture parameters of the camera outside the camera. Determine scaling adjustment parameters; Performing scaling correction on the corresponding image based on the scaling adjustment parameters of each camera, and performing posture correction on the corresponding image based on the posture adjustment parameters of each camera; The step of obtaining camera intrinsic parameters, camera extrinsic parameters and ideal camera posture parameters of different cameras includes: Calculate the fundamental matrix F based on the matching feature points of different camera images; The camera calibration parameters are calculated based on the basic matrix F and the spatial geometric relationship; wherein the camera calibration parameters include at least camera intrinsic parameters and camera extrinsic parameters, and the camera extrinsic parameters include at least rotation matrix R, camera center A n and the translation matrix t; Based on the rotation matrix R and the camera center A n Determine the ideal camera orientation; Based on the center coordinate A n Determine the baseline direction; Determining ideal posture parameters of the camera based on the ideal camera orientation and the baseline direction; The step of determining the camera posture adjustment parameters based on the ideal posture parameters of the external participating cameras includes: Determining calibration parameters of each camera based on the camera extrinsic parameters; Determining attitude adjustment parameters of each camera based on the difference between the calibrated attitude parameters of each camera and the ideal attitude parameters; The focal length f based on different cameras and the average focal length The steps for determining the scaling adjustment parameters include: Calculate the average focal length based on the focal length f of each camera Based on the focal length f of each camera and the average focal length The difference in values ​​determines the scaling adjustment parameter.

2. An image processing method for a free-viewpoint camera, comprising: Obtain camera intrinsic parameters, camera extrinsic parameters, and ideal camera posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera; Determine the camera's attitude adjustment parameters based on the ideal attitude parameters of the camera outside the camera, and The focal length f and the average focal length of different cameras Determine scaling adjustment parameters; The corresponding image is scaled and corrected based on the scale adjustment parameters of each camera, and the corresponding image is posture-corrected based on the posture adjustment parameters of each camera.

3. The image processing method according to claim 2, wherein the step of obtaining camera intrinsic parameters, camera extrinsic parameters, and ideal camera posture parameters of different cameras comprises: Calculate the fundamental matrix F based on the matching feature points of different camera images; The camera calibration parameters are calculated based on the basic matrix F and the spatial geometric relationship; wherein the camera calibration parameters include at least camera intrinsic parameters and camera extrinsic parameters, and the camera extrinsic parameters include at least rotation matrix R, camera center A n and the translation matrix t; Based on the rotation matrix R and the camera center A n Determine the ideal camera orientation; Based on the center coordinate A n Determine the baseline direction; Determine ideal pose parameters of the camera based on the ideal camera orientation and the baseline direction.

4. The image processing method according to claim 3, wherein the rotation matrix R and the camera center A are used to determine the image processing method. n The steps to determine the ideal camera orientation include: The real-time principal optical axis parameters of the camera are obtained based on the rotation matrix R Determine the line connecting any spatial point and the camera center based on the preset angle formula The angle between the camera's real-time principal optical axis is θ n ; Wherein, the preset angle formula is: The average value of the sum of all angles is calculated based on a preset average value calculation formula, wherein the preset average value calculation formula is: based on The coordinates of the ideal alignment point X′ are determined by minimizing the value of Based on the camera center point coordinates A of each camera n The direction of the line connecting the ideal alignment point X' Determine the ideal orientation for each camera.

5. The image processing method according to claim 3, wherein the image processing method is based on the center coordinate A. n The steps to determine the baseline orientation include: Get the camera center point coordinates A1, A2, A3, ..., A of all cameras n ; The steps of obtaining the center point coordinates are executed cyclically until the coordinates of the last two points are obtained; wherein the cyclic steps are: Calculate the coordinates of the camera center points of adjacent serial numbers in sequence: A n =(A n +A n+1 ) / 2, as the new spatial point coordinates; The baseline direction is determined based on the obtained coordinates of the last two points.

6. The image processing method according to claim 2, wherein the step of determining the camera posture adjustment parameters based on the ideal posture parameters of the external participating camera comprises: Determining calibration parameters of each camera based on the camera extrinsic parameters; The attitude adjustment parameters of each camera are determined based on the difference between the calibrated attitude parameters of each camera and the ideal attitude parameters.

7. The image processing method according to claim 3, wherein the step of determining the ideal posture parameters of each camera based on the ideal camera orientation and the baseline direction comprises: Calculating the cross product of the ideal camera orientation and the baseline direction to determine the Y-axis parameter corresponding to the camera; Determining an X-axis parameter of each camera based on the ideal camera orientation and the Y-axis parameter; The ideal posture parameters of each camera are determined based on the X-axis parameters, the Y-axis parameters, and the ideal camera orientation.

8. The image processing method according to claim 2, wherein the focal length f based on different cameras is equal to the average focal length. The steps for determining the scaling adjustment parameters include: Calculate the average focal length based on the focal length f of each camera Based on the focal length f of each camera and the average focal length The difference in values ​​determines the scaling adjustment parameter.

9. The image processing method according to claim 3, wherein the step of calculating the fundamental matrix F based on the matching feature points of different camera images comprises: Obtain real-time images taken by each camera at the same time; Extracting feature points from each of the real-time images; Feature point matching is performed based on the extracted feature points to obtain multiple groups of matching feature points.

10. An image processing device, comprising: An acquisition module is used to obtain camera intrinsic parameters, camera extrinsic parameters and ideal camera posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera; A determination module is used to determine the camera posture adjustment parameters based on the ideal posture parameters of the camera outside the camera, and based on the focal length f of different cameras and the average focal length Determine scaling adjustment parameters; A processing module for scaling the corresponding image based on the scaling adjustment parameters of each camera Correction is performed, and posture correction is performed on the corresponding image based on the posture adjustment parameters of each camera.

Images