Image processing method, device and storage medium for free-viewing-angle camera

Abstract

The present invention discloses an image processing method for a free-viewing-angle camera, comprising: obtaining camera intrinsic parameters, camera extrinsic parameters and ideal posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera; determining the posture adjustment parameters of the camera based on the camera extrinsic parameters and the ideal posture parameters of the camera, and determining the zoom adjustment parameters based on the focal length f and the average focal length of different cameras; performing zoom correction on the corresponding image based on the zoom adjustment parameters of each camera, and performing posture correction on the corresponding image based on the posture adjustment parameters of each camera. The present invention also discloses an image processing device and a storage medium. The present invention can avoid the phenomenon of image jitter when switching camera positions for viewing, thereby improving the viewing experience.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device and storage medium for a free-viewing-angle camera. Background Art

[0002] The free view camera system is composed of several cameras with synchronized start and end times of recording and shooting. When watching the free view video, the viewer can switch the camera position at any time to achieve the viewing experience of switching view. Since the cameras of the free view system cannot be placed strictly according to a certain standard when they are placed manually, the relative position or posture angle of the same object in the pictures of adjacent cameras in space varies greatly, and the picture will shake when switching the camera position for viewing, which affects the imaging effect and reduces the viewing experience. Summary of the invention

[0003] The main purpose of the present invention is to provide an image processing method, device and storage medium for a free-viewpoint camera, aiming to solve the problem that the free-viewpoint camera system in the prior art generates jitter when switching viewing positions, thereby affecting the viewing experience.

[0004] To achieve the above object, the present invention provides an image processing method, the method comprising:

[0005] Obtain camera intrinsic parameters, camera extrinsic parameters and ideal camera posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera;

[0006] The camera posture adjustment parameters are determined based on the ideal posture parameters of the camera outside the camera, and the focal length f of different cameras and the average focal length are determined based on the focal length f of the camera outside the camera. Determine scaling adjustment parameters;

[0007] The corresponding image is scaled and corrected based on the scale adjustment parameters of each camera, and the corresponding image is posture-corrected based on the posture adjustment parameters of each camera.

[0008] Optionally, the step of obtaining camera intrinsic parameters, camera extrinsic parameters and ideal posture parameters of different cameras includes:

[0009] Calculate the fundamental matrix F based on the matching feature points of different camera images;

[0010] The camera calibration parameters are calculated based on the basic matrix F and the spatial geometric relationship; wherein the camera calibration parameters at least include camera intrinsic parameters and camera extrinsic parameters, and the camera extrinsic parameters at least include the rotation matrix R, the camera center A n and the translation matrix t;

[0011] Based on the rotation matrix R and the camera center An Determine the ideal orientation of the camera;

[0012] Based on the center coordinate A n Determine the baseline direction;

[0013] Determine ideal posture parameters of the camera based on the ideal camera orientation and the baseline direction.

[0014] Optionally, the rotation matrix R and the camera center A are based on n The steps to determine the ideal camera orientation include:

[0015] Based on the rotation matrix R, the real-time principal optical axis parameters of the camera are obtained.

[0016] Determine the line between any spatial point and the camera center based on the preset angle formula The angle between the camera's real-time principal optical axis is θ n ; Wherein, the preset angle formula is:

[0017]

[0018] The average value of the sum of all angles is calculated based on a preset average value calculation formula, wherein the preset average value calculation formula is:

[0019]

[0020] based on The minimized value of determines the coordinates of the ideal alignment point X′;

[0021] Based on the camera center point coordinates A of each camera n The direction of the line connecting the ideal alignment point X' Determine the ideal orientation for each camera.

[0022] Optionally, the center coordinate A n The steps to determine the direction of the baseline include:

[0023] Get the camera center point coordinates A1, A2, A3, ..., A of all cameras n ;

[0024] The step of obtaining the center point coordinates is executed repeatedly until the coordinates of the last two points are obtained; wherein the steps of the cycle are:

[0025] Calculate the coordinates of the camera center points of adjacent serial numbers in sequence: A n =(A n +A n+1 ) / 2, as the new spatial point coordinates;

[0026] The baseline direction is determined based on the coordinates of the last two points obtained.

[0027] Optionally,

[0028] The step of determining the camera's posture adjustment parameters based on the ideal posture parameters of the external participating cameras comprises:

[0029] Determining calibration posture parameters of each camera based on the camera extrinsic parameters;

[0030] The attitude adjustment parameters of each camera are determined based on the difference between the calibrated attitude parameters of each camera and the ideal attitude parameters.

[0031] Optionally, the step of determining the ideal posture parameters of each camera based on the ideal camera orientation and the baseline direction includes:

[0032] Calculate the cross product of the ideal orientation of the camera and the baseline direction to determine the Y-axis parameter corresponding to the camera;

[0033] Determine the X-axis parameters of each camera based on the ideal camera orientation and the Y-axis parameters;

[0034] The ideal posture parameters of each camera are determined based on the X-axis parameters, the Y-axis parameters and the ideal camera orientation.

[0035] Optionally, the focal length f based on different cameras and the average focal length The steps to determine the scaling adjustment parameters include:

[0036] Calculate the average focal length based on the focal length f of each camera

[0037] Based on the focal length f of each camera and the average focal length The difference determines the scaling adjustment parameter.

[0038] Optionally, the step of calculating the basic matrix F based on the matching feature points of different camera images includes:

[0039] Obtain real-time images taken by each camera at the same time;

[0040] Extracting feature points from each of the real-time images;

[0041] Feature point matching is performed based on the extracted feature points to obtain multiple groups of matching feature points.

[0042] To achieve the above object, the present invention further provides an image processing device, the image processing device comprising:

[0043] An acquisition module, used to acquire camera intrinsic parameters, camera extrinsic parameters and ideal posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera;

[0044] A determination module is used to determine the camera posture adjustment parameters based on the ideal posture parameters of the camera outside the camera, and based on the focal length f of different cameras and the average focal length Determine scaling adjustment parameters;

[0045] The processing module is used to perform scaling correction on the corresponding image based on the scaling adjustment parameters of each camera, and perform posture correction on the corresponding image based on the posture adjustment parameters of each camera.

[0046] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which an image processing program is stored, and when the image processing program is executed by a processor, the steps of the image processing method described above are implemented.

[0047] The image processing method, device and computer-readable storage medium provided by the present invention obtain camera intrinsic parameters, camera extrinsic parameters and ideal posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera; determine the camera posture adjustment parameters based on the camera extrinsic parameters and the ideal posture parameters of the camera, and calculate the posture adjustment parameters based on the focal length f of different cameras and the average focal length Determine the zoom adjustment parameters; perform zoom correction on the corresponding image based on the zoom adjustment parameters of each camera, and perform image posture correction on the corresponding image based on the posture adjustment parameters of each camera. Compared with the existing technology, each camera is first calibrated to obtain the camera internal parameters and camera external parameters, and then the ideal posture of each camera is calculated by mathematical methods. Finally, the image correction algorithm is used to perform image processing such as rotation, translation, and cropping to obtain the image of each camera in the ideal posture. In the ideal posture, all cameras look at the same point in space, and the longitudinal axis Y axis of the camera is parallel to ensure the viewing effect of switching camera positions. This method ensures that the relative position of the object in the center of the picture remains unchanged in the pictures of different camera positions, thereby achieving a "shake-free" viewing experience when switching camera positions. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the system structure of the hardware operating environment involved in the embodiment of the present invention;

[0049] Figure 2 This is a flow chart of an embodiment of an image processing method of the present invention;

[0050] Figure 3 This is an example diagram of image comparison before image processing by different cameras in an embodiment of the image processing method of the present invention;

[0051] Figure 4 This is an example diagram of image comparison after image processing by different cameras in an embodiment of the image processing method of the present invention. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0054] In the prior art, the multi-terminal display system is difficult to implement.

[0055] In order to solve the above technical problems, the present invention provides an image processing method, firstly calibrates each camera to obtain the camera intrinsic parameters and camera extrinsic parameters, then calculates the ideal posture of each camera by mathematical methods, and finally performs image processing such as rotation, translation, and cropping through a picture correction algorithm to obtain the picture of each camera in the ideal posture. In the ideal posture, all cameras look at the same point in space, and the longitudinal axis Y axis of the camera is parallel to ensure the viewing effect of switching camera positions. This method ensures that the relative position of the object in the center of the picture remains unchanged in the pictures of different camera positions, thereby achieving a "shake-free" viewing experience when switching camera positions.

[0056] like Figure 1 As shown, Figure 1 It is a schematic diagram of the system structure of the hardware operating environment involved in the embodiment of the present invention.

[0057] The terminal of the embodiment of the present invention can be a terminal device with computing capability, or a PC, or a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group AudioLayer IV) player, a portable computer, or other mobile terminal devices with display capabilities.

[0058] like Figure 1As shown, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0059] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0060] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0061] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an image processing program.

[0062] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and perform image processing with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and perform image processing with the client; and the processor 1001 can be used to call the image processing program stored in the memory 1005 and perform the following operations:

[0063] Obtain camera intrinsic parameters, camera extrinsic parameters and ideal camera posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera;

[0064] The camera posture adjustment parameters are determined based on the ideal posture parameters of the camera outside the camera, and the focal length f of different cameras and the average focal length are determined based on the focal length f of the camera outside the camera. Determine scaling adjustment parameters;

[0065] The corresponding image is scaled and corrected based on the scale adjustment parameters of each camera, and the corresponding image is posture-corrected based on the posture adjustment parameters of each camera.

[0066] Further, the processor 1001 may call the image processing program stored in the memory 1005, and also perform the following operations:

[0067] Calculate the fundamental matrix F based on the matching feature points of different camera images;

[0068] The camera calibration parameters are calculated based on the basic matrix F and the spatial geometric relationship; wherein the camera calibration parameters at least include camera intrinsic parameters and camera extrinsic parameters, and the camera extrinsic parameters at least include the rotation matrix R, the camera center A n and the translation matrix t;

[0069] Based on the rotation matrix R and the camera center A n Determine the ideal orientation of the camera;

[0070] Based on the center coordinate A n Determine the baseline direction;

[0071] Determine ideal posture parameters of the camera based on the ideal camera orientation and the baseline direction.

[0072] Further, the processor 1001 may call the image processing program stored in the memory 1005, and also perform the following operations:

[0073] Based on the rotation matrix R, the real-time principal optical axis parameters of the camera are obtained.

[0074] Determine the line between any spatial point and the camera center based on the preset angle formula The angle between the camera's real-time principal optical axis is θ n ; Wherein, the preset angle formula is:

[0075]

[0076] The average value of the sum of all angles is calculated based on a preset average value calculation formula, wherein the preset average value calculation formula is:

[0077]

[0078] based on The minimized value of determines the coordinates of the ideal alignment point X′;

[0079] Based on the camera center point coordinates A of each camera n The direction of the line connecting the ideal alignment point X′ Determine the ideal orientation for each camera.

[0080] Further, the processor 1001 may call the image processing program stored in the memory 1005, and also perform the following operations:

[0081] Get the camera center point coordinates A1, A2, A3, ..., A of all cameras n ;

[0082] The step of obtaining the center point coordinates is executed repeatedly until the coordinates of the last two points are obtained; wherein the steps of the cycle are:

[0083] Calculate the coordinates of the camera center points of adjacent serial numbers in sequence: A n =(A n +A n+1 ) / 2, as the new spatial point coordinates;

[0084] The baseline direction is determined based on the coordinates of the last two points obtained.

[0085] Further, the processor 1001 may call the image processing program stored in the memory 1005, and also perform the following operations:

[0086] Determining calibration posture parameters of each camera based on the camera extrinsic parameters;

[0087] The attitude adjustment parameters of each camera are determined based on the difference between the calibrated attitude parameters of each camera and the ideal attitude parameters.

[0088] Furthermore, the processor 1001 may call the image processing program stored in the memory 1005, and further perform the following operations:

[0089] Calculate the cross product of the ideal orientation of the camera and the baseline direction to determine the Y-axis parameter corresponding to the camera;

[0090] Determine the X-axis parameters of each camera based on the ideal camera orientation and the Y-axis parameters;

[0091] The ideal posture parameters of each camera are determined based on the X-axis parameters, the Y-axis parameters and the ideal camera orientation.

[0092] Further, the processor 1001 may call the image processing program stored in the memory 1005, and further perform the following operations:

[0093] Calculate the average focal length based on the focal length f of each camera

[0094] Based on the focal length f of each camera and the average focal length The difference determines the scaling adjustment parameter.

[0095] Furthermore, the processor 1001 may call the image processing program stored in the memory 1005, and further perform the following operations:

[0096] Obtain real-time images taken by each camera at the same time;

[0097] Extracting feature points from each of the real-time images;

[0098] Feature point matching is performed based on the extracted feature points to obtain multiple groups of matching feature points.

[0099] Reference Figure 2 , Figure 2 The figure is a flowchart of an embodiment of the image processing method for a free-viewing-angle camera of the present invention. Based on the algorithm of the present invention, each camera is first calibrated to obtain the camera internal parameters and camera external parameters, and then the ideal posture of each camera is calculated by mathematical methods to ensure the viewing effect of switching camera positions. Finally, the image is rotated, translated, cropped, etc. by the image correction algorithm to obtain the picture of each camera in the ideal posture. This method ensures that the relative position of the object in the center of the picture remains unchanged in the pictures of different camera positions, so as to achieve a "shake-free" viewing experience when switching camera positions. In some embodiments, the image processing method includes:

[0100] Step S10, obtaining camera intrinsic parameters, camera extrinsic parameters and ideal camera posture parameters of different cameras, wherein the camera intrinsic parameters at least include the focal length f of the camera.

[0101] The image processing method of the present invention is mainly used in a free-viewing-angle camera system, in which the number of cameras is more than two, and the number is not specifically limited. After the cameras are placed, all cameras are used to take a picture at the same time to obtain a real-time image taken by each camera at the same time. Then, feature points are extracted from each real-time image, and then the feature points of each real-time image are matched according to the features of the feature points, and the corresponding relationship between the feature points of each real-time image is established to obtain multiple groups of feature points.

[0102] After obtaining multiple sets of feature points, the intrinsic and extrinsic parameters of the camera can be obtained based on the multiple sets of feature points, thereby obtaining the focal length f of the camera and the rotation matrix R in the extrinsic parameters. Then, the real-time principal optical axis parameters of the camera are determined based on the rotation matrix, the ideal orientation of the camera is determined based on the real-time principal optical axis parameters of the camera, and then the ideal posture parameters of the camera are determined according to the ideal orientation of the camera, where the ideal posture parameters of the camera include X-axis parameters, Y-axis parameters, and Z-axis parameters.

[0103] Step S20, determining the camera posture adjustment parameters based on the ideal posture parameters of the external participating cameras, and based on the focal lengths f of different cameras and the average focal lengths Determines the scaling adjustment parameters.

[0104] The actual attitude parameters of the camera can be determined based on the external parameters of the camera, and the attitude adjustment parameters of each camera can be determined based on the difference between the actual attitude parameters of the camera and the ideal attitude parameters of the camera. The zoom adjustment parameters of each camera can be determined based on the difference between the focal length f of each camera and the average value of f of all cameras.

[0105] Step S30: performing scaling correction on the corresponding image based on the scaling adjustment parameters of each camera, and performing image posture correction on the corresponding image based on the posture adjustment parameters of each camera.

[0106] Based on the posture adjustment parameters of each camera, the images taken by each camera are scaled and corrected accordingly. Based on the posture adjustment parameters of each camera, the images taken by each camera are translated, rotated, etc. (achieved by changing the projection relationship of the image) to achieve posture correction of the image. For image processing effects, refer to Figure 3 and Figure 4 ,in, Figure 3 is an example diagram of image comparison before image processing by different cameras in an embodiment of the image processing method of the present invention, Figure 4 1 is an example diagram of image comparison after image processing by different cameras in an embodiment of the image processing method of the present invention. In some embodiments, the corrected image may be cropped to obtain a corrected image without black edges.

[0107] In the above embodiment, the camera intrinsic parameters, camera extrinsic parameters and ideal posture parameters of different cameras are obtained, wherein the camera intrinsic parameters at least include the focal length f of the camera; the camera posture adjustment parameters are determined based on the camera extrinsic parameters and the ideal posture parameters of the camera, and the focal length f of different cameras and the average focal length are compared. Determine the zoom adjustment parameters; perform zoom correction on the corresponding images based on the zoom adjustment parameters of each camera, and perform posture correction on the corresponding images based on the posture adjustment parameters of each camera. Through the above method, first calibrate each camera to obtain the camera internal parameters and camera external parameters, then calculate the ideal posture of each camera by mathematical methods, and finally perform image processing such as rotation, translation, and cropping through the picture correction algorithm to obtain the picture of each camera in the ideal posture. In the ideal posture, all cameras look at the same point in space, and the longitudinal axis Y axis of the camera is parallel to ensure the viewing effect of switching positions. This method ensures that the relative position of the object in the center of the picture remains unchanged in the pictures of different positions, thereby achieving a "shake-free" viewing experience when switching positions.

[0108] In some embodiments, step S10 includes:

[0109] Step S11, calculating a basic matrix F based on matching feature points of different camera images.

[0110] In some embodiments, before step S11, the method further includes:

[0111] Step S40, obtaining real-time images taken by each camera at the same time;

[0112] Step S50, extracting feature points from each of the real-time images;

[0113] Step S60: performing feature point matching based on the extracted feature points to obtain multiple groups of matching feature points.

[0114] After the cameras are placed, all cameras are used to take a picture at the same time to obtain the real-time images taken by each camera at the same time. Then, feature points are extracted from each real-time image, and then the feature points of each real-time image are matched according to the features of the feature points, and the corresponding relationship between the feature points of each real-time image is established to obtain multiple groups of feature points.

[0115] For a set of matching feature points in two real-time images, the homogenized image coordinates are x and x′ respectively. According to the spatial geometric relationship, x′ can be obtained T Fx=0, the basic matrix F can be determined based on more than eight groups of matching feature points.

[0116] Step S12, calculate the camera calibration parameters based on the basic matrix F and the spatial geometric relationship; wherein the camera calibration parameters include at least the camera intrinsics and camera extrinsic parameters, and the camera extrinsic parameters include at least the rotation matrix R, the camera center An and the translation matrix t (where An = -R't, R' is the inverse matrix of R), wherein each column of the rotation matrix R can represent the orientation of the x, y, and z axes of the camera coordinate system in the world coordinate system. The translation matrix t is a 3x1 matrix, which represents the displacement of the camera center relative to the center of the world coordinate system. Among them, the method for calculating the camera intrinsics and camera extrinsic parameters based on the basic matrix and the spatial geometric relationship can refer to the existing calculation method, which will not be repeated here. The rotation matrix R is a 3x3 matrix. Assume that the rotation angle θ around the x, y, and z axes in space can be represented by the following matrix:

[0117]

[0118]

[0119]

[0120] The rotation matrix R is R x , R y With R z The product of the three, that is, if we rotate around the x, y, and z axes by angles Ψ, θ, and Φ respectively, the rotation matrix is:

[0121]

[0122] It represents the rotation of the camera coordinate system relative to the world coordinate system. Each column of the rotation matrix R can represent the orientation of the x, y, and z axes of the camera coordinate system in the world coordinate system. The translation matrix t is a 3x1 matrix that represents the displacement of the camera center relative to the center of the world coordinate system. The above formula can be used to obtain the final calibration result: the camera intrinsic parameter matrix K, the camera extrinsic parameter matrix R and t.

[0123] Step S13, based on the rotation matrix R and the camera center A n Determine the ideal orientation of the camera.

[0124] Specifically, in some embodiments, step S13 includes:

[0125] Step S131, obtaining the real-time principal optical axis parameters of the camera based on the rotation matrix R

[0126] The main optical axis of the camera is a straight line passing through the centers of the two spherical surfaces of the thin lens. The real-time main optical axis parameters Refers to the matrix parameters corresponding to the main optical axis of the camera in real-time placement state. Among them, the real-time main optical axis parameters of the camera It can be obtained by converting the rotation matrix R, where the third column of the rotation matrix R corresponds to the direction of the main optical axis.

[0127] Step S132: determine the line connecting any spatial point and the camera center based on a preset angle formula The angle between the camera's real-time principal optical axis is θ n ; Wherein, the preset angle formula is:

[0128]

[0129] Step S133, based on the angle θ n The minimum average value of determines the coordinates of the ideal alignment point X′.

[0130] Specifically, suppose there is a point X in space, and the center of camera n is point A in space n (Among them, A n By calibrating the camera position in the information as described above, the principal optical axis is (Obtained by converting the rotation matrix in the camera external parameters). Ideally, if all cameras align the imaging center point to X, then the camera's main optical axis and direction vector The angle between them is 0 (i.e. coincident). In reality, suppose there is a point X′ in space, and the line connecting it and the center of the camera With camera main optical axis The angle between them is θ n , according to the properties of the vector product:

[0131]

[0132] set up minimize The coordinate of X' can be obtained. This is the ideal orientation of each camera (ideal Z axis).

[0133] Step S134: based on the camera center point coordinates A of each camera n The direction of the line connecting the ideal alignment point X′ Determine the ideal orientation for each camera.

[0134] Based on the above steps, after determining the coordinates of the ideal alignment point X′, the camera center coordinates A of each camera are n Calculate the coordinates A of each camera center point n The direction of the line connecting the ideal alignment point X′ This determines the ideal orientation of each camera.

[0135] In the above image processing method, the real-time principal optical axis parameters of the camera are obtained based on the rotation matrix R.

[0136] Determine the line between any spatial point and the camera center based on the preset angle formula The angle between the camera's real-time principal optical axis is θ n , based on the angle θ n The minimum average value of the ideal alignment point X′ is determined; based on the camera center point coordinates A of each camera n The direction of the line connecting the ideal alignment point X′ Determine the ideal orientation of each camera. Through the above method, the ideal orientation of all cameras towards the spatial point can be accurately determined, so that image processing can be performed based on the ideal orientation to obtain a better image viewing effect.

[0137] Step S14, based on the center coordinate A n Determine the baseline orientation.

[0138] Specifically, in some embodiments, step S40 includes:

[0139] Step S141, obtaining the camera center point coordinates A1, A2, A3, ..., A of all cameras n ;

[0140] Step S142, looping steps are executed repeatedly until the coordinates of the last two points are obtained; wherein the looping steps are:

[0141] Calculate the coordinates of the camera center points of adjacent serial numbers in sequence: A n =(A n +A n+1 ) / 2, as the new spatial point coordinates;

[0142] Step S143, determining the baseline direction based on the obtained coordinates of the last two points.

[0143] The spatial coordinates of the center of each camera can be obtained by the calibration steps: A1, A2, A3, ..., A n , calculate successively A new set of spatial points can be obtained. The above steps are repeated several times until the last two points are obtained. The direction vector of the two points is calculated. The direction vector composed of the two points is the free view baseline direction.

[0144] Step S15, determining ideal posture parameters of the camera based on the ideal camera orientation and the baseline direction.

[0145] Specifically, in some embodiments, step S15 includes:

[0146] Step S151, calculating the cross product of the ideal orientation of the camera and the baseline direction to determine the Y-axis parameter corresponding to the camera;

[0147] Step S152, determining the X-axis parameters of each camera based on the ideal camera orientation and the Y-axis parameters;

[0148] Step S153, determining the ideal posture parameters of each camera based on the X-axis parameters, the Y-axis parameters and the ideal camera orientation.

[0149] Based on the above steps, the ideal main orientation of the camera (the direction of the camera coordinate axis Z axis) is calculated, and then the baseline direction of the free perspective system is calculated. (Detailed explanation later), perform a vector product (cross product) on the two vectors You can get a perpendicular and The direction is the camera Y-axis parameter. Since the camera Z-axis parameter and Y-axis parameter have been determined, the X-axis parameter is also determined, and the ideal posture of the camera has been determined. Among them, the above X-axis parameter, Y-axis parameter, and Z-axis parameter are the direction vector parameters corresponding to each axis. The determined camera X-axis parameter, Y-axis parameter, and Z-axis parameter are used as the ideal posture parameters of each camera. In this ideal posture, all cameras look at the same point in space, and the longitudinal axis (Y axis) of the camera is parallel, ensuring the viewing experience.

[0150] In some embodiments, the step of determining the camera's posture adjustment parameters based on the ideal posture parameters of the external participating camera includes:

[0151] Step S21, determining the calibration posture parameters of each camera based on the camera extrinsic parameters;

[0152] Step S22, determining the attitude adjustment parameters of each camera based on the difference between the calibrated attitude parameters of each camera and the ideal attitude parameters.

[0153] Based on the embodiment, the calibration state of the camera refers to the actual real-time state of the camera. After obtaining the camera extrinsic parameters, the calibration X-axis parameters, calibration Y-axis parameters and calibration Z-axis parameters corresponding to the camera calibration state can be determined based on the camera extrinsic parameters, and then based on the ideal posture parameters obtained in the above embodiment, the difference between the two is calculated to determine the posture adjustment parameters of each camera.

[0154] In some embodiments, the focal lengths f of different cameras are based on the average focal lengths The steps to determine the scaling adjustment parameters include:

[0155] Step S23, calculating the average focal length based on the focal length f of each camera

[0156] Specifically, the average focal length is calculated based on the focal length f of each camera. This average focal length is used during calibration to obtain a calibrated image with consistent focal length, avoiding the feeling of zooming when switching camera positions.

[0157] Step S24, based on the focal length f of each camera and the average focal length The difference determines the scaling adjustment parameter.

[0158] Specifically, the average focal length For each image captured by a camera, a zoom adjustment parameter can be determined based on the difference between the focal length of the camera and the average focal length, and the corresponding image can be processed based on the zoom adjustment parameter.

[0159] In addition, the present invention also provides an image processing device.

[0160] The image processing device of the present invention comprises:

[0161] The basic matrix calculation module calculates the basic matrix F based on the matching feature points of different camera images;

[0162] A calibration parameter calculation module is used to calculate the camera calibration parameters based on the basic matrix F and the spatial geometric relationship; wherein the camera calibration parameters at least include camera intrinsic parameters and camera extrinsic parameters, and the camera extrinsic parameters at least include the rotation matrix R, the camera center A n and the translation matrix t;

[0163] An ideal orientation determination module is used to determine the orientation of the camera based on the rotation matrix R and the camera center A. n Determine the ideal orientation of the camera;

[0164] A baseline direction determination module is used to determine the direction of the base line based on the center coordinate A n Determine the baseline direction;

[0165] An image correction module is used to perform image correction on the camera based on the camera intrinsic parameters, the camera extrinsic parameters, the camera ideal orientation and the baseline direction.

[0166] The specific implementation of the image processing device of the present invention can refer to the various embodiments of the image processing method of the present invention, which will not be described in detail here.

[0167] In addition, an embodiment of the present invention further provides an electronic device.

[0168] The electronic device of the present invention comprises a memory and a processor. An image processing program is stored in the memory. When the image processing program is executed by the processor, the steps of the image processing method described above are implemented.

[0169] The method implemented when the image processing program running on the processor is executed can refer to the various embodiments of the image processing method of the present invention, and will not be repeated here.

[0170] In addition, an embodiment of the present invention further provides a computer-readable storage medium.

[0171] The computer-readable storage medium of the present invention stores an image processing program, and when the image processing program is executed by a processor, the steps of the image processing method described above are implemented.

[0172] Among them, the method implemented when the image processing program running on the processor is executed can refer to the various embodiments of the image processing method of the present invention, and will not be repeated here.

[0173] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0174] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0175] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0176] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image processing method for a free-viewing-angle camera, characterized in that: include: Calculate the fundamental matrix F based on the matching feature points of different camera images; The camera calibration parameters are calculated based on the basic matrix F and the spatial geometric relationship; wherein the camera calibration parameters at least include camera intrinsic parameters and camera extrinsic parameters, wherein the camera intrinsic parameters at least include the focal length f of the camera, and the camera extrinsic parameters at least include the rotation matrix R, the camera center point coordinates A n and the translation matrix t; Based on the rotation matrix R and the camera center point coordinate A n Determine the ideal orientation of the camera; Based on the center point coordinate A n Determine the baseline direction; Determining ideal posture parameters of the camera based on the ideal camera orientation and the baseline direction; The camera posture adjustment parameters are determined based on the ideal posture parameters of the camera outside the camera, and the focal length f of different cameras and the average focal length are determined based on the focal length f of the camera outside the camera. Determine zoom adjustment parameters for different cameras; The corresponding image is scaled and corrected based on the scale adjustment parameters of each camera, and the corresponding image is posture-corrected based on the posture adjustment parameters of each camera.

2. The image processing method according to claim 1, characterized in that: The rotation matrix R and the camera center A are based on n The steps to determine the ideal camera orientation include: Based on the rotation matrix R, the real-time principal optical axis parameters of the camera are obtained. ; Determine the line between any spatial point and the camera center based on the preset angle formula The angle between the real-time principal optical axis of the camera is ; Wherein, the preset angle formula is: The average value of the sum of all angles is calculated based on a preset average value calculation formula, wherein the preset average value calculation formula is: based on The ideal alignment point is determined by minimizing the value of The coordinates of Based on the camera center point coordinates A of each camera n With the ideal alignment point Connection direction Determine the ideal orientation for each camera.

3. The image processing method according to claim 1, characterized in that: Based on the center coordinate point A n The steps to determine the direction of the baseline include: Get the camera center point coordinates of all cameras ; The step of obtaining the center point coordinates is executed repeatedly until the coordinates of the last two points are obtained; wherein the steps of the cycle are: Calculate the coordinates of the camera center points of adjacent serial numbers in sequence: A n =(A n +A n+1 ) / 2, as the new spatial point coordinates; The baseline direction is determined based on the coordinates of the last two points obtained.

4. The image processing method according to claim 1, wherein: The step of determining the camera's posture adjustment parameters based on the ideal posture parameters of the external participating cameras comprises: Determining calibration posture parameters of each camera based on the camera extrinsic parameters; The attitude adjustment parameters of each camera are determined based on the difference between the calibrated attitude parameters of each camera and the ideal attitude parameters.

5. The image processing method according to claim 1, wherein: The step of determining the ideal posture parameters of the camera based on the ideal camera orientation and the baseline direction comprises: Calculate the cross product of the ideal orientation of the camera and the baseline direction to determine the Y-axis parameter corresponding to the camera; Determine an X-axis parameter of the camera based on the ideal camera orientation and the Y-axis parameter; Determine ideal posture parameters of the camera based on the X-axis parameters, the Y-axis parameters and the ideal camera orientation.

6. The image processing method according to claim 1, wherein: The focal length f based on different cameras and the average focal length The steps to determine the scaling adjustment parameters include: Calculate the average focal length based on the focal length f of each camera ; Based on the focal length f of each camera and the average focal length The difference in value determines the scaling adjustment parameter.

7. The image processing method according to claim 1, characterized in that: The step of calculating the basic matrix F based on the matching feature points of different camera images includes: Obtain real-time images taken by each camera at the same time; Extracting feature points from each of the real-time images; Feature point matching is performed based on the extracted feature points to obtain multiple groups of matching feature points.

8. An image processing device, characterized in that: include: An acquisition module is used to calculate a basic matrix F based on matching feature points of different camera images, and to calculate camera calibration parameters based on the basic matrix F and spatial geometric relationships, wherein the camera calibration parameters at least include camera intrinsic parameters and camera extrinsic parameters, wherein the camera intrinsic parameters at least include the focal length f of the camera, and the camera extrinsic parameters at least include the rotation matrix R, the camera center point coordinates A n And the translation matrix t, based on the rotation matrix R and the camera center point coordinates A n Determine the ideal orientation of the camera, based on the center point coordinates A n Determine a baseline direction, and determine ideal posture parameters of the camera based on the ideal camera orientation and the baseline direction; A determination module is used to determine the camera posture adjustment parameters based on the ideal posture parameters of the camera outside the camera, and based on the focal length f of different cameras and the average focal length Determine zoom adjustment parameters for different cameras; The processing module is used to perform scaling correction on the corresponding image based on the scaling adjustment parameters of each camera, and perform posture correction on the corresponding image based on the posture adjustment parameters of each camera.

9. A computer-readable storage medium, characterized in that: An image processing program is stored, and when the image processing program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images