Method and device for generating virtual viewpoint based on camera posture

Abstract

The present application provides a method and device for generating a virtual viewpoint based on camera posture. The method comprises: constructing a three-dimensional point corresponding to the matching point pair according to the matching point pair between two adjacent viewpoint images and the internal parameters and external parameters of two cameras in a camera array corresponding to the adjacent viewpoint images; reprojecting each of the three-dimensional points according to the target camera posture and internal parameters of each target camera corresponding to the selected camera posture, and determining a virtual viewpoint image corresponding to the camera posture. The method for generating a virtual viewpoint based on camera posture provided in the embodiment of the present application can improve the efficiency of generating virtual viewpoints.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for generating a virtual viewpoint based on camera posture. Background Art

[0002] The real world is three-dimensional, but the current mainstream display devices are still two-dimensional. Three-dimensional display, especially naked-eye three-dimensional display, has received more and more attention. Naked-eye three-dimensional display requires dense viewpoint images, but there are many difficulties in using camera arrays for dense viewpoint acquisition, such as synchronization adjustment between camera arrays, camera calibration and attitude solution, as well as data storage and transmission. Therefore, in practical applications, a small number of real binocular cameras are usually used to acquire sparse viewpoints, and virtual viewpoints are generated through virtual viewpoint generation methods.

[0003] In the related art, DIBR (Depth Image Based Rendering) can be used to generate virtual viewpoints. However, when generating virtual viewpoints through DIBR, since the virtual viewpoint generation method of DIBR needs to correct the left and right images so that the two image epipolar lines are on the same horizontal line, and can only generate virtual viewpoints located on the same epipolar line, but cannot generate virtual viewpoint images at any position, resulting in low efficiency in generating virtual viewpoints. Summary of the invention

[0004] The embodiments of the present application provide a method and device for generating a virtual viewpoint based on a camera posture, thereby improving the efficiency of generating a virtual viewpoint.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a virtual viewpoint based on a camera posture, comprising:

[0006] Constructing a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and internal parameters and external parameters of two cameras in a camera array corresponding to the adjacent viewpoint images;

[0007] According to the target camera posture and internal parameters of each target camera corresponding to the selected camera posture, each of the three-dimensional points is reprojected to determine a virtual viewpoint image corresponding to the camera posture.

[0008] In one embodiment, before constructing the three-dimensional points corresponding to the matching point pairs between adjacent viewpoint images and the internal parameters and external parameters of two cameras corresponding to the adjacent viewpoint images, the method further includes:

[0009] Extracting a first matching point from a first viewpoint image of two adjacent viewpoint images;

[0010] Extracting, according to the trained optical flow estimation model, a second matching point matching the first matching point from a second viewpoint image of two adjacent viewpoint images;

[0011] A matching point pair is formed according to the first matching point and the second matching point.

[0012] In one embodiment, constructing a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and internal parameters and external parameters of two cameras corresponding to the adjacent viewpoint images includes:

[0013] Determine two light directions corresponding to the two cameras according to the matching point pairs between the two adjacent viewpoint images and the internal parameters and external parameters of the two cameras corresponding to the adjacent viewpoint images;

[0014] The two light directions are processed according to direct linear transformation to construct three-dimensional points corresponding to the matching point pairs.

[0015] In one embodiment, reprojecting each of the three-dimensional points according to the target camera pose and internal parameters of each target camera corresponding to the selected camera pose to determine the virtual viewpoint image corresponding to the camera pose includes:

[0016] When the selected camera posture is a horizontal position, performing camera position leveling on each of the target cameras in the camera array according to the three-dimensional coordinates of each of the target cameras, and obtaining the leveling coordinates of each of the target cameras;

[0017] Determining the target camera posture of each target camera according to the leveling coordinates of each target camera and the leveling rotation angle of each target camera obtained by averaging the rotation angles of each target camera;

[0018] According to the target camera posture and internal parameters of each target camera, each of the three-dimensional points is reprojected to determine a virtual viewpoint image corresponding to the camera posture.

[0019] In one embodiment, reprojecting each of the three-dimensional points to determine a virtual viewpoint image corresponding to the camera posture includes:

[0020] Reprojecting each of the three-dimensional points using two adjacent viewpoint images as reference images to determine two adjacent initial virtual viewpoint images corresponding to the camera posture;

[0021] Fill the holes in the left image of the adjacent initial virtual viewpoint image to generate a virtual viewpoint image.

[0022] In one embodiment, the step of performing hole filling on the left image in the adjacent initial virtual viewpoint image includes:

[0023] Acquire a first hole position in the left image of the adjacent initial virtual viewpoint image, where the hole area is larger than a preset area;

[0024] Pixels corresponding to the first hole position are obtained from the right image and filled into the first hole position.

[0025] In one embodiment, it further includes:

[0026] Acquire a second hole position in the left image whose hole area is less than or equal to the preset area;

[0027] The second hole position is filled by a closing operation.

[0028] In a second aspect, an embodiment of the present application provides a virtual viewpoint generation device based on camera posture, comprising:

[0029] A three-dimensional point construction module, used to construct a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and the internal parameters and external parameters of two cameras in the camera array corresponding to the adjacent viewpoint images;

[0030] The virtual viewpoint generation module is used to reproject each of the three-dimensional points according to the target camera posture and internal parameters of each target camera corresponding to the selected camera posture, and determine the virtual viewpoint image corresponding to the camera posture.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory storing a computer program, wherein when the processor executes the program, the steps of the method for generating a virtual viewpoint based on camera posture described in the first aspect are implemented.

[0032] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for generating a virtual viewpoint based on camera posture described in the first aspect.

[0033] The camera posture-based virtual viewpoint generation method and device provided in the embodiments of the present application reconstruct the three-dimensional points through the matching point pairs between two adjacent viewpoint images and the internal parameters and external parameters of the two cameras corresponding to the two viewpoint images, and then reproject each of the three-dimensional points according to the target camera posture and internal parameters of each target camera corresponding to the selected camera posture to determine the virtual viewpoint image corresponding to the camera posture, so that the reconstructed three-dimensional points can be reprojected based on any camera posture to generate a virtual viewpoint image at the corresponding position, so that virtual viewpoint images at different positions can be generated by selecting different camera postures, and the virtual viewpoint image can be generated without correcting the left and right images, thereby improving the generation efficiency of the virtual viewpoint. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 It is a flowchart of a method for generating a virtual viewpoint based on camera posture provided in an embodiment of the present application;

[0036] Figure 2 is a triangulation schematic diagram provided in an embodiment of the present application;

[0037] Figure 3 is a schematic diagram of the camera position and optical axis direction during real horizontal shooting provided by an embodiment of the present application;

[0038] Figure 4 is a schematic diagram of the camera position and optical axis direction after the position and optical axis are leveled according to an embodiment of the present application;

[0039] Figure 5 is a structural schematic diagram of a virtual viewpoint generation device based on camera posture provided in an embodiment of the present application;

[0040] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] In order to better understand the solution, the professional terms involved in the embodiments of the present invention are explained.

[0043] Camera internal parameters refer to parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc.

[0044] Camera posture refers to the external parameters of the camera, that is, the parameters of the camera in the world coordinate system, such as the camera's position and rotation direction;

[0045] Reprojection refers to the second projection. The first projection refers to projecting 3D points onto the image when the camera takes a photo. The 3D points can be reconstructed using the matching points on the image pair. The reconstructed 3D points are projected onto the image plane again through the camera pose, which is called reprojection.

[0046] Optical flow refers to the instantaneous speed and direction of the pixel movement of a moving object in space on the observation imaging plane;

[0047] Convolutional neural network (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning.

[0048] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0049] See also Figure 1 , is one of the flow diagrams of a method for generating a virtual viewpoint based on camera posture provided by an embodiment of the present invention, the method is applied in a server or a terminal device to generate a virtual viewpoint. Figure 1 As shown, a method for generating a virtual viewpoint based on camera posture provided in this embodiment includes:

[0050] Step 101, constructing a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and the internal parameters and external parameters of two cameras in a camera array corresponding to the adjacent viewpoint images;

[0051] Step 102 , reprojecting each of the three-dimensional points according to the target camera pose and internal parameters of each target camera corresponding to the selected camera pose, to determine a virtual viewpoint image corresponding to the camera pose.

[0052] After reconstructing the three-dimensional points through matching point pairs between two adjacent viewpoint images and internal parameters and external parameters of two cameras corresponding to the two viewpoint images, each of the three-dimensional points is reprojected according to the target camera posture and internal parameters of each target camera corresponding to the selected camera posture to determine the virtual viewpoint image corresponding to the camera posture, so that the reconstructed three-dimensional points can be reprojected based on any camera posture to generate a virtual viewpoint image at the corresponding position, so that virtual viewpoint images at different positions can be generated by selecting different camera postures, and the virtual viewpoint image can be generated without correcting the left and right images, thereby improving the generation efficiency of the virtual viewpoint.

[0053] In step 101, the internal and external parameters of the camera can be obtained by calibrating the camera using the checkerboard image taken by the camera. Specifically, the real world coordinate system is defined in advance using the checkerboard image taken by the camera, and the 3D coordinates of each checkerboard point in the checkerboard image in the world coordinate system are obtained. At the same time, the checkerboard image is subjected to corner point detection to obtain the 2D pixel coordinates of each checkerboard point. After obtaining the 3D coordinates of each checkerboard point in the world coordinate system and the 2D pixel coordinates of each checkerboard point in the plane rectangular coordinate system, the camera that takes the checkerboard image is calibrated based on the relationship between the 3D coordinates and the 2D pixel coordinates of any checkerboard point through the conversion formula x=K[R|T]X (Formula 1), and the internal and external parameters of the camera are obtained.

[0054] Where x is the homogeneous 2D pixel coordinate, X is the homogeneous 3D coordinate, K is the camera intrinsic parameter, and R and T are the rotation and translation in the camera extrinsic parameter respectively.

[0055] In one embodiment, for two adjacent viewpoint images, the two viewpoint images can be matched by image matching to obtain a first matching point of the first viewpoint image in the two viewpoint images and a second matching point of the second viewpoint image in the two viewpoint images that matches the first matching point of the first viewpoint image, and the first matching point and the second matching point are used as a matching point pair.

[0056] Considering that for adjacent viewpoint images captured by a camera array, since multiple cameras are difficult to place horizontally, the optical axes are not necessarily parallel, and there is both horizontal parallax and vertical parallax between the captured viewpoint images. Therefore, in order to obtain accurate matching point pairs, in one embodiment, before constructing the three-dimensional points corresponding to the matching point pairs based on the matching point pairs between adjacent viewpoint images and the internal parameters and external parameters of the two cameras corresponding to the adjacent viewpoint images, it also includes:

[0057] Extracting a first matching point from a first viewpoint image of two adjacent viewpoint images;

[0058] Extracting, according to the trained optical flow estimation model, a second matching point matching the first matching point from a second viewpoint image of two adjacent viewpoint images;

[0059] A matching point pair is formed according to the first matching point and the second matching point.

[0060] In one embodiment, a first matching point is first extracted from the first viewpoint image, and then an optical flow estimation model is used to perform matching estimation between viewpoints, and a second matching point matching the first matching point is extracted from the second viewpoint image. Considering that the convolutional neural network has high accuracy and efficiency in optical flow estimation, the optical flow estimation model can be an optical flow estimation model obtained by training a convolutional neural network.

[0061] Exemplarily, the optical flow estimation model may be a RAFT model. Since RAFT uses a recurrent neural network for multiple iterations, the balance between the calculation speed of the matching point pairs and the prediction performance requirements can be met by increasing or decreasing the number of iterations.

[0062] The optical flow estimation model is used to extract two mutually matching matching points from two adjacent viewpoint images to form matching point pairs, thereby reducing the problem of inaccurate matching point pairs obtained due to horizontal and vertical parallax between adjacent viewpoint images, improving the accuracy and efficiency of the obtained matching point pairs, and further improving the accuracy of the virtual viewpoint images subsequently determined using the matching point pairs.

[0063] In one embodiment, after obtaining the matching point pairs of two adjacent viewpoint images and the corresponding internal parameters and external parameters of two cameras, the position of the three-dimensional point can be obtained through triangulation.

[0064] For example, after knowing the internal and external parameters of the two cameras and the first matching point p1 and the second matching point p2 in the matching point pair, the positions O1 and O2 of the two cameras obtained by the internal and external parameters, as well as the first matching point p1 and the second matching point p2, can be used to obtain the light directions O1p1 and O2p2 corresponding to the two cameras. If all parameter calculations are absolutely accurate, the two light rays O1p1 and O2p2 will intersect at a point P in the three-dimensional space, such as Figure 2 As shown. Then the point P is the three-dimensional point corresponding to the matching point pair.

[0065] However, considering that there may be errors in camera calibration or optical flow estimation, O1p1 and O2p2 may not intersect due to the errors, so that the corresponding three-dimensional points cannot be obtained, affecting the subsequent generation of virtual viewpoint images. To this end, in one embodiment, the three-dimensional points corresponding to the matching point pairs between two adjacent viewpoint images and the internal parameters and external parameters of the two cameras corresponding to the adjacent viewpoint images are constructed, including:

[0066] Determine two light directions corresponding to the two cameras according to the matching point pairs between the two adjacent viewpoint images and the internal parameters and external parameters of the two cameras corresponding to the adjacent viewpoint images;

[0067] The two light directions are processed according to direct linear transformation to construct three-dimensional points corresponding to the matching point pairs.

[0068] In one embodiment, after determining the two light rays O1p1 and O2p2, it is detected whether the light rays O1p1 and O2p2 intersect. If they intersect, the intersection point is directly used as a three-dimensional point. If they do not intersect, the nearest 3D point can be solved by a direct linear change method as a reconstructed 3D point. The specific implementation can be solved by the triangulatePoints function in the opencv library, thereby constructing a three-dimensional point corresponding to the matching point pair.

[0069] When it is determined that two light rays of two cameras corresponding to two adjacent viewpoint images do not intersect, three-dimensional points are constructed by processing the directions of the two light rays through direct linear changes, thereby avoiding the situation where the three-dimensional points cannot be constructed due to camera calibration errors or matching point errors, and further improving the generation efficiency of subsequent virtual viewpoint images.

[0070] In step 102, after obtaining each three-dimensional point, the target camera posture and internal parameters of each corresponding target camera can be determined from the camera array based on any selected camera posture, and the target camera posture, that is, the external parameters and the internal parameters are reprojected to each three-dimensional point through the above formula 1 to obtain a virtual viewpoint image at the position corresponding to the given camera posture.

[0071] Among them, any selected camera posture may be each target camera in a horizontal position, that is, each target camera for horizontal shooting, or each target camera in a circular posture, and so on.

[0072] In particular, when the selected camera posture is a horizontal position, since the positions of the target cameras for horizontal shooting may not be on the same horizontal line, the optical axes are not parallel, such as Figure 3 As shown. The resulting virtual viewpoint image may not be accurate enough. To this end, in one embodiment, according to the target camera posture and internal parameters of each target camera corresponding to the selected camera posture, each of the three-dimensional points is reprojected to determine the virtual viewpoint image corresponding to the camera posture, including:

[0073] When the selected camera posture is a horizontal position, performing camera position leveling on each of the target cameras in the camera array according to the three-dimensional coordinates of each of the target cameras, and obtaining the leveling coordinates of each of the target cameras;

[0074] Determining the target camera posture of each target camera according to the leveling coordinates of each target camera and the leveling rotation angle of each target camera obtained by averaging the rotation angles of each target camera;

[0075] According to the target camera posture and internal parameters of each target camera, each of the three-dimensional points is reprojected to determine a virtual viewpoint image corresponding to the camera posture.

[0076] In one embodiment, when the selected camera posture is a horizontal position, the three-dimensional coordinates of each target camera can be determined according to the internal parameters and external parameters of each target camera in the horizontal position. The vertical coordinates in the three-dimensional coordinates of each target camera are averaged to obtain an average vertical coordinate, and the horizontal coordinates and vertical coordinates in the three-dimensional coordinates of each target camera are linearly fitted according to the least squares method. Since each target camera is a camera in a horizontal position, the horizontal coordinate of each camera remains unchanged, and the new vertical coordinate of each target camera can be obtained according to the least squares method. According to the horizontal coordinate of the target camera, the new vertical coordinate and the average vertical coordinate, the leveling coordinates of the target camera are obtained.

[0077] For example, assuming that there are 5 target cameras, for the vertical coordinate Z, assuming that the Z coordinates of the 5 camera positions are z1-z5 respectively, then the average vertical coordinate of the 5 target virtual cameras is:

[0078]

[0079] For the X and Y directions, the least squares method is used for straight line fitting, and the error function is established as follows:

[0080]

[0081] Where a and b are the slope and intercept of the line, and the equation of the line can be obtained after solving. The original X coordinate of each target camera remains unchanged, and the new Y coordinate can be obtained through the equation of the line. Then the leveled coordinates of each target camera after position leveling are

[0082] Due to the different rotations of different cameras, the optical axes of the target cameras may not be parallel. In order to level the optical axes, the rotation matrix in the external parameters of each target camera can be converted into a rotation angle. Specifically, assuming that the rotation matrix R of the target camera is:

[0083]

[0084] The three rotation angles of the target camera are:

[0085] θ x =atan2(r 32,r 33 )

[0086]

[0087] θ z =atan2(r 21 ,r 11 )

[0088] After getting the rotation angle of each target camera After that, the rotation angles of all target cameras are averaged to obtain the leveling rotation angle of each target camera: Where i represents the serial number of the target camera.

[0089] After obtaining the leveling coordinates and leveling rotation angles of each target camera, each target camera can be leveled as a whole to determine the target camera posture of each target camera, such as Figure 4 shown.

[0090] After the target camera posture of each target camera is obtained, each three-dimensional point can be reprojected according to the target camera posture and internal parameters of each target camera to determine the virtual viewpoint image.

[0091] By leveling the camera position and optical axis of each target camera before reprojecting the three-dimensional points, it is avoided that when reprojecting the horizontally shooting camera to generate a virtual viewpoint image, the generated virtual viewpoint image is inaccurate due to the position of each camera not being on the same horizontal line and / or the optical axes not being parallel, thereby improving the quality of the generated virtual viewpoint image.

[0092] In one embodiment, reprojecting each of the three-dimensional points to determine a virtual viewpoint image corresponding to the camera posture includes:

[0093] Reprojecting each of the three-dimensional points using two adjacent viewpoint images as reference images to determine two adjacent initial virtual viewpoint images corresponding to the camera posture;

[0094] Fill the holes in the left image of the adjacent initial virtual viewpoint image to generate a virtual viewpoint image.

[0095] In actual practice, it is found that the virtual viewpoint image obtained after reprojection may have holes. In order to improve the quality of the virtual viewpoint image, the holes can be filled to improve the quality of the virtual viewpoint image.

[0096] In one embodiment, for a hole with a larger area, bidirectional fusion may be used to fill it. Specifically, the hole filling of the left image in the adjacent initial virtual viewpoint image includes:

[0097] Acquire a first hole position in the left image of the adjacent initial virtual viewpoint image, where the hole area is larger than a preset area;

[0098] Pixels corresponding to the first hole position are obtained from the right image and filled into the first hole position.

[0099] Since the reference image includes two adjacent viewpoint images during reprojection, the image obtained after reprojection includes the left image V l And right figure V r Two adjacent initial virtual viewpoint images. l and V r The left and right images in the reference image are translated respectively. The directions of the holes are inconsistent. Therefore, when there is a large hole position, V r The corresponding pixel fill V l After the hole position on the top is filled, the V l As a virtual viewpoint image.

[0100] In one embodiment, it further includes:

[0101] Acquire a second hole position in the left image whose hole area is less than or equal to the preset area;

[0102] The second hole position is filled by a closing operation.

[0103] In one embodiment, small holes may be directly filled by a closing operation to obtain a virtual viewpoint image.

[0104] The following is a description of a virtual viewpoint generation device based on camera posture provided in an embodiment of the present application. The virtual viewpoint generation device based on camera posture described below and the virtual viewpoint generation method based on camera posture described above can refer to each other.

[0105] In one embodiment, if Figure 5 As shown, a virtual viewpoint generation device based on camera posture is provided, comprising:

[0106] A three-dimensional point construction module 210 is used to construct a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and the internal parameters and external parameters of two cameras in the camera array corresponding to the adjacent viewpoint images;

[0107] The virtual viewpoint generation module 220 is used to reproject each of the three-dimensional points according to the target camera posture and internal parameters of each target camera corresponding to the selected camera posture, and determine the virtual viewpoint image corresponding to the camera posture.

[0108] In one embodiment, the 3D point construction module 210 is further configured to:

[0109] Extracting a first matching point from a first viewpoint image of two adjacent viewpoint images;

[0110] Extracting, according to the trained optical flow estimation model, a second matching point matching the first matching point from a second viewpoint image of two adjacent viewpoint images;

[0111] A matching point pair is formed according to the first matching point and the second matching point.

[0112] In one embodiment, the three-dimensional point construction module 210 is specifically used for:

[0113] Determine two light directions corresponding to the two cameras according to the matching point pairs between the two adjacent viewpoint images and the internal parameters and external parameters of the two cameras corresponding to the adjacent viewpoint images;

[0114] The two light directions are processed according to direct linear transformation to construct three-dimensional points corresponding to the matching point pairs.

[0115] In one embodiment, the virtual viewpoint generation module 220 is specifically used for:

[0116] When the selected camera posture is a horizontal position, performing camera position leveling on each of the target cameras in the camera array according to the three-dimensional coordinates of each of the target cameras, and obtaining the leveling coordinates of each of the target cameras;

[0117] Determining the target camera posture of each target camera according to the leveling coordinates of each target camera and the leveling rotation angle of each target camera obtained by averaging the rotation angles of each target camera;

[0118] According to the target camera posture and internal parameters of each target camera, each of the three-dimensional points is reprojected to determine a virtual viewpoint image corresponding to the camera posture.

[0119] In one embodiment, the virtual viewpoint generation module 220 is specifically used for:

[0120] Reprojecting each of the three-dimensional points using two adjacent viewpoint images as reference images to determine two adjacent initial virtual viewpoint images corresponding to the camera posture;

[0121] Fill the holes in the left image of the adjacent initial virtual viewpoint image to generate a virtual viewpoint image.

[0122] In one embodiment, the virtual viewpoint generation module 220 is specifically used for:

[0123] Acquire a first hole position in the left image of the adjacent initial virtual viewpoint image, where the hole area is larger than a preset area;

[0124] Pixels corresponding to the first hole position are obtained from the right image and filled into the first hole position.

[0125] In one embodiment, the virtual viewpoint generation module 220 is further used for:

[0126] Acquire a second hole position in the left image whose hole area is less than or equal to the preset area;

[0127] The second hole position is filled by a closing operation.

[0128] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call a computer program in the memory 830 to execute the steps of the method for generating a virtual viewpoint based on a camera posture, for example including:

[0129] Constructing a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and internal parameters and external parameters of two cameras in a camera array corresponding to the adjacent viewpoint images;

[0130] According to the target camera posture and internal parameters of each target camera corresponding to the selected camera posture, each of the three-dimensional points is reprojected to determine a virtual viewpoint image corresponding to the camera posture.

[0131] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0132] On the other hand, an embodiment of the present application further provides a computer program product, the computer program product including a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can perform the steps of the method for generating a virtual viewpoint based on a camera posture provided in the above embodiments, for example, including:

[0133] Constructing a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and internal parameters and external parameters of two cameras in a camera array corresponding to the adjacent viewpoint images;

[0134] According to the target camera posture and internal parameters of each target camera corresponding to the selected camera posture, each of the three-dimensional points is reprojected to determine a virtual viewpoint image corresponding to the camera posture.

[0135] On the other hand, an embodiment of the present application further provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, wherein the computer program is used to enable the processor to execute the steps of the methods provided in the above embodiments, for example, including:

[0136] Constructing a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and internal parameters and external parameters of two cameras in a camera array corresponding to the adjacent viewpoint images;

[0137] According to the target camera posture and internal parameters of each target camera corresponding to the selected camera posture, each of the three-dimensional points is reprojected to determine a virtual viewpoint image corresponding to the camera posture.

[0138] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0139] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a virtual viewpoint based on camera posture, characterized in that: include: Constructing a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and internal parameters and external parameters of two cameras in a camera array corresponding to the adjacent viewpoint images; Reprojecting each of the three-dimensional points according to the target camera pose and internal parameters of each target camera corresponding to the selected camera pose, and determining a virtual viewpoint image corresponding to the camera pose; The method of reprojecting each of the three-dimensional points according to the target camera pose and internal parameters of each target camera corresponding to the selected camera pose, and determining a virtual viewpoint image corresponding to the camera pose, comprises: When the selected camera posture is a horizontal position, performing camera position leveling on each of the target cameras in the camera array according to the three-dimensional coordinates of each of the target cameras, and obtaining the leveling coordinates of each of the target cameras; Determining the target camera posture of each target camera according to the leveling coordinates of each target camera and the leveling rotation angle of each target camera obtained by averaging the rotation angles of each target camera; Reprojecting each of the three-dimensional points according to the target camera posture and internal parameters of each of the target cameras to determine a virtual viewpoint image corresponding to the camera posture; Reprojecting each of the three-dimensional points to determine a virtual viewpoint image corresponding to the camera posture includes: Reprojecting each of the three-dimensional points using two adjacent viewpoint images as reference images to determine two adjacent initial virtual viewpoint images corresponding to the camera posture; Fill the holes in the left image of the adjacent initial virtual viewpoint image to generate a virtual viewpoint image.

2. The method for generating a virtual viewpoint based on camera posture according to claim 1, characterized in that: Before constructing the three-dimensional points corresponding to the matching point pairs between adjacent viewpoint images and the internal parameters and external parameters of two cameras corresponding to the adjacent viewpoint images, the method further includes: Extracting a first matching point from a first viewpoint image of two adjacent viewpoint images; Extracting, according to the trained optical flow estimation model, a second matching point matching the first matching point from a second viewpoint image of two adjacent viewpoint images; A matching point pair is formed according to the first matching point and the second matching point.

3. The method for generating a virtual viewpoint based on camera posture according to claim 1, characterized in that: The step of constructing a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and internal parameters and external parameters of two cameras corresponding to the adjacent viewpoint images comprises: Determine two light directions corresponding to the two cameras according to the matching point pairs between the two adjacent viewpoint images and the internal parameters and external parameters of the two cameras corresponding to the adjacent viewpoint images; The two light directions are processed according to direct linear transformation to construct three-dimensional points corresponding to the matching point pairs.

4. The method for generating a virtual viewpoint based on camera posture according to claim 1, characterized in that: The step of filling the holes in the left image of the adjacent initial virtual viewpoint image comprises: Acquire a first hole position in the left image of the adjacent initial virtual viewpoint image, where the hole area is larger than a preset area; Obtain pixels corresponding to the first hole position from the right image and fill them into the first hole position.

5. The method for generating a virtual viewpoint based on camera posture according to claim 4, characterized in that: Also includes: Acquire a second hole position in the left image whose hole area is less than or equal to the preset area; The second hole position is filled by a closing operation.

6. A virtual viewpoint generation device based on camera posture, characterized in that: include: A three-dimensional point construction module, used to construct a three-dimensional point corresponding to the matching point pair between two adjacent viewpoint images and the internal parameters and external parameters of two cameras in the camera array corresponding to the adjacent viewpoint images; A virtual viewpoint generation module, used for reprojecting each of the three-dimensional points according to the target camera pose and internal parameters of each target camera corresponding to the selected camera pose, and determining a virtual viewpoint image corresponding to the camera pose; The method of reprojecting each of the three-dimensional points according to the target camera pose and internal parameters of each target camera corresponding to the selected camera pose, and determining a virtual viewpoint image corresponding to the camera pose, comprises: When the selected camera posture is a horizontal position, performing camera position leveling on each of the target cameras in the camera array according to the three-dimensional coordinates of each of the target cameras, and obtaining the leveling coordinates of each of the target cameras; Determining the target camera posture of each target camera according to the leveling coordinates of each target camera and the leveling rotation angle of each target camera obtained by averaging the rotation angles of each target camera; Reprojecting each of the three-dimensional points according to the target camera posture and internal parameters of each of the target cameras to determine a virtual viewpoint image corresponding to the camera posture; Reprojecting each of the three-dimensional points to determine a virtual viewpoint image corresponding to the camera posture includes: Reprojecting each of the three-dimensional points using two adjacent viewpoint images as reference images to determine two adjacent initial virtual viewpoint images corresponding to the camera posture; Fill the holes in the left image of the adjacent initial virtual viewpoint image to generate a virtual viewpoint image.

7. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, the steps of the method for generating a virtual viewpoint based on camera posture according to any one of claims 1 to 5 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for generating a virtual viewpoint based on camera posture according to any one of claims 1 to 5 are implemented.

Images