Virtual viewpoint image synthesis method and device

Abstract

The invention discloses a virtual viewpoint image synthesis method and device, and relates to the technical field of image synthesis. The image corresponding to the same content in the virtual viewpoint image is generated according to the part, with the same content, of the first viewpoint source image and the second viewpoint source image, and the edge region image prone to distortion in the virtual viewpoint image is generated according to the edge region image of the first viewpoint source image, so that the edge distortion of the virtual viewpoint image can be eliminated; and the quality of the virtual viewpoint image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image synthesis, and particularly to a method and device for synthesizing virtual viewpoint images. Background Art

[0002] The virtual viewpoint synthesis technology is one of the key technologies in 3D video processing, which is used to synthesize the images of virtual viewpoints based on the images of different viewpoints. It is an important research direction in the field of computer vision and has been widely applied in industries such as military, medical equipment, video games, and sports events, promoting the development of the next-generation interactive video technology.

[0003] The known viewpoint source images used for synthesizing virtual viewpoint images usually include two front and back images taken at two positions with different distances between the camera and the shooting target. When using the two front and back images to synthesize the virtual viewpoint image, since some image contents in the image taken at the farther position do not exist in the image taken at the nearer position, distortion occurs at the edge of the synthesized virtual viewpoint image, and the quality of the synthesized virtual viewpoint image is poor. Summary of the Invention

[0004] The present invention solves the technical problem of how to improve the quality of virtual viewpoint images by providing a method and device for synthesizing virtual viewpoint images.

[0005] On the one hand, the present invention provides the following technical solution:

[0006] A method for synthesizing virtual viewpoint images, comprising:

[0007] Obtaining a first viewpoint source image and a second viewpoint source image; the first viewpoint source image is an image taken by an image acquisition device at a first position in a target direction, and the second viewpoint source image is an image taken by the image acquisition device at a second position in the target direction, and the target direction is the direction from the first position to the second position;

[0008] Determining a first region and a second region of the first viewpoint source image according to the second viewpoint source image; the second viewpoint source image contains the image content of the first region and does not contain the image content of the second region;

[0009] Determining each target pixel point that constitutes the virtual viewpoint image according to the first viewpoint source image and the second viewpoint source image;

[0010] Determining each first pixel point among the target pixel points and the pixel value of each first pixel point according to the image of the first region and the second viewpoint source image; the image content constituted by each first pixel point is the same as the image content of the first region;

[0011] Determine the pixel values of each second pixel point among the target pixel points according to the image of the second region; the second pixel point is a pixel point other than the first pixel point among the target pixel points.

[0012] In some embodiments, the determining the first region and the second region of the first viewpoint source image according to the second viewpoint source image includes:

[0013] Determine a scale factor of the second viewpoint source image according to the first viewpoint source image and the second viewpoint source image;

[0014] Scale the second viewpoint source image according to the scale factor;

[0015] Determine the region of the first viewpoint source image that is the same as the scaled second viewpoint source image as the first region, and the region other than the first region as the second region.

[0016] In some embodiments, the determining each target pixel point that constitutes the virtual viewpoint image according to the first viewpoint source image and the second viewpoint source image includes:

[0017] Determine multiple first epipolar lines traversing the first viewpoint source image, multiple second epipolar lines traversing the second viewpoint source image, and respectively determine the one-to-one correspondence between each of the first epipolar lines and each of the second epipolar lines;

[0018] Determine multiple target epipolar lines according to each of the first epipolar lines and each of the second epipolar lines, and each of the first epipolar lines, each of the second epipolar lines, and each of the target epipolar lines corresponds one-to-one;

[0019] Determine the target pixel points as the pixel points on each of the target epipolar lines.

[0020] In some embodiments, the determining each first pixel point among the target pixel points and the pixel value of each of the first pixel points according to the image of the first region and the second viewpoint source image includes:

[0021] Determine each of the first pixel points and the pixel value of each of the first pixel points according to each third pixel point and the pixel points on each of the second epipolar lines; the third pixel point is a pixel point on the first epipolar line and within the first region.

[0022] In some embodiments, the determining each of the first pixel points and the pixel value of each of the first pixel points according to each third pixel point and the pixel points on each of the second epipolar lines includes:

[0023] Dividing each of the third pixel points on the same first pole line into a plurality of consecutive first pixel segments based on the pixel values of the respective third pixel points;

[0024] Dividing each of the pixel points on the second pole line into a plurality of consecutive second pixel segments based on the pixel values of the respective pixel points on the second pole line;

[0025] Determining a one-to-one correspondence between each of the first pixel segments and each of the second pixel segments;

[0026] Determining a first target pixel segment on each of the target pole lines and the pixel values of each of the first target pixel segments according to each of the first pixel segments and each of the second pixel segments;

[0027] Wherein, each of the first pixel segments, each of the second pixel segments, and each of the first target pixel segments correspond one-to-one, and the first pixel point is the pixel point in each of the first target pixel segments.

[0028] In some embodiments, the determining the pixel values of each of the second pixel points in the target pixel points according to the image of the second region includes:

[0029] Obtaining a viewpoint position scale factor corresponding to the virtual viewpoint;

[0030] Determining a third pixel segment on each of the first pole lines according to the viewpoint position scale factor and the pixel segments on each of the first pole lines within the second region;

[0031] Determining the pixel values of each of the second target pixel segments according to the pixel values of each of the third pixel segments;

[0032] Wherein, the second target pixel segment is the pixel segment on each of the target pole lines other than the first target pixel segment, and the second pixel point is the pixel point in each of the second target pixel segments.

[0033] In some embodiments, after the determining the pixel values of each of the second pixel points in the target pixel points according to the image of the second region, the method further includes:

[0034] Taking the weighted average pixel value of the pixel points within a preset range of the crack or hole in the virtual viewpoint image as the pixel value of the crack or the hole.

[0035] On the other hand, the present invention also provides the following technical solution:

[0036] A virtual viewpoint image synthesis device, comprising:

[0037] An acquisition module, configured to acquire a first viewpoint source image and a second viewpoint source image; the first viewpoint source image is an image captured by an image acquisition device at a first position in a target direction, and the second viewpoint source image is an image captured by the image acquisition device at a second position in the target direction, where the target direction is the direction from the first position to the second position;

[0038] A determination module, configured to determine a first region and a second region of the first viewpoint source image according to the second viewpoint source image; the second viewpoint source image includes the image content of the first region and does not include the image content of the second region;

[0039] And configured to determine each target pixel point constituting the virtual viewpoint image according to the first viewpoint source image and the second viewpoint source image;

[0040] And configured to determine each first pixel point among the target pixel points and the pixel value of each first pixel point according to the image of the first region and the second viewpoint source image; the image content formed by each first pixel point is the same as the image content of the first region;

[0041] And configured to determine the pixel value of each second pixel point among the target pixel points according to the image of the second region; the second pixel points are the pixel points among the target pixel points other than the first pixel points.

[0042] On the other hand, the present invention also provides the following technical solution:

[0043] A computer device, including a memory, a processor, and a computer program stored on the memory, where the processor executes the computer program to implement the steps of any one of the virtual viewpoint image synthesis methods described above.

[0044] On the other hand, the present invention also provides the following technical solution:

[0045] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the virtual viewpoint image synthesis methods described above are implemented.

[0046] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0047] The present invention generates an image corresponding to the same content in the virtual viewpoint image according to the same part of the content of the first viewpoint source image and the second viewpoint source image, and generates an image of the edge region that is prone to distortion in the virtual viewpoint image according to the edge region image of the first viewpoint source image, which can eliminate the edge distortion of the virtual viewpoint image and improve the quality of the virtual viewpoint image. Description of the Drawings

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0049] Figure 1 It is a flowchart of the virtual view point image synthesis method in the embodiments of the present invention;

[0050] Figure 2 It is a schematic diagram of the principle of shooting the first view point source image and the second view point source image in the embodiments of the present invention;

[0051] Figure 3 It is a schematic diagram of the principle of the virtual view point image synthesis method in the embodiments of the present invention;

[0052] Figure 4 It is a schematic diagram of the principle of determining the first epipolar line and the second epipolar line in the embodiments of the present invention;

[0053] Figure 5 It is an effect diagram of the traditional virtual view point image synthesis method;

[0054] Figure 6 It is an effect diagram of the virtual view point image synthesis method in the embodiments of the present invention;

[0055] Figure 7 It is a schematic diagram of the virtual view point image synthesis device in the embodiments of the present invention. Detailed implementation manners

[0056] By providing the virtual view point image synthesis method and device in the embodiments of the present invention, the technical problem of how to improve the quality of virtual view point images is solved.

[0057] To better understand the technical solutions of the present invention, the following will describe the technical solutions of the present invention in detail in combination with the accompanying drawings of the specification and specific implementation manners.

[0058] As Figure 1 shown, the virtual view point image synthesis method in the embodiments of the present invention includes:

[0059] Step S1, obtaining a first view point source image and a second view point source image; the first view point source image is an image captured by an image acquisition device in a first position towards a target direction, and the second view point source image is an image captured by the image acquisition device in a second position towards the target direction, and the target direction is the direction from the first position to the second position;

[0060] As Figure 2As shown, the first position is a position farther from the shooting target, and the second position is a position closer to the shooting target. Therefore, the first viewpoint source image is a rear-viewpoint source image, and the second viewpoint source image is a front-viewpoint source image. It can be understood that the scene captured by the image acquisition device at the first position is wider, and the scene captured at the second position is smaller. As a result, the image content at the edge of the first viewpoint source image does not exist in the second viewpoint source image. In some embodiments, the image acquisition device may be a camera;

[0061] Step S2: Determine a first region and a second region of the first viewpoint source image according to the second viewpoint source image; the second viewpoint source image includes the image content of the first region and does not include the image content of the second region;

[0062] Among them, the second region is also the edge region of the first viewpoint source image. In some embodiments, step S2 may include: determining a scaling factor of the second viewpoint source image according to the first viewpoint source image and the second viewpoint source image; scaling the second viewpoint source image according to the scaling factor; determining the region of the first viewpoint source image that is the same as the scaled second viewpoint source image as the first region, and the region outside the first region as the second region;

[0063] Among them, the scaling factor is related to the distance between the first position and the second position. It can be understood that the greater the distance between the first position and the second position, the smaller the range of the first region, the larger the range of the second region, the smaller the second viewpoint source image needs to be scaled, and the larger the scaling factor;

[0064] In some embodiments, determining the scaling factor of the second viewpoint source image according to the first viewpoint source image and the second viewpoint source image may include: performing Fourier transforms on the first viewpoint source image and the second viewpoint source image respectively to obtain two modulus images in rectangular coordinates; using bilinear interpolation to convert the two modulus images in rectangular coordinates into two modulus images in logarithmic polar coordinates; performing phase correlation calculation on the two modulus images in logarithmic polar coordinates to obtain a scaling-down factor;

[0065] It can be understood that if the line connecting the first position and the second position is not exactly the same as the target direction, after scaling the second viewpoint source image according to the scaling factor, it is also necessary to perform phase correlation calculation on the first viewpoint source image and the scaled second viewpoint source image to obtain the translation amount between the first viewpoint source image and the scaled second viewpoint source image, and translate the scaled second viewpoint source image according to the translation amount before determining the first region and the second region;

[0066] The first region and the second region of the first viewpoint source image determined in step S2 are as Figure 3 shown. The first viewpoint source image is composed of the first region and the second region;

[0067] Step S3: Determine each target pixel point that constitutes the virtual view image according to the first-viewpoint source image and the second-viewpoint source image;

[0068] In some embodiments, step S3 may include: determining multiple first epipolar lines that traverse the first-viewpoint source image, multiple second epipolar lines that traverse the second-viewpoint source image, and respectively determining the one-to-one correspondence between each first epipolar line and each second epipolar line; determining multiple target epipolar lines according to each first epipolar line and each second epipolar line, with each first epipolar line, each second epipolar line, and each target epipolar line corresponding one-to-one; determining the target pixel points as the pixel points on each target epipolar line;

[0069] In some embodiments, determining each second epipolar line (or first epipolar line) may include: solving the fundamental matrix of the first-viewpoint source image and the second-viewpoint source image to obtain the epipole; taking the epipole of the second-viewpoint source image as the center of a circle to draw a circle that contains the second-viewpoint source image. As Figure 4 shown, the radius of the circle is r. Draw a line that makes an angle of 1 / r with the horizontal line through the epipole. The part of this line within the second-viewpoint source image is the first second epipolar line; then, with a step size of 1 / r, make the angle between the second epipolar line and the horizontal line step from 1 / r to π, and all second epipolar lines that traverse the second-viewpoint source image can be obtained;

[0070] According to Figure 4 similarly, all first epipolar lines that traverse the first-viewpoint source image can be obtained;

[0071] In some embodiments, determining the one-to-one correspondence between each first epipolar line and each second epipolar line may include: determining the intersection point of the extension line of the second epipolar line and the Figure 4 circle in. According to the epipolar geometry constraint relationship, the corresponding point of this intersection point must be on the corresponding first epipolar line. Therefore, further according to the overall matching property of the epipolar lines, the first epipolar line corresponding to this second epipolar line can be determined;

[0072] In some embodiments, interpolation calculations may be performed on the positions of the first epipolar line and the corresponding second epipolar line to obtain the position of the corresponding target epipolar line, as Figure 3 shown;

[0073] Step S4: Determine each first pixel point among the target pixel points and the pixel value of each first pixel point according to the image of the first region and the second-viewpoint source image; the image content formed by each first pixel point is the same as the image content of the first region;

[0074] In some embodiments, step S4 may include: determining each first pixel point and the pixel value of each first pixel point according to each third pixel point and the pixel points on each second epipolar line; the third pixel point is the pixel point on the first epipolar line that is within the first region;

[0075] In some embodiments, in step S4, determining each first pixel point and the pixel value of each first pixel point according to each third pixel point and the pixel points on each second pole line may include: dividing each third pixel point on the same first pole line into multiple consecutive first pixel segments based on the pixel values of each third pixel point; dividing each pixel point on the second pole line into multiple consecutive second pixel segments based on the pixel values of the pixel points on the second pole line; determining a one-to-one correspondence between each first pixel segment and each second pixel segment; determining the first target pixel segment on each target pole line and the pixel value of each first target pixel segment according to each first pixel segment and each second pixel segment; wherein, each first pixel segment, each second pixel segment, and each first target pixel segment correspond one-to-one, and the first pixel point is the pixel point in each first target pixel segment;

[0076] Wherein, the difference between the pixel value of each pixel point in the first pixel segment and the pixel average value of the first pixel segment is lower than a preset threshold, and the difference between the pixel value of each pixel point in the second pixel segment and the pixel average value of the second pixel segment is lower than a preset threshold. Therefore, the pixel values of all pixel points in each first pixel segment are close, and the pixel values of all pixel points in each second pixel segment are close; for example, the first pixel point on the second pole line belongs to the first second pixel segment. Suppose the pixel value of the first pixel point on the second pole line is f 1 , if the pixel value f 2 of the second pixel point on the second pole line satisfies (where T is the preset threshold), then the second pixel point on the second pole line also belongs to the first second pixel segment; if the pixel value f 3 of the third pixel point on the second pole line does not satisfy , then the third pixel point on the second pole line belongs to the second second pixel segment;

[0077] In some embodiments, in step S4, determining the one-to-one correspondence between each first pixel segment and each second pixel segment may include: determining the second pixel segment corresponding to each first pixel segment according to the dynamic programming matching algorithm;

[0078] Let the first pixel point, the last pixel point of each first pixel segment, the first pixel point, and the last pixel point of each second pixel segment be boundary points. The principle of the dynamic programming matching algorithm is: if d(c k ) = d(i, j) = ||a i - b j || represents the distance between the boundary point i of the first pixel segment and the boundary point j of the second pixel segment (where a i and b j are the pixel values of the boundary points i and j), and s(c k ) = s(i, j) represents the dot matrix c1 , c 2 , …, c k (c k (which is the sum of (i, j)), the formula of the dynamic programming matching algorithm is;

[0079]

[0080] Set the last boundary points of the first epipolar line and the second epipolar line as the end matching point pair. The dynamic programming matching algorithm will gradually search forward from the end matching point pair for the remaining matching point pairs until the starting boundary point, obtaining the boundary points of the second pixel segment corresponding to the boundary points of each first pixel segment, that is, obtaining the second pixel segment corresponding to each first pixel segment; specifically, set i and j as the matching point pair. If s(i, j) is calculated from s(i, j - 1), then i and j - 1 are the matching point pair; if s(i, j) is calculated from s(i - 1, j - 1), then i - 1 and j - 1 are the matching point pair; if s(i, j) is calculated from s(i - 1, j), then i - 1 and j are the matching point pair;

[0081] In some embodiments, as Figure 3 shown, in step S4, determining the first target pixel segments on each target epipolar line and the pixel values of each first target pixel segment according to each first pixel segment and each second pixel segment may include: According to the formula calculate the position m of the first pixel point and the position n of the last pixel point of the first target pixel segment, m 1 is the position of the first pixel point of the first pixel segment, m 2 is the position of the first pixel point of the corresponding second pixel segment, n 1 is the position of the last pixel point of the first pixel segment, n 2 is the position of the last pixel point of the corresponding second pixel segment, α (0 < α < 1) is the view point position ratio factor corresponding to the virtual view point, and the view point position ratio factors corresponding to different positions of the virtual view point are different; According to the formula V = V 1 * α + V 2 * (1 - α) calculate the pixel value V of the first target pixel segment, V 1 is the pixel value of the first pixel segment, V 2 is the pixel value of the corresponding second pixel segment;

[0082] So far, step S4 can generate the image corresponding to the same content in the virtual view point image according to the same part of the first view point source image and the second view point source image, and the image pixel points corresponding to the same image content have actual pixel values;

[0083] Step S5: Determine the pixel values of each second pixel point in the target pixel points according to the image of the second region; the second pixel points are the pixel points in the target pixel points other than the first pixel point.

[0084] The regions where each second pixel point is located can be regarded as the edge regions where the virtual view image is prone to distortion. Before step S5, each second pixel point has no actual pixel value.

[0085] In some embodiments, step S5 may include: obtaining the view point position scale factor corresponding to the virtual view point; determining the third pixel segments on each first epipolar line according to the view point position scale factor and the pixel segments in the second region on each first epipolar line; determining the pixel values of each second target pixel segment according to the pixel values of each third pixel segment; wherein, the second target pixel segment is the pixel segment other than the first target pixel segment on each target epipolar line, and the second pixel point is the pixel point in each second target pixel segment.

[0086] Among them, the third pixel segment is in the second region, and one end pixel point of the third pixel segment is located at the boundary between the first region and the second region.

[0087] As Figure 3 shown, if the pixel segment in the second region on a certain first epipolar line includes segment ab and segment cd, and the second target pixel segment on the corresponding target epipolar line includes segment gh and segment kl, then the pixel value of segment ae (ae = ab * α) can be assigned to segment gh, and the pixel value of segment cf (cf = cd * α) can be assigned to segment kl, that is, the third pixel segment includes segment ae and segment cf.

[0088] Of course, in some cases, there are cracks and holes in the generated virtual view image, and it is also necessary to fill the cracks and holes. That is, after step S5, the virtual view image synthesis method may further include: taking the pixel weighted average value of the pixel points within the preset range of the cracks or holes in the virtual view image as the pixel value of the cracks or holes. Among them, the pixel points within the preset range may be the pixel points adjacent to the cracks or holes, and the pixel points of the adjacent pixel points must be close to the actual pixel values of the cracks or holes, which can make the image after filling the cracks or holes more in line with the actual situation.

[0089] The first view point source image obtained in an embodiment of the present invention is as Figure 5 and Figure 6 shown in I 0 ; the second view point source image (non-standard second view point source image) is as Figure 5 and Figure 6 shown in I 1 . Figure 5 shown in I 0.2 , I 0.4 , I 0.6 , I0.8 is a virtual view point image generated by a conventional virtual view point image synthesis method, Figure 6 wherein the I in 0.2 、I in 0.4 、I in 0.6 、I in 0.8 is a virtual view point image generated by the virtual view point image synthesis method according to the embodiments of the present invention. It can be seen that, Figure 5 there is an obvious distortion phenomenon in the square area of the virtual view point image in Figure 6 while there is basically no distortion phenomenon in the square area of the virtual view point image in , proving the good effect of the present solution.

[0090] As can be seen from the above, according to the embodiments of the present invention, the virtual view point image is generated based on the same part of the first view point source image and the second view point source image, and the edge area image of the first view point source image is used to generate the edge area image of the virtual view point image that is prone to distortion, which can eliminate the edge distortion of the virtual view point image and improve the quality of the virtual view point image.

[0091] As Figure 7 shown, the embodiments of the present invention further provide a virtual view point image synthesis device, including:

[0092] An acquisition module, configured to acquire a first view point source image and a second view point source image; the first view point source image is an image captured by an image acquisition device in a first position towards a target direction, and the second view point source image is an image captured by the image acquisition device in a second position towards the target direction, and the target direction is the direction from the first position to the second position;

[0093] A determination module, configured to determine a first area and a second area of the first view point source image according to the second view point source image; the second view point source image includes the image content of the first area and does not include the image content of the second area;

[0094] and configured to determine each target pixel point constituting the virtual view point image according to the first view point source image and the second view point source image;

[0095] and configured to determine each first pixel point among the target pixel points and the pixel value of each first pixel point according to the image of the first area and the second view point source image; the image content formed by each first pixel point is the same as the image content of the first area;

[0096] and configured to determine the pixel value of each second pixel point among the target pixel points according to the image of the second area; the second pixel point is a pixel point other than the first pixel point among the target pixel points.

[0097] In some embodiments, the determination module may be specifically configured to:

[0098] Determine the scaling factor of the second viewpoint source image based on the first viewpoint source image and the second viewpoint source image;

[0099] Scale the second viewpoint source image according to the scaling factor;

[0100] Determine that the area in the first viewpoint source image that is the same as the scaled second viewpoint source image is the first area, and the area outside the first area is the second area.

[0101] In some embodiments, the determining module may specifically be configured to:

[0102] Determine multiple first epipolar lines traversing the first viewpoint source image and multiple second epipolar lines traversing the second viewpoint source image, and respectively determine the one-to-one correspondence between each first epipolar line and each second epipolar line;

[0103] Determine multiple target epipolar lines based on each first epipolar line and each second epipolar line, and each first epipolar line, each second epipolar line, and each target epipolar line are in one-to-one correspondence;

[0104] Determine that the target pixel points are the pixel points on each target epipolar line.

[0105] In some embodiments, the determining module may specifically be configured to:

[0106] Determine each first pixel point and the pixel value of each first pixel point based on each third pixel point and the pixel points on each second epipolar line; the third pixel point is the pixel point on the first epipolar line within the first area.

[0107] In some embodiments, the determining module may specifically be configured to:

[0108] Divide the third pixel points on the same first epipolar line into multiple consecutive first pixel segments based on the pixel values of each third pixel point;

[0109] Divide the pixel points on the second epipolar line into multiple consecutive second pixel segments based on the pixel values of the pixel points on the second epipolar line;

[0110] Determine the one-to-one correspondence between each first pixel segment and each second pixel segment;

[0111] Determine the first target pixel segments on each target epipolar line and the pixel values of each first target pixel segment based on each first pixel segment and each second pixel segment;

[0112] Wherein, each first pixel segment, each second pixel segment, and each first target pixel segment are in one-to-one correspondence, and the first pixel point is the pixel point in each first target pixel segment.

[0113] In some embodiments, the determining module may specifically be configured to:

[0114] Obtain the viewpoint position scale factor corresponding to the virtual viewpoint;

[0115] Determine the third pixel segment on each first epipolar line according to the viewpoint position scale factor and the pixel segments in the second region on each first epipolar line;

[0116] Determine the pixel values of each second target pixel segment according to the pixel values of each third pixel segment;

[0117] Wherein, the second target pixel segment is the pixel segment other than the first target pixel segment on each target epipolar line, and the second pixel point is the pixel point in each second target pixel segment.

[0118] In some embodiments, the determining module can also be used for:

[0119] Use the pixel weighted average value of the pixel points within the preset range of the crack or hole in the virtual viewpoint image as the pixel value of the crack or hole.

[0120] Based on the same inventive concept as the virtual viewpoint image synthesis method described above, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of any one of the virtual viewpoint image synthesis methods described above.

[0121] Since the computer device introduced in the embodiment of the present invention is the computer device used to implement the virtual viewpoint image synthesis method in the embodiment of the present invention, based on the virtual viewpoint image synthesis method introduced in the embodiment of the present invention, those skilled in the art can understand the specific implementation manners and various variations of the computer device in the embodiment of the present invention. Therefore, the implementation of how the computer device implements the method in the embodiment of the present invention will not be described in detail here. As long as the computer device used by those skilled in the art to implement the virtual viewpoint image synthesis method in the embodiment of the present invention belongs to the scope protected by the present invention.

[0122] Based on the same inventive concept as the virtual viewpoint image synthesis method described above, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of any one of the virtual viewpoint image synthesis methods described above.

[0123] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0124] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0126] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0127] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims

1. A virtual viewpoint image synthesis method, characterized in that: include: Acquire a first viewpoint source image and a second viewpoint source image; the first viewpoint source image is an image taken by an image acquisition device at a first position toward a target direction, and the second viewpoint source image is an image taken by the image acquisition device at a second position toward the target direction, and the target direction is a direction from the first position to the second position; Determine a first region and a second region of the first viewpoint source image according to the second viewpoint source image; the second viewpoint source image includes image content of the first region and does not include image content of the second region; Determine each target pixel point constituting a virtual viewpoint image according to the first viewpoint source image and the second viewpoint source image; Determine, according to the image of the first area and the second viewpoint source image, each first pixel point in the target pixel points and a pixel value of each first pixel point; The image content formed by each of the first pixel points is the same as the image content of the first area; Determine the pixel value of each second pixel point in the target pixel point according to the image of the second area; The second pixel point is a pixel point among the target pixel points except the first pixel point.

2. The virtual viewpoint image synthesis method according to claim 1, characterized in that: The determining the first area and the second area of ​​the first viewpoint source image according to the second viewpoint source image comprises: Determining a scaling factor of the second viewpoint source image according to the first viewpoint source image and the second viewpoint source image; Scaling the second viewpoint source image according to the scaling factor; It is determined that the area in the first viewpoint source image that is the same as the scaled second viewpoint source image is the first area, and the area outside the first area is the second area.

3. The virtual viewpoint image synthesis method according to claim 1, characterized in that: The determining of each target pixel point constituting a virtual viewpoint image according to the first viewpoint source image and the second viewpoint source image comprises: Determine a plurality of first epipolar lines traversing the first viewpoint source image, a plurality of second epipolar lines traversing the second viewpoint source image, and respectively determine a one-to-one correspondence between each of the first epipolar lines and each of the second epipolar lines; Determine a plurality of target epipolar lines according to each of the first epipolar lines and each of the second epipolar lines, wherein each of the first epipolar lines, each of the second epipolar lines and each of the target epipolar lines correspond to each other one by one; The target pixel points are determined to be pixel points on each of the target epipolar lines.

4. The virtual viewpoint image synthesis method according to claim 3, characterized in that: The determining, according to the image of the first area and the second viewpoint source image, each first pixel point in the target pixel points and the pixel value of each first pixel point comprises: The pixel values ​​of each of the first pixel points and each of the first pixel points are determined according to each of the third pixel points and the pixel points on each of the second polar lines; the third pixel point is a pixel point on the first polar line that is within the first area.

5. The virtual viewpoint image synthesis method according to claim 4, characterized in that: The determining each of the first pixel points and the pixel values ​​of each of the first pixel points according to each of the third pixel points and the pixel points on each of the second epipolar lines comprises: Dividing each of the third pixel points on the same first epipolar line into a plurality of continuous first pixel segments based on the pixel values ​​of each of the third pixel points; dividing each pixel point on the second epipolar line into a plurality of continuous second pixel segments based on the pixel values ​​of each pixel point on the second epipolar line; Determining a one-to-one correspondence between each of the first pixel segments and each of the second pixel segments; Determine a first target pixel segment on each of the target epipolar lines and a pixel value of each of the first target pixel segments according to each of the first pixel segments and each of the second pixel segments; There is a one-to-one correspondence between each of the first pixel segments, each of the second pixel segments and each of the first target pixel segments, and the first pixel points are pixel points in each of the first target pixel segments.

6. The virtual viewpoint image synthesis method according to claim 5, characterized in that: The step of determining the pixel value of each second pixel point in the target pixel point according to the image of the second area includes: Get the viewpoint position scale factor corresponding to the virtual viewpoint; determining a third pixel segment on each of the first epipolar lines according to the viewpoint position scale factor and the pixel segment on each of the first epipolar lines that is within the second region; Determine the pixel value of each second target pixel segment according to the pixel value of each third pixel segment; The second target pixel segments are pixel segments on each of the target epipolar lines except the first target pixel segments, and the second pixel points are pixel points in each of the second target pixel segments.

7. The virtual viewpoint image synthesis method according to claim 1, characterized in that: After determining the pixel value of each second pixel point in the target pixel point according to the image of the second area, the method further includes: The pixel weighted average value of the pixel points within a preset range of the crack or the hole in the virtual viewpoint image is used as the pixel value of the crack or the hole.

8. A virtual viewpoint image synthesis device, characterized in that: include: An acquisition module, used to acquire a first viewpoint source image and a second viewpoint source image; the first viewpoint source image is an image taken by an image acquisition device at a first position toward a target direction, and the second viewpoint source image is an image taken by the image acquisition device at a second position toward the target direction, and the target direction is a direction from the first position to the second position; a determination module, configured to determine a first region and a second region of the first viewpoint source image according to the second viewpoint source image; the second viewpoint source image includes image content of the first region and does not include image content of the second region; and used to determine each target pixel point constituting a virtual viewpoint image according to the first viewpoint source image and the second viewpoint source image; and used to determine each first pixel point in the target pixel point and the pixel value of each first pixel point according to the image of the first area and the second viewpoint source image; The image content formed by each of the first pixel points is the same as the image content of the first area; and for determining the pixel value of each second pixel point in the target pixel point according to the image of the second area; The second pixel point is a pixel point among the target pixel points except the first pixel point.

9. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images