Method and apparatus, medium, and device for generating multi-angle free-viewpoint image data

Abstract

Embodiments of the present application provide a method and apparatus, medium, and device for generating multi-angle free-viewpoint data. The method includes: obtaining a plurality of synchronized images with different viewpoints; determining a virtual path, where the virtual path is a set of viewpoints for viewing a region to be viewed; preprocessing the plurality of images according to the virtual path and the viewpoint of each image in the plurality of images to obtain a set of preprocessed images that is consistent with the virtual path; and generating multi-angle free-viewpoint data based on the set of preprocessed images. The technical solution in the present application is beneficial to ensuring the image quality when adjusting the viewing angle for viewing.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular, to a method and apparatus, medium, and device for generating multi-angle degree-of-freedom image data. Background Art

[0002] Image data is data that supports image display for users to view. In a method for generating image data, the image data only supports users to view from one perspective, and users cannot adjust the viewing perspective, so the user experience needs to be improved.

[0003] One way to provide multi-angle viewing is to take multiple images with different shooting perspectives by multiple cameras, and perform image reconstruction based on the multiple images to meet the needs of users to adjust the viewing perspective. The data basis for image reconstruction is called multi-angle free-viewpoint data.

[0004] In the existing methods for adjusting the viewing perspective to view, the image quality during viewing is difficult to guarantee. Summary of the Invention

[0005] In view of this, an embodiment of the present application provides a method for generating multi-angle free-viewpoint data, including:

[0006] Obtaining a plurality of synchronized images with different viewpoints;

[0007] Determining a virtual path, where the virtual path is a set of viewpoints for viewing a region to be viewed;

[0008] Preprocessing the plurality of images according to the virtual path and the viewpoint of each image in the plurality of images to obtain a set of preprocessed images that is consistent with the virtual path;

[0009] Generating multi-angle free-viewpoint data based on the set of preprocessed images.

[0010] Optionally, the virtual path includes a basic path and viewpoints obtained by performing focal length transformation on the viewpoints on the basic path, and the basic path is fitted based on the camera parameters of each image in the plurality of synchronized images.

[0011] Optionally, the viewpoint includes a viewpoint position and a viewpoint angle; preprocessing the plurality of images includes:

[0012] Mapping the viewpoint position of each image in the plurality of images to the viewpoint position in the virtual path;

[0013] Determining a target angle corresponding to the viewpoint position of each image in the plurality of images according to the correspondence between the viewpoint position and the viewpoint angle in the virtual path;

[0014] Based on the viewing angles and target angles of each of the multiple images, obtain an image that is consistent with the corresponding target angle.

[0015] Optionally, the obtaining an image that is consistent with the corresponding target angle based on the viewing angles and target angles of each of the multiple images includes:

[0016] Determine the images to be adjusted among the multiple images whose viewing angles are different from the target angle;

[0017] Interpolate the images to be adjusted according to the corresponding target angles.

[0018] Optionally, interpolating the images to be adjusted according to the corresponding target angles includes:

[0019] Determine the depth map and texture map of the images to be adjusted;

[0020] Interpolate the texture map of the images to be adjusted;

[0021] Interpolate the depth map of the images to be adjusted.

[0022] Optionally, interpolating the texture map of the images to be adjusted includes: Interpolating the texture map using an interpolation algorithm with a filtering template.

[0023] Optionally, interpolating the depth map of the images to be adjusted includes: Interpolating the depth map using a nearest neighbor interpolation algorithm.

[0024] Optionally, the preprocessing further includes:

[0025] Determine a cropping range according to the intersection of the valid pixel ranges in each of the images that are consistent with the corresponding target angles, where the valid pixel range is a region without holes;

[0026] Crop the images that are consistent with the corresponding target angles according to the cropping range.

[0027] Optionally, the obtaining an image that is consistent with the corresponding target angle based on the viewing angles and target angles of each of the multiple images includes:

[0028] Determine the images to be adjusted among the multiple images whose viewing angles are different from the target angle;

[0029] Perform image reconstruction on the images to be adjusted according to the viewing positions in the mapped virtual path and the corresponding target angles.

[0030] Optionally, the multiple synchronized images are images corresponding to the same frame moment in video data; generating multi-angle free-viewpoint data based on the preprocessed image set includes: storing the depth map and texture map corresponding to the processed images in the order of frame moments.

[0031] Optionally, before generating multi-angle free-viewpoint data based on the preprocessed image set, it further includes: determining the background part in the multiple synchronized images; replacing the background part in the multiple synchronized images.

[0032] The embodiment of the present application further provides a multi-angle free-viewpoint data generation device, including:

[0033] A synchronized image acquisition unit, adapted to acquire multiple synchronized images with different viewpoints;

[0034] A virtual path determination unit, adapted to determine a virtual path, where the virtual path is a set of viewpoints for viewing the area to be viewed;

[0035] A preprocessing unit, adapted to preprocess the multiple images according to the virtual path and the viewpoint of each image in the multiple images to obtain a preprocessed image set that is consistent with the virtual path;

[0036] A data generation unit, adapted to generate multi-angle free-viewpoint data based on the preprocessed image set.

[0037] The embodiment of the present application further provides an interaction method based on multi-angle free-viewpoint data, including:

[0038] Receiving a user's operation;

[0039] Performing image display based on the multi-angle free-viewpoint data according to the user's operation, where the multi-angle free-viewpoint data is generated based on an image set that is consistent with a virtual path after processing multiple synchronized images, the virtual path is a set of viewpoints for viewing the area to be viewed, and the viewpoints of the multiple synchronized images are different.

[0040] Optionally, receiving the user's operation through a terminal device and performing the image display on a display, where the screen of the display is larger than the screen of the terminal device.

[0041] Optionally, the receiving the user's operation includes:

[0042] Receiving the selection of a target object in the image displayed based on the multi-angle free-viewpoint data by the user; performing image display based on the multi-angle free-viewpoint data according to the user's operation includes:

[0043] Obtain a virtual information image generated based on the augmented reality special effect input data of the target object;

[0044] Perform compositing processing on the virtual information image and the corresponding image and display it.

[0045] An embodiment of the present application further provides a computer device, including a memory and a processor. A computer program that can run on the processor is stored on the memory. When the processor runs the computer program, it executes the multi-angle free-viewpoint data generation method described above.

[0046] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program runs, it executes the multi-angle free-viewpoint data generation method.

[0047] In the technical solution of the embodiment of the present application, by preprocessing a plurality of synchronized images, a set of preprocessed images that is consistent with the virtual path is obtained, and further multi-angle free-viewpoint data is obtained based on the set of processed images. Thus, the consistency between the multi-angle free-viewpoint data and the viewing viewpoints can be improved, and further the image quality during image reconstruction based on the multi-angle free-viewpoint data can be improved.

[0048] Furthermore, determining the virtual path according to the fitting result of the camera parameters for shooting a plurality of images can make the virtual path better consistent with the plurality of synchronized images, improve the utilization rate of the plurality of synchronized images, and further improve the reconstructed image quality when viewing the area to be viewed. Description of the Drawings

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for description in the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0050] Figure 1 It is a schematic diagram of a multi-angle free-viewpoint display system;

[0051] Figure 2a and Figure 2b It is two schematic diagrams of the device display for display;

[0052] Figure 3 It is a schematic flowchart of a multi-angle free-viewpoint data generation method in an embodiment of the present application;

[0053] Figure 4 In the embodiment of the present application Figure 3Flow diagram of a specific implementation of step S33 shown;

[0054] Figure 5 Schematic diagram for mapping the viewpoint position of a mapped image to the viewpoint position in the virtual path in an embodiment of the present application;

[0055] Figure 6 In an embodiment of the present application Figure 4 Flow diagram of a specific implementation of step S43 shown;

[0056] Figure 7 In an embodiment of the present application Figure 3 Partial flow diagram of another specific implementation of step S33 shown;

[0057] Figure 8 Schematic of the pre - interpolation texture map of an image to be adjusted in an embodiment of the present application;

[0058] Figure 9 Schematic of the post - interpolation texture map of an image to be adjusted in an embodiment of the present application;

[0059] Figure 10 Schematic of a post - interpolation hole area in an embodiment of the present application;

[0060] Figure 11 Another schematic of a post - interpolation hole area in an embodiment of the present application;

[0061] Figure 12 Another schematic of a post - interpolation hole area in an embodiment of the present application;

[0062] Figure 13 Another schematic of a post - interpolation hole area in an embodiment of the present application;

[0063] Figure 14 Another schematic of a post - interpolation hole area in an embodiment of the present application;

[0064] Figure 15 In an embodiment of the present application Figures 10 to 14 Schematic of the intersection of non - hole areas shown;

[0065] Figure 16 Schematic of a cropping range determined according to the intersection of non - hole areas shown in 15 in an embodiment of the present application

[0066] Figure 17 Schematic of the structure of a multi - angle free - view data generation device in an embodiment of the present application;

[0067] Figure 18 Schematic of the playback effect of a terminal device in an embodiment of the present application. Detailed implementation mode

[0068] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0069] Multi-angle free-viewpoint data is data that can support users to switch viewpoints for viewing.

[0070] The multi-angle free-viewpoint data may include texture maps collected by multiple cameras, all camera parameters, and depth maps of each camera. The generation process of the multi-angle free-viewpoint data usually includes multi-camera video capturing, camera parameter estimation, and depth map calculation.

[0071] Based on the multi-angle free-viewpoint data and in combination with the user's instructions on the viewing position and viewing angle, the user terminal can perform image reconstruction in the form of depth image-based rendering (DIBR) to obtain a reconstructed image corresponding to the user's instructions, thereby meeting the user's needs for viewing at different positions and angles.

[0072] Among them, the viewing position and viewing angle, which can also be called the viewpoint, can be represented by six parameters. Among them, three parameters represent the viewing position or the viewpoint position, and three parameters represent the viewing angle or the viewpoint angle. The shooting position and shooting angle (corresponding to the camera pose) in the camera parameters of the shooting device can also represent these six degrees of freedom. For a captured image, the shooting position and shooting angle of the corresponding shooting device can also be called the viewpoint of the image. Similarly, the viewing position and viewing angle indicated by the user can be called the virtual viewpoint and can also be represented by these six dimensions. From this perspective, the multi-angle free-viewpoint data described above can also be called six-degree-of-freedom video data (6DoF video data).

[0073] Figure 1 It is a schematic diagram of a multi-angle free-viewpoint display system.

[0074] Combined with reference to Figure 1 , in the embodiment of the present invention, a data acquisition system 11 including multiple acquisition devices can be used to acquire data of the area to be viewed, where the acquisition device can be a camera CJ1 to CJ 6 ; The acquisition system 11 or the server 12 can process the acquired multiple synchronized images to generate multi-angle free-view data that can support the virtual viewpoint switching of the display device 13. The display device 13 can display a reconstructed image generated based on the multi-angle free-view data. The reconstructed image corresponds to a virtual viewpoint. According to the user's instruction, the reconstructed images corresponding to different virtual viewpoints can be displayed, and the viewing position and viewing angle can be switched. For example, with reference to Figure 2a and Figure 2b , the display device 13 can also display images viewed from different perspectives.

[0075] Continuing to refer to Figure 1 , in a specific implementation, the process of image reconstruction to obtain the reconstructed image can be implemented by the display device 13 or by a device located in the Content Delivery Network (CDN) in an edge computing manner. It can be understood that Figure 1 This is only an example and does not limit the acquisition system, the server, the display device, and the specific implementation manner.

[0076] In the technical solution of image reconstruction based on multi-angle free-view data to provide multi-viewpoint viewing, the quality of the reconstructed image needs to be improved.

[0077] With reference to Figure 1 , the multi-angle free-view data is acquired by the acquisition system 11. These images can be frame images in a video. An acquisition device for array shooting needs to be set in the acquisition system 11, such as an array of cameras or video cameras.

[0078] In the existing multi-angle viewing based on multi-angle free-view data, the image quality needs to be improved, and the cost of the acquisition system is relatively high.

[0079] After further research by the applicant, it is found that the position and angle of the acquisition devices in the acquisition system have a great impact on the quality of the reconstructed image. If the consistency between the position and angle of the virtual viewpoint corresponding to the reconstructed image and each acquisition device in the acquisition system is poor, a hole effect will occur in the reconstructed image.

[0080] The embodiment of the present application further provides a method for generating multi-angle free-view data. With reference to Figure 3 , it may include the following steps:

[0081] Step S31, acquiring multiple synchronized images with different viewpoints

[0082] Step S32, determining a virtual path, where the virtual path is the viewpoint for viewing the area to be viewed

[0083] Set;

[0084] Step S33, preprocess the multiple images according to the virtual path and the viewpoints of each image in the multiple images to obtain preprocessed images that conform to the viewpoints in the virtual path;

[0085] Step S34, generate multi-angle free-viewpoint data based on the preprocessed images.

[0086] Among them, the multiple synchronized images can be images collected by a collection device. Specifically, they can be synchronized frame images collected by multiple camera devices, or they can also be images at the same moment collected by multiple cameras. These multiple synchronized images are all images obtained by photographing the area to be viewed. The area to be viewed can be diverse and can be located in various scenarios. For example, it can be located in scenarios such as competitive sports scenarios and stage performance scenarios. The viewpoints for viewing the area to be viewed can be the viewpoints at which the user views on their terminal, which can be called the aforementioned virtual viewpoints.

[0087] In the embodiments of the present invention, by preprocessing multiple synchronized images, a set of preprocessed images that are consistent with the virtual path is obtained, and further multi-angle free-viewpoint data is obtained based on the set of processed images, which can improve the consistency between the multi-angle free-viewpoint data and the viewpoints for viewing, and further improve the image quality when reconstructing images based on the multi-angle free-viewpoint data.

[0088] In specific implementation, the determination method of the virtual path can be diverse. The virtual path can be preset according to specified viewing requirements or historical data. Or, the virtual path can also be obtained as a result of fitting based on the camera parameters corresponding to the multiple synchronized images. For example, it can be obtained as a result of fitting according to the camera parameters.

[0089] The camera parameters can include internal parameter data and external parameter data. The internal parameter data includes the attribute data of the image capture device, such as the attribute information of the optical center and focal length of the capture device. The external parameter data can include the capture position and capture angle of view.

[0090] In specific implementation, the basic path can be obtained by fitting according to the external parameters. As mentioned above, the external parameters of the camera can be represented by 6 parameters. Here, the three parameters representing the camera position (corresponding to the viewpoint position of the image) are denoted as (x, y, z), and the three parameters representing the camera attitude (corresponding to the viewpoint angle of view of the image) are denoted as These three parameters represent the rotation angles of the camera around the x, y, and z axes. The virtual path includes the movement trajectory (x, y, z) of the virtual camera in space, as well as the three rotation directions at each point on the trajectory. Thus, it includes the parameters of six degrees of freedom for each virtual position.

[0091] By determining the virtual path through fitting the camera parameters of multiple captured images, the consistency between the virtual path and multiple synchronized images can be made better, improving the utilization rate of multiple synchronized images, and further improving the quality of the reconstructed images when viewing the area to be viewed.

[0092] Based on the basic path, the virtual path can be further determined. The virtual path can include the basic path and the viewpoints supported by the results of focal length transformation based on the basic path.

[0093] By determining the virtual path according to the fitting results of the camera parameters of multiple captured images, the consistency between the virtual path and multiple synchronized images can be made better, improving the utilization rate of multiple synchronized images, and further improving the quality of the reconstructed images when viewing the area to be viewed.

[0094] As described above, the viewpoint includes the viewpoint position and the viewpoint angle. The virtual path is a set of viewpoints for viewing the area to be viewed, that is, it includes multiple virtual viewpoints. Further, the virtual path can include the viewpoint positions of multiple virtual viewpoints and the viewpoint angles corresponding to the viewpoint positions. Refer to Figure 3 and Figure 4 , Figure 3 In specific implementation, step S33 in

[0095] Step S41, map the viewpoint position of each image in the multiple images to the viewpoint position in the virtual path;

[0096] Step S42, according to the correspondence between the viewpoint position and the viewpoint angle in the virtual path, determine the target angle corresponding to the viewpoint position of each image in the multiple images;

[0097] Step S43, according to the viewpoint angle and the target angle of each image in the multiple images, obtain the image consistent with the corresponding target angle.

[0098] In specific implementation, the viewpoint position of each image in the multiple images can be mapped to the viewpoint position of the basic path, or the viewpoint position of each image in the multiple images can also be mapped to the viewpoint position after focal length transformation on the basic path.

[0099] In a specific implementation, the method of mapping the viewpoint position of each of the multiple images to the virtual path can be diverse. For example, when the viewpoint positions of the virtual viewpoint and the images are both expressed in the way of expressing positions in the 6-degree-of-freedom method as described above, the viewpoint position of the image can be mapped to the nearest viewpoint position on the virtual path according to their coordinate positions. Or, they can be unified to the same coordinate system first and then mapped. Or, mapping can also be performed in other ways.

[0100] It can be understood that mapping the viewpoint position of the image to the virtual path here aims to obtain the viewpoint angle corresponding to the viewpoint position mapped in the virtual path as the target angle. When processing each of the multiple images based on this target angle, the original viewpoint position of the original image in the multiple synchronized images can be retained for distortion operations, such as the interpolation operation described later. Or, a highlighting transformation operation can also be performed to map the viewpoints of each image in the multiple synchronized images so that they are consistent with the viewpoints mapped in the virtual path, such as performing a DIBR operation.

[0101] For example, with reference to Figure 5 , the arc-shaped dashed line in the figure schematically shows the basic path for the area to be viewed 51. The dashed triangle therein schematically shows some virtual viewpoints on the virtual path. The solid black triangle in the figure schematically shows the viewpoint 52 indicated by the camera parameters of one of the multiple synchronized images, and its viewpoint position is expressed as (x 1 , y 1 , z 1 ), and the viewpoint angle is expressed as By mapping to the virtual path, the viewpoint position (x 1 , y 1 , z 1 ) of this image is mapped to the viewpoint position (x 2 , y 2 , z 2 ) of the virtual viewpoint 53 schematically shown by the dashed triangle that partially coincides with it in the figure, and the viewpoint angle of this virtual viewpoint 53 is expressed as Then, the viewpoint angle can be used as the target angle of the viewpoint 52 indicated by the camera parameters of the image.

[0102] Furthermore, when preprocessing the image based on this target angle , the original viewpoint position (x 1 , y 1 , z 1 ) of the original image can be retained, and a distortion operation, such as the interpolation operation described later, can be performed on the image. Or, when preprocessing the image based on this target angle , the original viewpoint position (x1 , y 1 , z 1 ) is mapped to (x 2 , y 2 , z 2 ), and is processed by means of image transformation, such as DIBR.

[0103] It can be understood that although Figure 5 shows a basic path, the complete virtual path is not shown. However, it can be understood that the virtual path includes the viewpoints on this basic path, and includes viewpoints that are closer to or farther from the area to be viewed at the same viewing angle through the magnification or reduction of the image. For example, as Figure 5 shows the viewpoint 54 obtained by shortening the focal length of the viewpoint 55. By changing the focal length, viewing effects closer to or farther from the stage can be supported.

[0104] In addition, it can be understood that Figure 5 is only for illustration and does not limit the specific implementation methods such as the virtual path and the mapping method.

[0105] In specific implementation, with reference to Figure 6 , an image consistent with the corresponding target viewing angle can be obtained through the following steps:

[0106] Step S61, determine the image to be adjusted in the multiple images whose viewing angle of the viewpoint of the image is different from the target viewing angle;

[0107] Step S62, perform interpolation on the image to be adjusted according to the corresponding target viewing angle.

[0108] In specific implementation, the determination method of the target viewing angle of each image to be adjusted in the multiple images is as described above, and will not be elaborated here. The algorithms for performing interpolation on each image to be adjusted according to the corresponding target viewing angle can be diverse, and the algorithms for performing interpolation on each image to be adjusted in the multiple images can be the same or different, and are not limited here.

[0109] In specific implementation, interpolation can be performed on the texture map and the depth map of the image to be adjusted respectively. The interpolation operation here is used to warp the image so that the adjusted image is consistent with the target viewing angle.

[0110] Furthermore, when performing interpolation on the texture map of the image to be adjusted, an interpolation algorithm with a filtering template can be selected to ensure the smoothness of the corrected texture map. When performing interpolation on the depth map of the image to be adjusted, the nearest neighbor interpolation algorithm can be selected to ensure that the depth value will not change during the correction process.

[0111] Adjusting the image in a distorted manner can avoid image quality loss to a greater extent, thus ensuring the image quality of the generated multi-angle free-viewpoint data.

[0112] Combined with reference Figure 4 and Figure 7 , in a specific implementation, in step S43, after obtaining an image consistent with the corresponding target view angle according to the view point angle and the target view angle of each image in the multiple images, the following steps may further be included:

[0113] Step S71, determining a cropping range according to the intersection of the valid pixel ranges in each image consistent with the corresponding target view angle, where the valid pixel range is a region without holes;

[0114] Step S72, cropping the image consistent with the corresponding target view angle according to the cropping range.

[0115] Combined with reference Figure 8 and Figure 9 , taking a stage scene as an example, Figure 8 shows a texture map before interpolation of an image to be adjusted in a stage scene, Figure 8 shows the corresponding texture map after interpolation. It can be seen that after the interpolation process corresponding to the view point angle transformation, holes will be generated at the image edge, that is, Figure 9 the black area at the image edge in

[0116] It can be understood that Figure 8 and Figure 9 are only for illustration and are not limitations on the shape transformation before and after the actual interpolation. Moreover, the scenes applicable to the embodiments of the present application are diverse and are not limited to the stage scene.

[0117] In addition, Figure 8 and Figure 9 for clearer illustration, a larger hole area is shown. In the actual application scenario, the range of the hole area may be smaller. In the actual application scenario, the range of the hole area can be diverse. For example, see Figures 10 to 14 , where the black area indicates a part of the hole area after the interpolation process. It can be understood that Figures 8 to 14 is only for illustration and is not a limitation on the specific implementation manner or presentation form in the embodiments of the present application.

[0118] Since the hole area does not contain valid pixels. By cropping the part that does not contain valid pixels, resources for subsequent processing, storage, and transmission can be saved.

[0119] In a specific implementation, the cropping range can be determined according to the intersection of the regions without holes in each image after interpolation. For example Figures 10 to 14 the intersection of the white areas inFigure 15 as shown by the white area therein.

[0120] In a specific implementation, around the center of the image, the cropping area can be determined in combination with the above intersections. For example, with reference to Figure 16 , the range indicated by the dashed box therein can be used as the cropping area. In this way, the center of the image can be kept consistent with that before cropping, ensuring the visual effect.

[0121] It can be understood that Figures 10 to 16 this is only for illustration and does not limit the specific implementations such as the number of synchronized images, the shape of the hollow area, and the specific cropping method.

[0122] In a specific implementation, the multiple synchronized images can be the images corresponding to the same frame moment in the video data collected by multiple acquisition devices; generating multi-angle free-viewpoint data based on the preprocessed image set can include: storing the depth map and texture map corresponding to the processed images in the order of frame moments. The generated multi-angle free-viewpoint data can be multi-angle free-viewpoint video data.

[0123] The specific implementation manner of generating multi-angle free-viewpoint data is not limited herein. Specific implementation manners such as adjusting the image resolution in the preprocessed image set, downsampling, stitching the images, and encapsulating the stitched data, as well as other implementation manners that can be implemented by those skilled in the art, can be combined with other steps in this application for use.

[0124] In a specific implementation, the multi-angle free-viewpoint data is usually compressed and then transmitted to the user. By determining the virtual path before compressing the multi-angle free-viewpoint data and preprocessing the multiple synchronized images based on the virtual path, the texture and depth images of each view of the multi-angle free-viewpoint data finally entering the compression link can have a high consistency with the virtual path, thereby significantly reducing the hollow effect at the image edge when the user switches views, and also reducing the camera debugging requirements of the on-site setup to a practical level, so that in a low-cost scenario without camera position adjustment devices such as electric pan-tilts, a high free-viewpoint experience can also be achieved.

[0125] In a specific implementation, before generating the multi-angle free-viewpoint data based on the preprocessed image set, it can further include: determining the background part in the multiple synchronized images; replacing the background part in the multiple synchronized images.

[0126] In a specific implementation, the determination of the background part can be diverse, and there can be diverse determination methods according to different scenarios. For example, it can be determined that the part other than the people in the picture is the background part. Continuing with the previous example for illustration, with reference to Figure 2a andFigure 2b , the part other than the person holding the ball can be used as the background part. With reference to Figure 8 , the part other than the two persons shown therein can be used as the background part.

[0127] In specific implementation, the way to replace the background part and the screen effect after replacement can be diverse and are not limited herein. For example, the background in the picture can be replaced with a scene in a comic, or after processing such as recoloring the original background in the picture, the original background in the picture can be replaced.

[0128] The embodiment of the present application further provides an interaction method based on multi-angle free-viewpoint data, which may include: receiving an operation of a user; performing image display based on the multi-angle free-viewpoint data according to the operation of the user, where the multi-angle free-viewpoint data is generated based on an image set having consistency with a virtual path obtained by processing multiple synchronized images, the virtual path is a set of viewpoints for viewing a to-be-viewed area, and the viewpoints of the multiple synchronized images are different.

[0129] In specific implementation, the way to receive the operation of the user can be diverse and are not limited herein. For example, an operation on the touch screen of the terminal device by the user can be received, or an operation on a shortcut key by the user can be received, or the operation of the user can be received through other sensors installed in the terminal device, such as a gravity sensor and a sound sensor, or alternatively, the action instruction of the user can be received through a wearable device.

[0130] In specific implementation, the way to perform image display based on the multi-angle free-viewpoint data according to the operation of the user can be diverse. For example, a virtual viewpoint can be determined according to the operation of the user, and display based on the virtual viewpoint can be performed.

[0131] In specific implementation, the information indicated by the operation of the user can be diverse, and may include the aforementioned virtual viewpoint, or may also include an indication of the screen display range, or may further include the selection of a target object in the image displayed based on the multi-angle free-viewpoint data by the user.

[0132] Correspondingly, performing image display based on the multi-angle free-viewpoint data according to the operation of the user can correspond to the information indicated by the operation. For example, when the information indicated by the operation of the user includes the selection of a target object, performing image display on the image may include: obtaining a virtual information image generated based on the augmented reality special effect input data of the target object; performing synthesis processing on the virtual information image and the corresponding image and displaying the result.

[0133] For example, with reference to Figure 18 The target object can be Figure 18For the person in the picture, the virtual information image can be the Crown M3. Synthesizing and displaying the virtual information image with the corresponding image can be to display the crown above the person. The virtual information image can also be a virtual generated footprint M2, a basic information board M1, or other images matching the scene.

[0134] It can be understood that the target object can be diverse in different scenarios, and the virtual image information can also be diverse information matching the scene, which is not limited here.

[0135] In a specific implementation, the terminal device can receive the operation of the user, and the image can be displayed on the display. The screen of the display is larger than the screen of the terminal device. The terminal device can also display the corresponding image to facilitate the user's operation, or the image can also not be displayed. In this way, a better visual experience can be brought to the viewing user.

[0136] Furthermore, the image displayed by the terminal device can be a low-definition image, so as to improve the user's operation experience and visual experience when viewing.

[0137] It can be understood that the foregoing image can be a frame image in a multi-angle free-view video, or can also be called a video frame. Among them, the multi-angle free-view video is video data generated according to multi-angle free-view data and supports multi-viewpoint viewing.

[0138] In a specific implementation manner of the present application, the data generated by the foregoing data generation method and the foregoing data interaction method can be used for various live broadcasts. For example, with reference to 1, it can be used for live broadcasts of sports events. The sports event can be filmed by the acquisition system 11, and the display device 13 for display can watch the sports event in real time. Further, during the viewing process, the viewing user can adjust the viewpoint, display the virtual information image on the viewing interface, or can also switch the background, etc.

[0139] It can be understood that the content of the live broadcast can be diverse, which is not limited here. For example, it can also be a live broadcast of an educational classroom scene. The viewing student users can adjust the viewpoint, display the virtual information image on the viewing interface, or can also switch the background, etc. In this scenario, the displayed virtual information image can be related to the classroom content.

[0140] Those skilled in the art can understand that descriptions such as "in a specific implementation", "in an embodiment", and "for example" in this application mean that the specific features, structures, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. Moreover, the specific features, structures, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine different embodiments or examples described in this application and the features of different embodiments or examples.

[0141] In addition, any process or method description in the flowchart in the foregoing embodiments or described in other ways herein can be understood to represent a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process. And the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed.

[0142] The embodiments of this application also provide a multi-angle free-viewpoint data generation device. With reference to Figure 17 , it may specifically include the following units:

[0143] A synchronous image acquisition unit 151, adapted to acquire a plurality of synchronous images with different viewpoints.

[0144] A virtual path determination unit 152, adapted to determine a virtual path, where the virtual path is a set of viewpoints for viewing the area to be viewed.

[0145] A preprocessing unit 153, adapted to preprocess the plurality of images according to the virtual path and the viewpoints of each image in the plurality of images to obtain a set of preprocessed images that are consistent with the virtual path.

[0146] A data generation unit 154, adapted to generate multi-angle free-viewpoint data based on the set of preprocessed images.

[0147] In a specific implementation, the viewpoint may include a viewpoint position and a viewpoint angle; the preprocessing unit 153 may further include:

[0148] A mapping unit, adapted to map the viewpoint position of each image in the plurality of images to the viewpoint position in the virtual path.

[0149] A target angle determination unit, adapted to determine the target angle corresponding to the viewpoint position of each image in the plurality of images according to the correspondence between the viewpoint position and the viewpoint angle in the virtual path.

[0150] A viewing angle consistency unit, adapted to obtain an image consistent with a corresponding target viewing angle according to the viewing angles of viewpoints of each of the multiple images and the target viewing angle.

[0151] In a specific implementation, the viewing angle consistency unit (not shown in the figure) may further include:

[0152] An image to be adjusted determination unit, adapted to determine an image to be adjusted among the multiple images, whose viewing angle of the viewpoint is different from the target viewing angle;

[0153] An interpolation unit, adapted to perform interpolation on the image to be adjusted according to the corresponding target viewing angle.

[0154] Furthermore, in a specific implementation, the interpolation unit (not shown in the figure) may further include:

[0155] A depth map and texture map determination unit, adapted to determine the depth map and the texture map of the image to be adjusted;

[0156] A texture map interpolation unit, adapted to perform interpolation on the texture map of the image to be adjusted;

[0157] A depth map interpolation unit, adapted to perform interpolation on the depth map of the image to be adjusted.

[0158] In a specific implementation, the texture map interpolation unit is adapted to perform interpolation on the texture map by using an interpolation algorithm with a filtering template.

[0159] In a specific implementation, the depth map interpolation unit is adapted to perform interpolation on the depth map by using a nearest neighbor interpolation algorithm.

[0160] In a specific implementation, the preprocessing unit 153 may further include:

[0161] A cropping range determination unit, adapted to determine a cropping range according to the intersection of the valid pixel ranges in each of the images consistent with the corresponding target viewing angle, where the valid pixel range is a region without holes;

[0162] A cropping unit, adapted to crop the image consistent with the corresponding target viewing angle according to the cropping range.

[0163] In a specific implementation, the multiple synchronized images may be image data corresponding to the same frame moment in video data. The image data generation unit 154 may store the depth map and the texture map corresponding to the processed image in the order of the frame moments.

[0164] The multi-angle free-viewpoint data generation device in the embodiments of the present application corresponds to the multi-angle free-viewpoint data generation method. The principles, noun explanations, specific implementation manners, and beneficial effects involved can refer to the multi-angle free-viewpoint data generation method in the embodiments of the present application, and will not be elaborated herein.

[0165] Each unit described in the multi-angle free-viewpoint data generation device in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0166] Moreover, the various functional modules can be integrated in a processing component, or each module can exist physically alone, or two or more functional modules can be integrated in one component. The above-mentioned integrated component can be implemented in the form of hardware or in the form of a software functional module. When the above-mentioned integrated component is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a magnetic disk, an optical disk, etc.

[0167] The embodiments of the present application also provide a computer device suitable for running the foregoing multi-angle free-viewpoint generation method.

[0168] The computer device includes but is not limited to: servers, desktop computers, smart phones, laptop computers, tablet computers, smart bracelets, smart watches, other smart devices, or a distributed processing system formed by communicating and connecting multiple devices of any one or more of them.

[0169] In specific implementation, with reference to Figure 1 , the computer device can be located in Figure 1 Server 12 in

[0170] The embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, and when the computer program runs, it executes the foregoing multi-angle free-viewpoint data generation method.

[0171] That is, the multi-angle free-viewpoint data generation method in the embodiments of the present application can be implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-transitory machine-readable medium and downloaded through a network and will be stored in a local recording medium, so that the method described herein can be stored on such a recording medium for software processing using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (such as RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the multi-angle free-viewpoint data generation method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the multi-angle free-viewpoint data generation method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the multi-angle free-viewpoint data generation method shown herein.

[0172] Although the embodiments of the present application are disclosed as above, the present application is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, the protection scope of the present application should be subject to the scope defined by the claims.

Claims

1. A method for generating multi - angle free - view data, comprising: acquiring a plurality of synchronized images with different viewpoints; determining a virtual path, where the virtual path is a set of viewpoints for viewing a to - be - viewed area; pre - processing the plurality of images according to the virtual path and the viewpoint of each image in the plurality of images to obtain a set of pre - processed images that is consistent with the virtual path. Among them, the virtual path includes the viewpoint positions of a plurality of virtual viewpoints and the viewpoint perspectives corresponding to the viewpoint positions. By mapping the viewpoint position of each image in the plurality of images to the viewpoint position in the virtual path; according to the corresponding relationship between the viewpoint position and the viewpoint perspective in the virtual path, determining the target perspective corresponding to the viewpoint position of each image in the plurality of images; according to the viewpoint perspective and the target perspective of each image in the plurality of images, obtaining an image that is consistent with the corresponding target perspective; generating multi - angle free - view data based on the set of pre - processed images.

2. The multi - angle free - view data generating method according to claim 1, wherein, the virtual path includes a basic path and viewpoints obtained by performing focal length transformation on the viewpoints on the basic path, and the basic path is fitted based on the camera parameters of each image in the plurality of synchronized images.

3. The multi - angle free - view data generating method according to claim 1, wherein, the obtaining an image that is consistent with the corresponding target perspective according to the viewpoint perspective and the target perspective of each image in the plurality of images includes: determining the to - be - adjusted images in the plurality of images whose viewpoint perspectives are different from the target perspective; interpolating the to - be - adjusted images according to the corresponding target perspective.

4. The multi - angle free - view data generating method according to claim 3, wherein, the interpolating the to - be - adjusted images according to the corresponding target perspective includes: determining the depth map and texture map of the to - be - adjusted images; interpolating the texture map of the to - be - adjusted images; interpolating the depth map of the to - be - adjusted images.

5. The multi - angle free - view data generating method according to claim 4, wherein, the interpolating the texture map of the to - be - adjusted images includes: interpolating the texture map using an interpolation algorithm with a filtering template.

6. The multi - angle free - view data generating method according to claim 4, wherein, the interpolating the depth map of the to - be - adjusted images includes: interpolating the depth map using the nearest - neighbor interpolation algorithm.

7. The multi - angle free - view data generating method according to claim 1, wherein, the pre - processing further includes: determining a cropping range according to the intersection of the valid pixel ranges in each image that is consistent with the corresponding target perspective, and the valid pixel range is an area without holes; cropping the image that is consistent with the corresponding target perspective according to the cropping range.

8. The multi - angle free - view data generating method according to claim 1, wherein, Obtaining an image consistent with the corresponding target perspective according to the viewing perspectives and target perspectives of each of the multiple images includes: Determining an image to be adjusted among the multiple images, where the viewing perspective of the image is different from the target perspective; Performing image reconstruction on the image to be adjusted according to the viewing point position in the mapped virtual path and the corresponding target perspective.

9. The multi-angle free perspective data generation method according to claim 1, wherein, the synchronized multiple images are images corresponding to the same frame moment in video data; generating multi-angle free perspective data based on the preprocessed image set includes: storing the depth map and texture map corresponding to the processed image in the order of frame moments.

10. The multi-angle free perspective data generation method according to claim 1, wherein, before generating multi-angle free perspective data based on the preprocessed image set, it further includes: determining the background part in the synchronized multiple images; replacing the background part in the synchronized multiple images.

11. A multi-angle free perspective data generation device, wherein, it includes: a synchronized image acquisition unit, adapted to acquire synchronized multiple images with different viewing points; a virtual path determination unit, adapted to determine a virtual path, where the virtual path is a set of viewing points for viewing the area to be viewed; a preprocessing unit, adapted to preprocess the multiple images according to the virtual path and the viewing point of each of the multiple images to obtain a preprocessed image set consistent with the virtual path, where the virtual path includes the viewing point positions of multiple virtual viewing points and the viewing perspectives corresponding to the viewing point positions, by mapping the viewing point position of each of the multiple images to the viewing point position in the virtual path; according to the correspondence between the viewing point position and the viewing perspective in the virtual path, determining the target perspective corresponding to the viewing point position of each of the multiple images; obtaining an image consistent with the corresponding target perspective according to the viewing perspective and target perspective of each of the multiple images; a data generation unit, adapted to generate multi-angle free perspective data based on the preprocessed image set.

12. An interaction method based on multi-angle free perspective data, wherein, it includes: receiving the operation of the user; Based on the operations of the user, image display is performed based on multi-angle free-viewpoint data. The multi-angle free-viewpoint data is generated based on an image set that is consistent with a virtual path and obtained after preprocessing multiple synchronized images. The virtual path is a set of viewpoints for viewing the area to be viewed. The viewpoints of the multiple synchronized images are different. The virtual path includes the viewpoint positions of multiple virtual viewpoints and the viewpoint perspectives corresponding to the viewpoint positions. By mapping the viewpoint position of each image in the multiple images to the viewpoint position in the virtual path; according to the correspondence between the viewpoint position and the viewpoint perspective in the virtual path, determine the target perspective corresponding to the viewpoint position of each image in the multiple images; according to the viewpoint perspective and the target perspective of each image in the multiple images, obtain an image that is consistent with the corresponding target perspective.

13. The interaction method according to claim 12, characterized in that the operation of the user is received by a terminal device, and the image display is performed on a display. The screen of the display is larger than the screen of the terminal device.

14. The interaction method according to claim 12, characterized in that the receiving of the operation of the user includes: receiving the selection of a target object in the image displayed based on the multi-angle free-viewpoint data by the user; the performing of the image display based on the multi-angle free-viewpoint data according to the operation of the user includes: obtaining a virtual information image generated based on the augmented reality special effect input data of the target object; performing a synthesis process on the virtual information image and the corresponding image and displaying the result.

15. A computer device, comprising a memory and a processor, and a computer program is stored on the memory and can run on the processor, characterized in that when the processor runs the computer program, it executes the multi-angle free-viewpoint data generation method according to any one of claims 1 to 9, or the multi-angle free-viewpoint data interaction method according to any one of claims 12 to 14.

16. A computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program runs, it executes the multi-angle free-viewpoint data generation method according to any one of claims 1 to 9, or the multi-angle free-viewpoint data interaction method according to any one of claims 12 to 14.

Images