Image processing method and device, electronic equipment, chip and storage medium
By acquiring images of different viewpoints in the same scene, determining depth information and performing interpolation operations and mask fusion, the problem of being unable to generate any virtual viewpoint images in the prior art is solved, improving the quality and flexibility of viewpoint generation, and improving the stereoscopic viewing effect.
Patent Information
- Application Number
- CN202411350865.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2025-07-25
AI Technical Summary
The prior art cannot generate any virtual viewpoint images, resulting in poor visual comfort.
By obtaining images of different viewpoints in the same scene, determining depth information, generating virtual viewpoint images, and performing interpolation operations and mask diagram fusion, the generation of any virtual viewpoint is realized.
It improves the quality and flexibility of viewpoint generation, improves the three-dimensional viewing effect, and provides users with an immersive experience.
Smart Images
Figure CN120378591A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of virtual view generation, and in particular, to an image processing method, apparatus, electronic device, chip, and storage medium. Background Art
[0002] In traditional near-eye display technologies or autostereoscopic display technologies, since the reconstructed virtual scene is generally at a fixed depth, and this depth often differs from the position of the physical display, it will cause the problem of inconsistent convergence depth and accommodation depth of the human eye, and thus conflicts will occur. Therefore, based on the multi-viewpoint light field display technology, by projecting multi-angle information onto a single eye, the convergence accommodation conflict can be effectively alleviated and the visual comfort can be improved. Summary of the Invention
[0003] The present disclosure provides an image processing method, apparatus, electronic device, chip, storage medium, and computer program product to at least solve the problem in the prior art that arbitrary virtual view images cannot be generated. The technical solution of the present disclosure is as follows:
[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including: obtaining a first image at a first view point and a second image at a second view point; determining depth information according to the first image and the second image, and determining a virtual view point based on the depth information; mapping the first image and the second image based on the depth information and the position of the virtual view point to generate a first virtual view point image; performing an interpolation operation on the first image and the second image according to the position of the virtual view point to obtain a second virtual view point image; performing object detection on the first image and the second image to obtain the target position of the same moving object at the two view points under the virtual view point, and determining a mask image of the moving object based on the target position; and fusing the first virtual view point image, the second virtual view point image, and the mask image to obtain a target virtual view point image of the virtual view point.
[0005] According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, including: an acquisition module configured to acquire a first image at a first viewpoint and a second image at a second viewpoint; a determination module configured to determine depth information based on the first image and the second image, and determine a virtual viewpoint based on the depth information; a first generation module configured to map the first image and the second image based on the depth information and the position of the virtual viewpoint to generate a first virtual viewpoint image; a second generation module configured to perform an interpolation operation on the first image and the second image according to the position of the virtual viewpoint to obtain a second virtual viewpoint image; a third generation module configured to perform object detection on the first image and the second image to obtain the target positions of the same moving object at the two viewpoints at the virtual viewpoint, and determine a mask image of the moving object based on the target positions; and a fusion module configured to fuse the first virtual viewpoint image, the second virtual viewpoint image, and the mask image to obtain a target virtual viewpoint image of the virtual viewpoint.
[0006] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps of the method according to the first aspect of the embodiments of the present disclosure.
[0007] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the method according to the first aspect of the embodiments of the present disclosure are implemented.
[0008] According to a fifth aspect of the embodiments of the present disclosure, there is provided a chip, including one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal and send the signal to the processor, and the signal includes computer instructions stored in a memory, and when the processor executes the computer instructions, the chip executes the steps of the method according to the first aspect of the embodiments of the present disclosure.
[0009] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects: By acquiring a first image and a second image of different viewpoints in the same scene, and determining depth information from the first image and the second image, a first virtual viewpoint image is generated according to the depth information and the position of a preset virtual viewpoint. By performing an interpolation operation on the first image and the second image, a second virtual viewpoint image can be obtained. Further, according to the same moving object in the first image and the second image, a mask image of the moving object is generated, and according to the first virtual viewpoint image, the second virtual viewpoint image and the mask image, a target virtual viewpoint image of the virtual viewpoint is fused, so that images of any virtual viewpoint can be generated, improving the quality and flexibility of viewpoint generation, further enhancing the stereoscopic viewing effect of different viewpoints, and providing a more immersive experience for users.
[0010] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.
[0012] Figure 1 is a flowchart of an image processing method shown according to an exemplary embodiment;
[0013] Figure 2 is a flowchart of an image processing method shown according to another exemplary embodiment;
[0014] Figure 3 is a flowchart of an image processing method shown according to another exemplary embodiment;
[0015] Figure 4 is a flowchart of an image processing method shown according to another exemplary embodiment;
[0016] Figure 5 is a flowchart of a target virtual viewpoint image shown according to an exemplary embodiment;
[0017] Figure 6 is a block diagram of an image processing apparatus shown according to an exemplary embodiment;
[0018] Figure 7 is a block diagram of an electronic device shown according to an exemplary embodiment;
[0019] Figure 8 is a block diagram of a chip shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0020] To enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0022] In the technical solutions of the present disclosure, the acquisition, storage, use, processing, etc. of data all comply with the provisions of relevant laws and regulations.
[0023] The image processing method and apparatus of the embodiments of the present disclosure will be described below with reference to the accompanying drawings.
[0024] Figure 1 is a flowchart of an image processing method shown according to an exemplary embodiment. As Figure 1 shown, the image processing method of the embodiments of the present disclosure includes the following steps:
[0025] S101, acquire a first image at a first viewing point and a second image at a second viewing point.
[0026] It should be noted that the execution subject of the image processing method of the embodiments of the present disclosure is an electronic device, which can also be a chip. The electronic device can be a terminal device. Optionally, the terminal device includes but is not limited to mobile phones, computers, intelligent voice interaction devices, etc. The image processing method of the embodiments of the present disclosure can be executed by the image processing apparatus of the embodiments of the present disclosure. The image processing apparatus of the embodiments of the present disclosure can be configured in any electronic device to execute the image processing method of the embodiments of the present disclosure.
[0027] In some implementations, the first image and the second image at different viewing points in the same scene can be acquired based on an image acquisition device. Optionally, an image acquisition device can be set at the first viewing point at the same position, and an image acquisition device can be set at the second viewing point to acquire the first image and the second image in the same scene. Optionally, the graphic acquisition device of the electronic device itself, such as a camera, can also be used to acquire the first image and the second image at different viewing points at the same position.
[0028] S102, determine depth information based on the first image and the second image, and determine a virtual viewing point based on the depth information.
[0029] In some implementations, binocular depth estimation can be performed on the first image and the second image to obtain depth information. Optionally, a neural network can be used for binocular depth estimation, and by calculating the pixel position differences in the first image and the second image, the depth information of the moving objects in the scene can be estimated.
[0030] Optionally, the depth information can be converted into a depth image, and the first image, the second image, and the depth image can be converted into point cloud data, where the point cloud data contains the position information of each pixel point. Furthermore, by obtaining the position of the virtual viewpoint required by the user, a viewpoint transformation is performed on the point cloud data, that is, the point cloud data is transformed from the current perspective to the virtual perspective to obtain the virtual viewpoint.
[0031] S103, map the first image and the second image based on the depth information and the position of the virtual viewpoint to generate a first virtual viewpoint image.
[0032] In some implementations, virtual viewpoint generation can be performed based on the depth information to obtain a first virtual viewpoint image. Optionally, by mapping the first image and the second image into a three-dimensional space based on the depth information and obtaining the position of the set virtual viewpoint, and then mapping the three-dimensional data of this position into a two-dimensional plane, a first virtual viewpoint image is obtained.
[0033] Optionally, the Depth-Image-Based Rendering (DIBR) algorithm can be used to generate the first virtual viewpoint image. That is, the depth information, the position of the virtual viewpoint, the first image, and the second image are input into the DIBR algorithm, and the algorithm outputs the first virtual viewpoint image.
[0034] S104, perform an interpolation operation on the first image and the second image according to the position of the virtual viewpoint to obtain a second virtual viewpoint image.
[0035] In some implementations, the interpolation operation includes an in-frame interpolation operation and an out-of-frame interpolation operation. The interpolation operation can be determined as an in-frame interpolation operation or an out-of-frame interpolation operation according to the position relationship between the virtual viewpoint and the first viewpoint and the first viewpoint, and then the interpolation operation is performed on the first image and the second image to obtain a second virtual viewpoint image.
[0036] Optionally, if the virtual viewpoint is within the first viewpoint and the first viewpoint, determine the interpolation operation as an in-frame interpolation operation; if the virtual viewpoint is outside the first viewpoint and the first viewpoint, determine the interpolation operation as an out-of-frame interpolation operation.
[0037] Optionally, frame interpolation operations can be performed on the first image and the second image based on the optical flow information of the first image and the second image. That is, by obtaining the intra-frame optical flow of the first image and the second image, frame interpolation operations are performed on the first image and the second image; by obtaining the extra-frame optical flow of the first image and the second image, frame interpolation operations are performed on the first image and the second image.
[0038] S105. Perform object detection on the first image and the second image to obtain the target positions of the same moving object at two viewpoints in the virtual viewpoint, and determine the mask image of the moving object based on the target positions.
[0039] Optionally, an object detection network can be used to detect the same moving object in the first image and the second image, obtain the motion trajectory of the moving object in the first image and the second image, determine the target position of the moving object in the virtual viewpoint based on the motion trajectory, and perform a texturing operation on the target position using the regional image of the area where the same moving object in the first image and the second image is located to obtain an image containing the moving object.
[0040] Furthermore, set the pixel values of the non-moving object regions in the image to zero to obtain a mask image containing only the moving object.
[0041] S106. Fuse the first virtual viewpoint image, the second virtual viewpoint image, and the mask image to obtain the target virtual viewpoint image of the virtual viewpoint.
[0042] In some implementations, the feature data in the first virtual viewpoint image, the second virtual viewpoint image, and the mask image can be extracted, and the feature data can be fused to obtain the fused multi-dimensional feature data, and the multi-dimensional feature data can be restored to obtain the target virtual viewpoint image of the virtual viewpoint.
[0043] The image processing method provided by the embodiments of the present disclosure obtains the first image and the second image of different viewpoints of the same scene, determines the depth information from the first image and the second image, and generates the first virtual viewpoint image according to the depth information and the position of the preset virtual viewpoint. By performing frame interpolation operations on the first image and the second image, the second virtual viewpoint image can be obtained. Further, according to the same moving object in the first image and the second image, the mask image of the moving object is generated, and according to the first virtual viewpoint image, the second virtual viewpoint image, and the mask image, the target virtual viewpoint image of the virtual viewpoint is fused, which can realize the generation of images of any virtual viewpoint, improve the quality and flexibility of viewpoint generation, further improve the stereoscopic viewing effect of different viewpoints, and provide a more immersive experience for users.
[0044] Figure 2is a flowchart of an image processing method shown according to an exemplary embodiment, as Figure 2 shown, the image processing method of the embodiments of the present disclosure includes the following steps:
[0045] S201, obtain a first image at a first viewing point and a second image at a second viewing point.
[0046] S202, determine depth information based on the first image and the second image, and determine a virtual viewing point based on the depth information.
[0047] For the relevant content of steps S201 - S202, reference can be made to the above embodiments, which will not be elaborated here.
[0048] S203, based on the depth information, map the pixel points in the first image and the second image to a three - dimensional space to obtain point cloud data.
[0049] In some implementations, by obtaining the coordinates of each pixel point in the first image and the second image, and obtaining the depth value of the pixel point from the depth information, and converting the pixel point coordinates and the depth value into three - dimensional coordinates, the pixel points in the first image and the second image are mapped to a three - dimensional space to obtain point cloud data.
[0050] S204, according to the position of the virtual viewing point, re - project the point cloud data onto the imaging plane of the virtual viewing point to obtain a first virtual viewing point image.
[0051] In some implementations, according to the position of the virtual viewing point, the point cloud data at that position can be determined, and the point cloud data is re - projected onto the imaging plane of the virtual viewing point to obtain a first virtual viewing point image. Optionally, the depth information, the first image, the first image and the position of the virtual viewing point can be input into the DIBR algorithm, and the first virtual viewing point image is output by this algorithm.
[0052] In some implementations, in order to optimize the virtual viewing point image to obtain a virtual viewing point image with higher image quality, first, according to the position of the virtual viewing point, the point cloud data is re - projected onto the imaging plane of the virtual viewing point to obtain a first initial virtual viewing point image.
[0053] Further, perform hole recognition on the first initial virtual viewing point image, and fill the recognized holes with texture to obtain a first virtual viewing point image. Optionally, the texture information corresponding to the holes can be determined from the first image or the second image, and then the holes are filled with texture using the texture information.
[0054] Optionally, by determining the hole position of the hole and obtaining the texture information at the hole from the first image or the second image according to the air conditioner position, the hole can be filled with texture based on the texture information at the hole to obtain the first virtual view image. For example, an Inpainting network can be used to fill the hole with texture.
[0055] S205. Perform an interpolation operation on the first image and the second image according to the position of the virtual view point to obtain the second virtual view image.
[0056] S206. Perform object detection on the first image and the second image to obtain the target positions of the same moving object at the two view points under the virtual view point, and determine the mask image of the moving object based on the target positions.
[0057] S207. Fuse the first virtual view image, the second virtual view image, and the mask image to obtain the target virtual view image of the virtual view point.
[0058] For the relevant content of steps S205 - S207, reference can be made to the above embodiments and will not be elaborated here.
[0059] The image processing method provided by the embodiments of the present disclosure obtains the first image and the second image of different view points of the same scene, determines the depth information from the first image and the second image, maps the first image and the second image into point cloud data according to the depth information, and re-projects the point cloud data into the first virtual view image according to the position of the virtual view point. By performing an interpolation operation on the first image and the second image, the second virtual view image can be obtained. Further, according to the same moving object in the first image and the second image, the mask image of the moving object is generated, and according to the first virtual view image, the second virtual view image, and the mask image, the target virtual view image of the virtual view point is fused, which can realize the generation of images of any virtual view point, improve the quality and flexibility of view point generation, further improve the stereoscopic viewing effect of different view points, and provide a more immersive experience for users.
[0060] Figure 3 is a flowchart of an image processing method shown according to an exemplary embodiment. As Figure 3 shown, the image processing method of the embodiments of the present disclosure includes the following steps:
[0061] S301. Obtain the first image at the first view point and the second image at the second view point.
[0062] S302. Determine the depth information according to the first image and the second image, and determine the virtual view point based on the depth information.
[0063] S303. Map the first image and the second image based on the depth information and the position of the virtual view point to generate a first virtual view point image.
[0064] For the relevant content of steps S301 - S303, refer to the above embodiments and will not be elaborated here.
[0065] S304. Determine the positional relationship between the virtual view point and the first view point and the second view point according to the position of the virtual view point.
[0066] In some implementations, the positional relationship between the virtual view point and the first view point and the second view point includes a first positional relationship where the virtual view point is within the first view point and the second view point, and a second positional relationship where the virtual view point is outside the first view point and the second view point.
[0067] In some implementations, the first position of the first view point and the second position of the second view point can be obtained, and the position interval can be determined according to the first position and the second position. Then, by identifying whether the position of the virtual view point is within the position interval, the positional relationship between the virtual view point and the first view point and the second view point can be determined.
[0068] Optionally, if the virtual view point is within the first view point and the second view point, that is, if the position of the virtual view point is within the position interval, the positional relationship is determined as the first positional relationship; if the virtual view point is outside the first view point and the second view point, that is, if the position of the virtual view point is not within the position interval, the positional relationship is determined as the second positional relationship.
[0069] S305. Perform an interpolation operation on the first image and the second image according to the positional relationship to obtain a second virtual view point image.
[0070] In some implementations, the interpolation operation includes an in - frame interpolation operation and an out - of - frame interpolation operation. If the positional relationship is the first positional relationship, perform an in - frame interpolation operation on the first image and the second image to obtain a second virtual view point image; if the positional relationship is the second positional relationship, perform an out - of - frame interpolation operation on the first image and the second image to obtain a second virtual view point image.
[0071] In some implementations, the interpolation operation on the first image and the second image can be performed based on the optical flow information of the first image and the second image. That is, when performing the in - frame interpolation operation, multi - scale feature extraction is performed on the first image and the second image, and the inter - frame optical flow of the first image and the second image is obtained according to the multi - scale features. Then, according to the inter - frame optical flow, the pixel points in the first image and the second image can be moved to obtain a second virtual view point image.
[0072] Optionally, when performing extrapolated frame operation, multi-scale feature extraction is performed on the first image and the second image, and the off-frame optical flow of the first image and the second image is obtained according to the multi-scale features. Furthermore, the pixel points in the first image and the second image can be moved according to the off-frame optical flow to obtain the second virtual viewpoint image.
[0073] Optionally, a neural network for video frame interpolation can be used to perform frame interpolation operations on the first image and the second image. For example, an intra-frame neural network is used for intra-frame interpolation operation, and an extra-frame neural network is used for extra-frame interpolation operation.
[0074] In some implementations, in order to enhance the image quality of the virtual viewpoint image, the image quality of the virtual viewpoint image can be enhanced to obtain the second virtual viewpoint image. That is to say, first, frame interpolation operations are performed on the first image and the second image according to the positional relationship to obtain the second initial virtual viewpoint image, and then the image quality of the second initial virtual viewpoint image is further enhanced to obtain the second virtual viewpoint image. For example, a U-shaped network can be used to further enhance the image quality of the second initial virtual viewpoint image.
[0075] S306, perform object detection on the first image and the second image to obtain the target positions of the same moving object under the two viewpoints in the virtual viewpoint, and determine the mask map of the moving object based on the target positions.
[0076] S307, fuse the first virtual viewpoint image, the second virtual viewpoint image, and the mask map to obtain the target virtual viewpoint image of the virtual viewpoint.
[0077] For the relevant content of steps S306 - S307, reference can be made to the above embodiments, which will not be elaborated here.
[0078] The image processing method provided by the embodiments of the present disclosure obtains the first image and the second image of the same scene from different viewpoints, and determines the depth information from the first image and the second image, so as to generate the first virtual viewpoint image according to the depth information and the position of the preset virtual viewpoint. According to the position of the virtual viewpoint, frame interpolation operations are determined, and frame interpolation operations are performed on the first image and the second image to obtain the second virtual viewpoint image. Further, according to the same moving object in the first image and the second image, the mask map of the moving object is generated, and according to the first virtual viewpoint image, the second virtual viewpoint image, and the mask map, the target virtual viewpoint image of the virtual viewpoint is fused, which can realize the generation of images for any virtual viewpoint, improve the quality and flexibility of viewpoint generation, further improve the stereoscopic viewing effect of different viewpoints, and provide a more immersive experience for users. The method proposed by the present disclosure supports off-frame interpolation operation, which greatly improves the quality and flexibility of virtual viewpoint image generation.
[0079] Figure 4is a flowchart of an image processing method shown according to an exemplary embodiment. As Figure 4 shown, the image processing method of the embodiments of the present disclosure includes the following steps:
[0080] S401, obtain a first image at a first viewpoint and a second image at a second viewpoint.
[0081] S402, determine depth information based on the first image and the second image, and determine a virtual viewpoint based on the depth information.
[0082] S403, map the first image and the second image based on the depth information and the position of the virtual viewpoint to generate a first virtual viewpoint image.
[0083] S404, perform an interpolation operation on the first image and the second image according to the position of the virtual viewpoint to obtain a second virtual viewpoint image.
[0084] For the relevant content of steps S401 - S404, reference can be made to the above embodiments and will not be elaborated here.
[0085] S405, perform object detection on the first image and the second image, and obtain a first position of the moving object in the first image and a second position of the moving object in the second image.
[0086] In some implementations, by performing object detection on the first image and the second image, the same moving object in the first image and the second image, as well as the detection frame of the moving object, can be detected. And according to the detection frame of the moving object, the first position of the moving object in the first image and the second position of the moving object in the second image can be determined.
[0087] Optionally, an object detection algorithm based on deep learning can be used to perform object detection on the first image and the second image. Optionally, an object detection algorithm or object detection model based on computer vision can also be used to perform object detection on the first image and the second image. The embodiments of the present disclosure do not limit the object detection algorithm.
[0088] S406, obtain the motion trajectory of the moving object according to the first position and the second position.
[0089] In some implementations, assuming that the moving object performs uniform linear motion as the viewpoint changes, then based on the first position and the second position, the linear equation can be calculated, and this line can be used as the motion trajectory of the moving object. For example, let the first position of the moving object be (x1, y1) and the second position be (x2, y2), then according to the first position and the second position, the slope information of the line can be calculated, and then a line in the form of y = kx + b can be constructed as the motion trajectory of the moving object.
[0090] S407. Determine the target position of the moving object under the virtual viewpoint based on the motion trajectory.
[0091] In some implementations, the target position of the moving object under the virtual viewpoint can also be determined according to the positional relationship between the virtual viewpoint and the first viewpoint and the second viewpoint, and in combination with the motion trajectory of the moving object. For example, if the positional relationship is the first positional relationship, interpolation operations can be performed on the motion trajectory to determine that the target position is (x1.5, y1.5); if the positional relationship is the second positional relationship, prediction can be performed based on the motion trajectory to determine that the target position is (x3, y3).
[0092] S408. Determine the mask image of the moving object based on the target position.
[0093] In some implementations, a mask image containing only the moving object can be obtained by performing a pixel zeroing operation on the non-moving object area. The area image where the moving object is located can be extracted from the reference image, where the reference image is one of the first image and the second image. Furthermore, based on the target position and the area image, a mapping operation can be performed on the reference image, and the pixel points in the non-moving object area after the mapping of the reference image are zeroed to obtain the mask image.
[0094] S409. Fuse the first virtual viewpoint image, the second virtual viewpoint image, and the mask image to obtain the target virtual viewpoint image of the virtual viewpoint.
[0095] In some implementations, a multi-channel image is obtained by merging the first virtual viewpoint image, the second virtual viewpoint image, and the mask image. For example, the first virtual viewpoint image, the second virtual viewpoint image, and the mask image can be stitched together to obtain a multi-channel image.
[0096] Furthermore, the multi-channel image is input into a network structure with a U-shaped structure for feature extraction and image restoration to obtain the target virtual viewpoint image. That is to say, by extracting multi-scale features from the multi-channel image and performing image restoration based on the multi-scale features, the target virtual viewpoint image can be obtained.
[0097] The image processing method provided by the embodiments of the present disclosure obtains a first image and a second image of the same scene from different viewpoints, determines depth information from the first image and the second image, and generates a first virtual viewpoint image according to the depth information and the position of a preset virtual viewpoint. By performing an interpolation operation on the first image and the second image, a second virtual viewpoint image can be obtained. Further, by determining the same moving object in the first image and the second image, and based on the motion trajectory of the moving object, determining the target position of the moving object under the virtual viewpoint, generating a mask image of the moving object, and fusing the first virtual viewpoint image, the second virtual viewpoint image, and the mask image to obtain the target virtual viewpoint image of the virtual viewpoint, it is possible to generate images of any virtual viewpoint, improve the quality and flexibility of viewpoint generation, further improve the stereoscopic viewing effect of different viewpoints, and provide a more immersive experience for users.
[0098] As Figure 5 shown is the flowchart of the target virtual viewpoint image. By obtaining the first image of the first viewpoint and the second image of the second viewpoint in the same scene, and performing binocular depth estimation on the first image and the second image using a neural network, the depth information can be determined. Based on the depth information, the position of the preset virtual viewpoint, the first image, and the second image, the DIBR algorithm is used to generate a virtual viewpoint image to obtain a first initial virtual viewpoint image, and the Inpainting network is used to fill the holes in the first initial virtual viewpoint image to obtain the first virtual viewpoint image.
[0099] According to the position of the virtual viewpoint, an interpolation operation can be performed on the first image and the second image, and either an interpolation operation or an extrapolation operation can be selected to obtain a second initial virtual viewpoint image, and the UNet is used for further image quality enhancement to obtain the second virtual viewpoint image.
[0100] Perform object detection on the first image and the second image to determine the same moving object in the first image and the second image, as well as the first position of the moving object in the first image and the second position of the moving object in the second image. Furthermore, the motion trajectory of the moving object can be calculated, and the target position of the moving object under the virtual viewpoint can be determined according to the motion trajectory. Perform a texturing operation on the moving object according to the target position, and perform a zeroing operation on the pixel points in the non-moving object area to obtain a mask image.
[0101] Further, fuse the first virtual viewpoint image, the second virtual viewpoint image, and the mask image to obtain the target virtual viewpoint image.
[0102] Figure 6 is a block diagram of an image processing apparatus shown according to an exemplary embodiment. Refer to Figure 6, the image processing apparatus 600 according to an embodiment of the present disclosure includes: an acquisition module 601, a determination module 602, a first generation module 603, a second generation module 604, a third generation module 605, and a fusion module 606.
[0103] The acquisition module 601 is configured to acquire a first image at a first viewpoint and a second image at a second viewpoint;
[0104] The determination module 602 is configured to determine depth information based on the first image and the second image, and determine a virtual viewpoint based on the depth information;
[0105] The first generation module 603 is configured to map the first image and the second image based on the depth information and the position of the virtual viewpoint to generate a first virtual viewpoint image;
[0106] The second generation module 604 is configured to perform an interpolation operation on the first image and the second image according to the position of the virtual viewpoint to obtain a second virtual viewpoint image;
[0107] The third generation module 605 is configured to perform object detection on the first image and the second image to obtain the target positions of the same moving object at the two viewpoints in the virtual viewpoint, and determine a mask image of the moving object based on the target positions;
[0108] The fusion module 606 is configured to fuse the first virtual viewpoint image, the second virtual viewpoint image, and the mask image to obtain a target virtual viewpoint image of the virtual viewpoint.
[0109] In an embodiment of the present disclosure, the first generation module 603 is further configured to: map the pixel points in the first image and the second image to a three-dimensional space based on the depth information to obtain point cloud data; re-project the point cloud data onto the imaging plane of the virtual viewpoint according to the position of the virtual viewpoint to obtain the first virtual viewpoint image.
[0110] In an embodiment of the present disclosure, the first generation module 603 is further configured to: re-project the point cloud data onto the imaging plane of the virtual viewpoint according to the position of the virtual viewpoint to obtain a first initial virtual viewpoint image; perform hole recognition on the first initial virtual viewpoint image, and fill the recognized holes with textures to obtain the first virtual viewpoint image.
[0111] In an embodiment of the present disclosure, the first generation module 603 is further configured to: determine the position of the hole in the hole, and obtain the texture information at the hole from the first image or the second image according to the air conditioner position; based on the texture information at the hole, fill the texture of the hole to obtain the first virtual view image.
[0112] In an embodiment of the present disclosure, the second generation module 604 is further configured to: determine the positional relationship between the virtual view point and the first view point and the second view point according to the position of the virtual view point; perform an interpolation operation on the first image and the second image according to the positional relationship to obtain the second virtual view image.
[0113] In an embodiment of the present disclosure, the second generation module 604 is further configured to: obtain the first position of the first view point and the second position of the second view point; determine a position interval according to the first position and the second position; identify whether the position of the virtual view point is within the position interval; if the position of the virtual view point is within the position interval, determine the positional relationship as a first positional relationship; if the position of the virtual view point is not within the position interval, determine the positional relationship as a second positional relationship.
[0114] In an embodiment of the present disclosure, the second generation module 604 is further configured to: if the positional relationship is a first positional relationship, perform an interpolation operation on the first image and the second image to obtain the second virtual view image; or, if the positional relationship is a second positional relationship, perform an extrapolation operation on the first image and the second image to obtain the second virtual view image.
[0115] In an embodiment of the present disclosure, the second generation module 604 is further configured to: perform multi-scale feature extraction on the first image and the second image; obtain the inter-frame optical flow of the first image and the second image according to the multi-scale features; move the pixel points in the first image and the second image according to the inter-frame optical flow to obtain the second virtual view image.
[0116] In an embodiment of the present disclosure, the second generation module 604 is further configured to: perform multi-scale feature extraction on the first image and the second image; obtain the extra-frame optical flow of the first image and the second image according to the multi-scale features; move the pixel points in the first image and the second image according to the extra-frame optical flow to obtain the second virtual view image.
[0117] In an embodiment of the present disclosure, the second generation module 604 is further configured to: perform an interpolation operation on the first image and the second image according to the positional relationship to obtain a second initial virtual viewpoint image; perform image quality enhancement on the second initial virtual viewpoint image to obtain the second virtual viewpoint image.
[0118] In an embodiment of the present disclosure, the third generation module 605 is further configured to: perform object detection on the first image and the second image to obtain a first position of the moving object in the first image and a second position of the moving object in the second image; obtain a motion trajectory of the moving object according to the first position and the second position; determine a target position of the moving object under the virtual viewpoint based on the motion trajectory.
[0119] In an embodiment of the present disclosure, the third generation module 605 is further configured to: extract a picture of the area where the moving object is located from the reference image, where the reference image is one of the first image and the second image; perform a texturing operation on the reference image based on the target position and the area picture, and perform a zeroing operation on the pixel points in the non-moving object area after the texturing of the reference image to obtain the mask image.
[0120] In an embodiment of the present disclosure, the fusion module 606 is further configured to: merge the first virtual viewpoint image, the second virtual viewpoint image, and the mask image to obtain a multi-channel image; input the multi-channel image into a network structure of a U-shaped structure for feature extraction and image restoration to obtain the target virtual viewpoint image.
[0121] The image processing device provided by the embodiment of the present disclosure obtains the first image and the second image of the same scene from different viewpoints, and determines depth information from the first image and the second image, so as to generate a first virtual viewpoint image according to the depth information and the position of the preset virtual viewpoint. By performing an interpolation operation on the first image and the second image, a second virtual viewpoint image can be obtained. Further, according to the same moving object in the first image and the second image, a mask image of the moving object is generated, and according to the first virtual viewpoint image, the second virtual viewpoint image, and the mask image, the target virtual viewpoint image of the virtual viewpoint is fused, which can realize the generation of images of any virtual viewpoint, improve the quality and flexibility of viewpoint generation, further improve the stereoscopic viewing effect of different viewpoints, and provide a more immersive experience for users.
[0122] Figure 7 It is a block diagram of an electronic device shown according to an exemplary embodiment.
[0123] As Figure 7 shown, the above-mentioned electronic device 700 includes:
[0124] A memory 701 and a processor 702, a bus 703 connecting different components (including the memory 701 and the processor 702), the memory 701 stores a computer program, and when the processor 702 executes the program, the image processing method described in the embodiments of the present disclosure is implemented.
[0125] The bus 703 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any bus structure in a variety of bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
[0126] The electronic device 700 typically includes a variety of electronically readable media. These media can be any available media that can be accessed by the electronic device 700, including volatile and non-volatile media, removable and non-removable media.
[0127] The memory 701 may further include a computer system readable medium in the form of volatile memory, such as random access memory (RAM) 704 and / or cache memory 705. The electronic device 700 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 706 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, commonly referred to as a "hard disk drive"). Although Figure 7 not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 703 through one or more data media interfaces. The memory 701 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present disclosure.
[0128] A program / utility 708 having a set (at least one) of program modules 707 may be stored in, for example, the memory 701. Such program modules 707 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules 707 generally perform the functions and / or methods in the embodiments described in the present disclosure.
[0129] The electronic device 700 can also communicate with one or more external devices 709 (such as a keyboard, a pointing device, a display 791, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 792. Moreover, the electronic device 700 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 793. As Figure 7 shown, the network adapter 793 communicates with other modules of the electronic device 700 through the bus 703. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0130] The processor 702 executes various functional applications and data processing by running the programs stored in the memory 701.
[0131] It should be noted that for the implementation process and technical principle of the electronic device in this embodiment, refer to the foregoing explanation of the image processing method in the embodiments of the present disclosure, and details are not described herein again.
[0132] To implement the above embodiment, the present disclosure also proposes a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the image processing method provided by the present disclosure are implemented.
[0133] Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0134] Figure 8 is a block diagram of a chip shown according to an exemplary embodiment.
[0135] As Figure 8 shown, the above chip 800 includes one or more interface circuits 802 and one or more processors 801; the interface circuit 802 is used to receive signals and send signals to the processor 801, and the signals include computer instructions stored in the memory. When the processor 801 executes the computer instructions, the chip 800 executes the steps of the image processing method provided by the present disclosure.
[0136] The processor 801 and the interface circuit 802 can be interconnected through lines.
[0137] The chip 800 further includes a memory 803, and all or part of the memory 803 may be outside the chip 800.
[0138] The interface circuit 802 is connected to the memory 803. The interface circuit 802 can be used to receive signals from the memory 803 or other devices, and the interface circuit 802 can be used to send signals to the processor 801 or other devices. For example, the interface circuit 802 can read the instructions stored in the memory 803 and send the instructions to the processor 801.
[0139] The interface circuit 802 can obtain data, program instructions, and / or information, etc. from the internal storage area of the chip 800; it can also obtain data, program instructions, and / or information, etc. outside the chip 800.
[0140] It should be noted that terms such as interface circuit, interface, transceiver pin, transceiver, etc. can be replaced with each other. Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and examples are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.
[0141] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.
Claims
1. An image processing method, characterized in that, The method includes: Obtaining a first image at a first viewpoint and a second image at a second viewpoint; Determining depth information based on the first image and the second image, and determining a virtual viewpoint based on the depth information; Mapping the first image and the second image based on the depth information and the position of the virtual viewpoint to generate a first virtual viewpoint image; Performing an interpolation operation on the first image and the second image according to the position of the virtual viewpoint to obtain a second virtual viewpoint image; Performing object detection on the first image and the second image to obtain the target positions of the same moving object at the two viewpoints under the virtual viewpoint, and determining a mask image of the moving object based on the target positions; Fusing the first virtual viewpoint image, the second virtual viewpoint image, and the mask image to obtain a target virtual viewpoint image of the virtual viewpoint.
2. The method according to claim 1, wherein The mapping the first image and the second image based on the depth information and the position of the virtual viewpoint to generate a first virtual viewpoint image includes: Based on the depth information, mapping the pixel points in the first image and the second image to a three-dimensional space to obtain point cloud data; According to the position of the virtual viewpoint, re-projecting the point cloud data onto the imaging plane of the virtual viewpoint to obtain the first virtual viewpoint image.
3. The method according to claim 2, wherein The re-projecting the point cloud data onto the imaging plane of the virtual viewpoint according to the position of the virtual viewpoint to obtain the first virtual viewpoint image includes: According to the position of the virtual viewpoint, re-projecting the point cloud data onto the imaging plane of the virtual viewpoint to obtain a first initial virtual viewpoint image; Performing hole recognition on the first initial virtual viewpoint image, and filling the recognized holes with textures to obtain the first virtual viewpoint image.
4. The method according to claim 3, wherein The filling the recognized holes with textures to obtain the first virtual viewpoint image includes: Determining the hole positions of the holes, and obtaining texture information at the holes from the first image or the second image according to the hole positions; Based on the texture information at the holes, filling the holes with textures to obtain the first virtual viewpoint image.
5. The method according to claim 1, characterized in that, The performing an interpolation operation on the first image and the second image according to the position of the virtual viewpoint to obtain a second virtual viewpoint image includes: According to the position of the virtual viewpoint, determining the positional relationship between the virtual viewpoint and the first viewpoint and the second viewpoint; According to the positional relationship, performing an interpolation operation on the first image and the second image to obtain the second virtual viewpoint image.
6. The method according to claim 5, wherein The determining the positional relationship between the virtual viewpoint and the first viewpoint and the second viewpoint according to the position of the virtual viewpoint includes: Obtaining a first position of the first viewpoint and a second position of the second viewpoint; Determining a position interval according to the first position and the second position; Identifying whether the position of the virtual viewpoint is within the position interval; If the position of the virtual viewpoint is within the position interval, determining the positional relationship as a first positional relationship; If the position of the virtual viewpoint is not within the position interval, determine that the position relationship is the second position relationship.
7. The method according to claim 6, wherein The performing an interpolation operation on the first image and the second image according to the position relationship to obtain the second virtual viewpoint image includes: If the position relationship is the first position relationship, performing an interpolation operation on the first image and the second image to obtain the second virtual viewpoint image; or, If the position relationship is the second position relationship, performing an extrapolation operation on the first image and the second image to obtain the second virtual viewpoint image.
8. The method according to claim 7, wherein The performing an interpolation operation on the first image and the second image to obtain the second virtual viewpoint image includes: Performing multi-scale feature extraction on the first image and the second image; Obtaining the inter-frame optical flow of the first image and the second image according to the multi-scale features; Moving the pixel points in the first image and the second image according to the inter-frame optical flow to obtain the second virtual viewpoint image.
9. The method according to claim 7, characterized in that The performing an extrapolation operation on the first image and the second image to obtain the second virtual viewpoint image includes: Performing multi-scale feature extraction on the first image and the second image; Obtaining the extra-frame optical flow of the first image and the second image according to the multi-scale features; Moving the pixel points in the first image and the second image according to the extra-frame optical flow to obtain the second virtual viewpoint image.
10. The method according to any one of claims 5-9, characterized in that, The performing an interpolation operation on the first image and the second image according to the position relationship to obtain the second virtual viewpoint image includes: Performing an interpolation operation on the first image and the second image according to the position relationship to obtain a second initial virtual viewpoint image; Performing image quality enhancement on the second initial virtual viewpoint image to obtain the second virtual viewpoint image.
11. The method according to any one of claims 1-9, characterized in that, The performing object detection on the first image and the second image to obtain the target positions of the same moving object at the two viewpoints under the virtual viewpoint includes: Performing object detection on the first image and the second image to obtain the first position of the moving object in the first image and the second position of the moving object in the second image; Obtaining the motion trajectory of the moving object according to the first position and the second position; Determining the target position of the moving object under the virtual viewpoint based on the motion trajectory.
12. The method according to claim 1, wherein The determining the mask image of the moving object based on the target position includes: Extracting the picture of the area where the moving object is located from the reference image, where the reference image is one of the first image and the second image; Performing a texturing operation on the reference image based on the target position and the area picture, and performing a zeroing operation on the pixel points in the non-moving object area after the texturing of the reference image to obtain the mask image.
13. The method according to any one of claims 1-9, characterized in that, The fusing the first virtual viewpoint image, the second virtual viewpoint image and the mask image to obtain the target virtual viewpoint image of the virtual viewpoint includes: Merging the first virtual viewpoint image, the second virtual viewpoint image and the mask image to obtain a multi-channel image; Input the multi-channel image into a U-shaped network structure for feature extraction and image restoration to obtain the target virtual viewpoint image.
14. An image processing apparatus, characterized in that, The device includes: An acquisition module, configured to acquire a first image at a first viewpoint and a second image at a second viewpoint; A determination module, configured to determine depth information based on the first image and the second image, and determine a virtual viewpoint based on the depth information; A first generation module, configured to map the first image and the second image based on the depth information and the position of the virtual viewpoint to generate a first virtual viewpoint image; A second generation module, configured to perform an interpolation operation on the first image and the second image according to the position of the virtual viewpoint to obtain a second virtual viewpoint image; A third generation module, configured to perform object detection on the first image and the second image to obtain the target positions of the same moving object at the two viewpoints under the virtual viewpoint, and determine a mask image of the moving object based on the target positions; A fusion module, configured to fuse the first virtual viewpoint image, the second virtual viewpoint image, and the mask image to obtain the target virtual viewpoint image of the virtual viewpoint.
15. An electronic device, characterized in that, It includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to: Implement the steps of the method according to any one of claims 1-13.
16. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, the steps of the method according to any one of claims 1-13 are implemented.
17. A chip, characterized in that, It includes one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal and send the signal to the processor, and the signal includes computer instructions stored in the memory. When the processor executes the computer instructions, the chip executes the steps of the method according to any one of claims 1-13.