Method, device, computer-readable storage medium and processor for reconstructing an image

Abstract

The present invention discloses a method, device, computer-readable storage medium and processor for reconstructing an image. The method comprises: detecting an interactive operation on an operation interface, obtaining a target position of a virtual viewpoint after displacement on the operation interface; controlling the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; and using an image deformation model to process an image with the virtual viewpoint at the predetermined position to obtain a reconstructed image. The present invention solves the technical problem of poor timeliness when reconstructing an image from a virtual viewpoint.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method, device, computer-readable storage medium and processor for reconstructing an image. Background Art

[0002] Currently, users use gesture interaction to determine the virtual viewpoint position of the video image on the terminal. During the viewing process, due to the small computing resources of the terminal or the poor video quality, the user stays on the virtual viewpoint for a long time to watch, resulting in video jams, and the timeliness of reconstructing the image at the virtual viewpoint is poor.

[0003] With respect to the above-mentioned technical problem of poor timeliness when reconstructing images from a virtual viewpoint, no effective solution has been proposed so far. Summary of the invention

[0004] Embodiments of the present invention provide a method, an apparatus, a computer-readable storage medium, and a processor for reconstructing an image, so as to at least solve the technical problem of poor timeliness when reconstructing an image at a virtual viewpoint.

[0005] According to one aspect of an embodiment of the present invention, a method for reconstructing an image is provided. The method may include: detecting an interactive operation on an operation interface, obtaining a target position of a virtual viewpoint after displacement on the operation interface; controlling the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial degree of freedom position of the virtual viewpoint; and using an image deformation model to process an image at a predetermined position of the virtual viewpoint to obtain a reconstructed image.

[0006] According to another aspect of an embodiment of the present invention, another method for reconstructing an image is provided. The method may include: detecting an interactive operation on an operation interface that controls a virtual viewpoint to move from a first position to a second position; controlling the virtual viewpoint to transition from the second position to a third position, wherein the third position is a position of a real viewpoint where a camera is deployed; reading an image of the camera at the third position; and processing the image using a plane-based image deformation model to obtain a reconstructed image.

[0007] According to another aspect of an embodiment of the present invention, another method for reconstructing an image is also provided. The method may include: sensing an interactive operation of an operating object on an operating interface; in response to the interactive operation, displaying a displacement of a virtual viewpoint on the operating interface based on the interactive operation, wherein the displacement is from a first position to a second position; displaying a transition of the virtual viewpoint from the second position to a third position, wherein the third position is the position of a real viewpoint where a camera is deployed; and displaying a reconstructed image generated by processing a target image using a plane-based image deformation model, wherein the target image is an image captured by a camera located at the third position.

[0008] According to another aspect of an embodiment of the present invention, another method for reconstructing an image is provided. The method may include: sensing a moving operation of an operating object on an operating interface; if the moving operation is sensed to be terminated, obtaining a stop position of the operating object on the operating interface when the moving operation is terminated; controlling a virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial degree of freedom position of the virtual viewpoint; and using a plane-based image deformation model to process an image of the camera at the predetermined position to obtain a reconstructed image.

[0009] According to another aspect of an embodiment of the present invention, another method for reconstructing an image is provided. The method may include: sensing a moving operation of an operating object on an operating interface; if the moving operation is sensed to be terminated, obtaining a stop position of the operating object on the operating interface when the moving operation is terminated; controlling a virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is a position of a real viewpoint where a camera is deployed; reading an image of a camera at the predetermined position; and processing the image using a plane-based image deformation model to obtain a reconstructed image.

[0010] According to another aspect of an embodiment of the present invention, another method for reconstructing an image is provided. The method may include: during a live broadcast, detecting that a moving operation is received on the live broadcast screen; obtaining a target position of a virtual viewpoint after displacement on the live broadcast screen; controlling the virtual viewpoint to move from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; and using an image deformation model to process an image of a virtual viewpoint at a predetermined position to obtain a reconstructed image.

[0011] According to another aspect of an embodiment of the present invention, a device for reconstructing an image is also provided. The device may include: a first acquisition module, used to detect an interactive operation on an operation interface, and acquire a target position of a virtual viewpoint after displacement on the operation interface; a first control module, used to control the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; and a second processing module, used to process an image with a virtual viewpoint at a predetermined position using an image deformation model, and acquire a reconstructed image.

[0012] According to another aspect of an embodiment of the present invention, another device for reconstructing an image is provided. The device may include: a detection module for detecting an interactive operation on an operation interface for controlling a virtual viewpoint to move from a first position to a second position; a second control module for controlling the virtual viewpoint to transition from the second position to a third position, wherein the third position is the position of a real viewpoint where a camera is deployed; a first reading module for reading an image of a camera located at the third position; and a second processing module for processing an image using a plane-based image deformation model to obtain a reconstructed image.

[0013] According to another aspect of an embodiment of the present invention, another device for reconstructing an image is also provided. The device may include: a first sensing module, used to sense the movement operation of the operating object on the operating interface; a second acquisition module, used to acquire the stop position of the operating object on the operating interface when the movement operation is stopped if the stop operation is sensed; a third control module, used to control the virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; and a third processing module, used to process the image of the camera at the predetermined position using a plane-based image deformation model to obtain a reconstructed image.

[0014] According to another aspect of an embodiment of the present invention, another device for reconstructing an image is also provided. The device may include: a second sensing module for sensing the movement operation of an operating object on an operating interface; a third acquisition module for acquiring the stop position of the operating object on the operating interface when the movement operation is stopped if the stop operation is sensed; a fourth control module for controlling the virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is the position of a real viewpoint where a camera is deployed; a second reading module for reading an image of a camera at a predetermined position; and a fourth processing module for processing an image using a plane-based image deformation model to obtain a reconstructed image.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the method for reconstructing an image according to the present invention.

[0016] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is used to run a program, wherein when the program is run, the method for reconstructing an image according to an embodiment of the present invention is executed.

[0017] According to another aspect of an embodiment of the present invention, a system for reconstructing an image is also provided. The system may include: a processor; a memory connected to the processor, and used to provide the processor with instructions for processing the following processing steps: detecting an interactive operation on an operation interface, obtaining a target position of a virtual viewpoint after displacement on the operation interface; controlling the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial degree of freedom position of the virtual viewpoint; and using an image deformation model to process an image at a predetermined position of the virtual viewpoint to obtain a reconstructed image.

[0018] In an embodiment of the present invention, an interactive operation is detected on an operation interface to obtain a target position of a virtual viewpoint after displacement on the operation interface; the virtual viewpoint is controlled to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; an image deformation model is used to process an image of the virtual viewpoint at the predetermined position to obtain a reconstructed image. In other words, the present application obtains the target position of a virtual viewpoint after displacement on the operation interface, and when the virtual viewpoint transitions from the target position to the predetermined position, an image deformation model can be used to process an image of the virtual viewpoint at the predetermined position, thereby achieving the purpose of obtaining a reconstructed image. Since the calculation process of the image deformation model is simple, the calculation speed is fast, and it is adaptable to the computing resources of low-end machines, the technical problem of poor timeliness when reconstructing images from virtual viewpoints is solved, and the technical effect of improving timeliness when reconstructing images from virtual viewpoints is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1A is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for reconstructing an image according to an embodiment of the present invention;

[0021] Figure 1B is a structural schematic diagram of a reconstructed image in a specific application scenario according to an embodiment of the present invention;

[0022] Figure 2 is a flow chart of a method for reconstructing an image according to an embodiment of the present invention;

[0023] Figure 3 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention;

[0024] Figure 4 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention;

[0025] Figure 5 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention;

[0026] Figure 6 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention;

[0027] Figure 7 It is a schematic diagram of a DIBR virtual viewpoint difference according to the related art;

[0028] Figure 8is a schematic diagram of virtual viewpoint interpolation based on Warping and DIBR according to an embodiment of the present invention;

[0029] Fig. 9 is a flow chart of a method for virtual viewpoint interpolation based on Warping and DIBR according to an embodiment of the present invention;

[0030] Fig.10 is a schematic diagram of a device for reconstructing an image according to an embodiment of the present invention;

[0031] Fig.11 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention;

[0032] Fig.12 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention;

[0033] Fig.13 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention; and

[0034] Fig.14 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0038] Depth Image Based Rendering (DIBR) first uses the depth information to project the reference image into a three-dimensional Euclidean space, and then projects the three-dimensional space point onto the imaging plane of the virtual camera;

[0039] Warping is the operation of deforming a two-dimensional image based on the three-dimensional information of the space;

[0040] An image deformation model is a model for processing a two-dimensional image using an image deformation technique, wherein the image deformation technique includes the above-mentioned deformation operation;

[0041] Free viewpoint video, in order to provide a high degree of freedom viewing experience, users can adjust the viewing angle through interactive means during viewing and watch from the free viewpoint they want;

[0042] The 6DoF parameter refers to the six degrees of freedom, specifically the translation parameters along three directions and the rotation parameters around three axes.

[0043] Example 1

[0044] According to an embodiment of the present invention, an embodiment of a method for reconstructing an image is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1A 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for reconstructing an image according to an embodiment of the present invention. Figure 1A As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1A The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1A More or fewer components as shown, or with Figure 1A Different configurations shown.

[0046] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0047] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method for reconstructing an image in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for reconstructing an image of the above-mentioned application program is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0050] It should be noted that, in some optional embodiments, the above Figure 1AThe computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1A This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above-described computer device (or mobile device).

[0051] Figure 1B 1 is a schematic diagram of a structure of a reconstructed image in a specific application scenario according to an embodiment of the present invention, wherein a layout scenario of a system 20 for reconstructing an image is shown, and the system 20 for reconstructing an image may include an acquisition array 21 composed of multiple acquisition devices, a data processing device 22, a server cluster 23 in the cloud (which may include: server 231, server 232, server 233, server 234), a playback control device 24, a playback terminal 25, and an interactive terminal 26. The system 20 for reconstructing an image may obtain a target position after a virtual viewpoint is displaced on an operation interface, and when the virtual viewpoint transitions from the target position to a predetermined position, an image deformation model may be used to process an image at a predetermined position where the virtual viewpoint is located, thereby achieving the purpose of obtaining a reconstructed image.

[0052] Specifically, refer to Figure 1B The acquisition array 21 may include a plurality of cameras, which may be placed in a fan-shaped manner at different positions of the on-site acquisition area according to a preset multi-angle free viewing angle range.

[0053] The data processing device 22 can send instructions to each camera in the acquisition array 21 through a wireless local area network. Based on the instructions sent by the data processing device 22, each acquisition device in the acquisition array 21 transmits the obtained camera position to the data processing device 22.

[0054] The interactive terminal 26 of this embodiment determines the target position of the virtual viewpoint after it is displaced on the operation interface based on the interactive operation. When the data processing device 22 detects that an interactive operation occurs on the operation interface of the interactive terminal 26, the target position of the virtual viewpoint after it is displaced on the operation interface sent by the interactive terminal 26 can be obtained, and the virtual viewpoint can be controlled to transition from the target position to a predetermined position, which can be a camera position that coincides with the spatial freedom position of the virtual viewpoint. The image deformation model is then used to process the image of the virtual viewpoint at the predetermined position to obtain a reconstructed image, and the obtained reconstructed image can be uploaded to the server cluster 23 in the cloud. The server cluster 23 can send the reconstructed image to the interactive terminal 26 for display.

[0055] As another optional implementation, when the data processing device 22 detects an interactive operation occurring on the operating interface of the interactive terminal 26, the target position of the virtual viewpoint sent by the interactive terminal 26 after displacement on the operating interface is obtained, the target position is uploaded to the server cluster 23 in the cloud, the virtual viewpoint is controlled to transition from the target position to the predetermined position through the server cluster 23, and then the image with the virtual viewpoint at the predetermined position is processed using an image deformation model to obtain a reconstructed image, and the obtained reconstructed image can be sent to the interactive terminal 26 for display.

[0056] Then, the playback control device 24 can receive the reconstructed image sent by the server cluster 23, and the playback terminal 25 receives the reconstructed image from the playback control device 24 and plays it in real time. The playback control device 24 can be a manual playback control device or a virtual playback control device. In a specific implementation, a director control device such as a director station can be used as a playback control device in an embodiment of the present invention.

[0057] In this embodiment, the reconstructed image may be an image reconstructed in a video. Generally speaking, an entity in a video will not be completely static. For example, using the above-mentioned system for reconstructing images, the reconstructed image will also change continuously with time.

[0058] This embodiment adopts the above-mentioned system for reconstructing images. On the one hand, the user can directly watch the reconstructed image through the playback terminal 25; on the other hand, when the user watches the image through the interactive terminal 26, the target position of the virtual viewpoint after displacement on the operation interface can be obtained through interactive operation, so as to control the transition of the virtual viewpoint from the target position to the predetermined position, and use the image deformation model to process the image with the virtual viewpoint at the predetermined position to obtain the reconstructed image. It can be understood that the above-mentioned system 20 for reconstructing images can also include only the playback terminal 25 or only the interactive terminal 26, or the same terminal device can be used as the playback terminal 25 and the interactive terminal 26.

[0059] It is understood by those skilled in the art that the user uses gesture interaction to determine the virtual viewpoint position of the video image being viewed on the terminal. During the viewing process, due to the small computing resources of the terminal or the poor video quality, the user stays on the virtual viewpoint for a long time to watch, resulting in a freeze in the video, and thus the timeliness of the image reconstruction at the virtual viewpoint is poor. Therefore, how to ensure the timeliness when reconstructing the image using the virtual viewpoint becomes a difficult problem to solve.

[0060] In view of this, the embodiment of this specification provides a solution. Figure 1A or Figure 1B In the operating environment shown, this application provides Figure 2It should be noted that the method for reconstructing an image in this embodiment can be Figure 1A The mobile terminal or Figure 1B The reconstruction of the image is performed by the system shown.

[0061] Figure 2 FIG. 1 is a flow chart of a method for reconstructing an image according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0062] Step S202: detecting that an interactive operation occurs on the operation interface, and obtaining a target position of the virtual viewpoint after displacement on the operation interface.

[0063] In the technical solution provided in the above step S202 of the present invention, the interactive operation can be an interaction performed by the terminal user on the virtual viewpoint, which can also be called user free viewpoint interaction (DIBR), wherein during the interactive operation on the operation interface, the virtual viewpoint will be displaced on the operation interface. Therefore, the above interactive operation can be a movement operation on the virtual viewpoint, for example, a movement operation starting from point A on the operation interface.

[0064] This embodiment can detect interactive operations occurring on the operating interface, and then obtain the target position where the virtual viewpoint stops after the displacement occurs on the operating interface. For example, the target position is point B on the operating interface, and the interactive operation stops at point B. It can be an operation in which the user's finger is lifted up at the moment of reaching point B. In this case, the displacement of the virtual viewpoint on the operating interface is the displacement between point A and point B.

[0065] Step S204, controlling the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint.

[0066] In the technical solution provided in the above step S204 of the present invention, after obtaining the target position after the virtual viewpoint is displaced on the operation interface, the target position does not necessarily correspond to the real camera position, so that image reconstruction cannot be performed directly at the target position. This embodiment can control the transition of the virtual viewpoint from the target position to the predetermined position, which can be to transition the virtual viewpoint from the target position where it stops along the interactive inertia according to a certain path until the transitioned position coincides with the spatial freedom position of the virtual viewpoint, and the position coincident with the spatial freedom position of the virtual viewpoint is the predetermined position. It should be noted that the above-mentioned predetermined position of this embodiment can be a position where a camera is deployed, wherein the camera can be a real camera, that is, the predetermined position of this embodiment can be a real camera position, which can also be called a real viewpoint position. Optionally, the above-mentioned predetermined position of this embodiment is the real viewpoint position closest to the above-mentioned target position.

[0067] In this embodiment, since the virtual viewpoint is no longer controlled by the interactive operation after the target position of the virtual viewpoint after displacement on the operation interface is obtained, this embodiment controls the virtual viewpoint to transition from the target position to the predetermined position. The interactive inertia involved can be understood as a rule for transitioning the virtual viewpoint, which can also be called inertial transition, which conforms to the user's interactive habits and experience. For example, the virtual viewpoint is controlled to slide from the target position to the predetermined position at a certain speed and then stop. The predetermined position can be the preferred position to which the target position is controlled to transition on the operation interface, or a position that conforms to the user's interactive habits and experience.

[0068] Combination Figure 7 and Figure 8 In an optional embodiment shown, the above embodiment is explained to some extent. The black solid points (such as A1 and A2) can represent virtual viewpoints, and the white hollow points (such as B1, B2, B3, etc.) can represent the positions of the real viewpoints where the real cameras are located. After a moving operation occurs on the operation interface, the virtual viewpoint moves from point A1 to point A2. The virtual viewpoint at position A2 will select a real viewpoint where the real camera is located based on a predetermined control mode and move to this real viewpoint. The control mode at this time may include any of the following: selecting a real viewpoint that is A2 away from the virtual viewpoint; selecting a real viewpoint that has idle computing resources and is closest to the virtual viewpoint; in the event that any one or more of the following situations occurs: the camera where the selected real viewpoint is located is damaged, the waiting processing time exceeds the predetermined time, the real viewpoint with the highest preset weight value is selected.

[0069] Optionally, the certain path in this embodiment may be a predefined smooth path. The spatial degree of freedom position may be a spatial three-degree-of-freedom position (x, y, z).

[0070] Step S206: Use the image deformation model to process the image with the virtual viewpoint located at the predetermined position to obtain a reconstructed image.

[0071] In the technical solution provided in the above step S206 of the present invention, the image deformation model may also be referred to as a Warping model, which may be used to process a two-dimensional image using an image deformation technique, that is, to perform a deformation operation on the two-dimensional image according to the three-dimensional information of the space. In this embodiment, after the virtual viewpoint transitions to the predetermined position, an image having the virtual viewpoint at the predetermined position is obtained. In this embodiment, the image deformation model may be used to process the image having the virtual viewpoint at the predetermined position in the image reconstruction scene, thereby obtaining a reconstructed image.

[0072] In this embodiment, the calculation process of the above-mentioned image deformation model is simple, so this embodiment is faster than the interpolation method of virtual viewpoint reconstruction (DIBR) based on the depth map, adapts to the computing resources of low-end machines, ensures the timeliness (real-time) of virtual viewpoint reconstruction of images, and the image quality in complex image reconstruction scenarios.

[0073] Through the above steps S204 to S206 of the present application, it is detected that an interactive operation occurs on the operation interface, and the target position of the virtual viewpoint after displacement on the operation interface is obtained; the virtual viewpoint is controlled to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; and the image deformation model is used to process the image at the predetermined position of the virtual viewpoint to obtain a reconstructed image. In other words, this embodiment obtains the target position of the virtual viewpoint after displacement on the operation interface, and when the virtual viewpoint transitions from the target position to the predetermined position, the image deformation model can be used to process the image at the predetermined position of the virtual viewpoint, thereby achieving the purpose of obtaining the reconstructed image. Since the calculation process of the image deformation model is simple, the calculation speed is fast, and it is adaptable to the computing resources of low-end machines, the technical problem of poor timeliness when reconstructing images from virtual viewpoints is solved, and the technical effect of improving the timeliness when reconstructing images from virtual viewpoints is achieved.

[0074] The above method of this embodiment is further introduced below.

[0075] As an optional implementation, if it is detected that the interactive operation at the target position is interrupted, a transition operation of executing the virtual viewpoint from the target position to a predetermined position is initiated.

[0076] In this embodiment, the interactive operation occurring on the operation interface may be a continuous operation process. In the process of displacement of the virtual viewpoint on the operation interface, the interactive operation occurring on the operation interface is detected. If it is detected that the interactive operation is interrupted when the virtual viewpoint moves to the target position, for example, the interactive operation stays at the target position, then the transition operation of the virtual viewpoint from the target position to the predetermined position is started, that is, once the interruption of the interactive operation at the target position is detected, the above transition operation is triggered.

[0077] As an optional implementation, in step S204, in the process of controlling the virtual viewpoint to transition from the target position to the predetermined position, the method further includes: using the DIBR virtual viewpoint interpolation algorithm to generate an interpolation of a transition image frame; and reconstructing the image during the transition process based on the interpolation of the transition image frame.

[0078] In this embodiment, in the intermediate process of controlling the transition of the virtual viewpoint from the target position to the predetermined position, the DIBR virtual viewpoint interpolation algorithm can be used to generate the interpolation of the transition image frame until the virtual viewpoint transitions from the target position to the predetermined position, for example, until the spatial three-degree-of-freedom position (x, y, z) of the virtual viewpoint coincides with the real camera position. After the spatial three-degree-of-freedom position (x, y, z) of the virtual viewpoint coincides with the real camera position, the difference between the interpolated planned 6DoF position and the real camera position is only the remaining three rotational degrees of freedom (the degrees of freedom of rotation around the three axes of x, y, and z, respectively).

[0079] After the interpolation of the transition image frame is generated, the image in the process of the virtual viewpoint transitioning from the target position to the predetermined position can be reconstructed based on the interpolation of the transition image frame.

[0080] As an optional implementation, in step S204, after controlling the virtual viewpoint to transition from the target position to the predetermined position, the Warping image deformation algorithm is called to switch to using the image deformation model to perform image reconstruction.

[0081] In this embodiment, the image deformation algorithm may be a Warping image deformation algorithm, which may also be referred to as a Warping operation. Since the calculation is simple, only one projection is required, and no other pre- and post-processing operations are required, and therefore, it is faster than DIBR interpolation. After controlling the virtual viewpoint to transition from the target position to the predetermined position, this embodiment may call the Warping image deformation algorithm, that is, switch the DIBR virtual viewpoint interpolation algorithm to the Warping image deformation algorithm, thereby using the image deformation model to perform image reconstruction. It is precisely because of the above-mentioned characteristics of the Warping operation that the timeliness of the virtual viewpoint image reconstruction and the image quality in complex image reconstruction scenarios are ensured.

[0082] This embodiment provides a DIBR method combined with a Warping function, which can support richer terminal interaction designs and can flexibly configure different interaction methods in combination with actual terminals and scenario conditions, thereby creating a good experience for users.

[0083] As an optional implementation, an image deformation model is used to process an image with a virtual viewpoint at a predetermined position, including: reading a corresponding image from a camera based on the predetermined position of the virtual viewpoint; and using an image deformation model to process the read image according to the rotational degrees of freedom in the spatial coordinate system where the virtual viewpoint is located.

[0084] In this implementation, when using the image deformation model to process an image with a virtual viewpoint at a predetermined position, the corresponding image can be read from the camera based on the predetermined position of the virtual viewpoint. The image read from the camera, that is, the image with the virtual viewpoint at the predetermined position, can also be called the original camera image. Then, the spatial coordinate system where the virtual viewpoint is located is determined, and the rotational degrees of freedom on the spatial coordinate system are determined, for example, three rotational degrees of freedom. According to the rotational degrees of freedom on the spatial coordinate system where the virtual viewpoint is located, the image deformation model is used to process the read image, and then the next viewpoint reconstruction function is updated. In this way, the timeliness problem of terminal reconstruction DIBR and the image quality problem in complex reconstruction scenes can be solved at the same time. At the same time, this embodiment can solve the smoothness problem of users switching free viewpoints by inertial DIBR interpolation transition from the virtual viewpoint position of user interaction to the real camera position.

[0085] As an optional implementation, the spatial degree of freedom is the coordinate value of the virtual viewpoint in the spatial coordinate system, and the rotational degree of freedom is the degree of freedom of rotation around the coordinate axis of the spatial coordinate system.

[0086] In the above scheme of this embodiment, the spatial degree of freedom is the coordinate value of the virtual viewpoint in the spatial coordinate system, for example, (x, y, z), and the rotational degree of freedom is the degree of freedom of rotation around the coordinate axis of the spatial coordinate system. For example, there are 3 degrees of freedom, which can be the degrees of freedom of rotation around the three axes of x, y, and z respectively.

[0087] As an optional implementation, in step S204, before controlling the virtual viewpoint to transition from the target position to the predetermined position, the method further includes: obtaining the distance between the target position where the virtual viewpoint is currently located and the predetermined position; if the distance exceeds a threshold, controlling the virtual viewpoint to move from the target position to the viewpoint position of the camera with the highest priority according to a preset camera priority; if the distance does not exceed the threshold, executing the step of transitioning the virtual viewpoint from the target position to the predetermined position.

[0088] In this embodiment, before controlling the virtual viewpoint to transition from the target position to the predetermined position, the distance between the target position where the virtual viewpoint is currently located after displacement on the operation interface and the predetermined position can also be obtained, and the predetermined position is the position where the camera is deployed. Then, it is determined whether the distance between the target position and the predetermined position exceeds the threshold. If it is determined that the distance between the target position and the predetermined position exceeds the threshold, the virtual viewpoint is not executed to transition from the target position to the predetermined position, but the preset camera priority is first obtained, and the virtual viewpoint is controlled to move from the target position to the viewpoint position where the camera with the highest priority is located according to the preset camera priority; optionally, if it is determined that the distance between the target position and the predetermined position does not exceed the threshold, the embodiment can continue to execute the transition of the virtual viewpoint from the target position to the predetermined position, which can be to transition the virtual viewpoint from the target position where it stops along the interaction inertia according to a certain path until the transitioned position coincides with the spatial freedom position of the virtual viewpoint, and the position coincident with the spatial freedom position of the virtual viewpoint is also the predetermined position, so that the embodiment achieves the purpose of controlling the position to which the virtual viewpoint transitions from the target position according to the distance between the target position where the virtual viewpoint is currently located and the predetermined position.

[0089] As an optional implementation, in step S202, after detecting that an interactive operation occurs on the operation interface, the method further includes: popping up a prompt message for indicating at least one selection control; and selecting a display resolution of the reconstructed image by triggering any one of the selection controls.

[0090] In this embodiment, the interactive operation may be an interaction performed by the terminal user on the virtual viewpoint. In order to enable the user to more conveniently set the reconstructed image to be acquired to meet the needs, for example, in order to enable the user to more conveniently set the display resolution of the reconstructed image to be acquired to meet the requirements for the resolution of the reconstructed image to be acquired, this embodiment may pop up a prompt message for indicating at least one selection control after detecting an interactive operation on the operation interface, wherein the selection control may be a functional control for selecting the resolution of the reconstructed image to be acquired, and the prompt message is also the interaction indication information to prompt the user to perform the selection operation.

[0091] After the prompt information indicating at least one selection control is popped up, the user's selection operation on any selection control can be received, and then in response to the selection operation on any selection control, the display resolution of the reconstructed image can be selected, wherein each selection control can correspond to a display resolution of the reconstructed image, and multiple selection controls can correspond to different display resolutions of the reconstructed images, respectively, to meet the demand for the display resolution of the reconstructed image to be acquired to have a variety of settings, for example, in the scenario where the user is watching a movie, there will be a reminder of the low display resolution of the reconstructed image. In this way, when the image deformation model is used to process an image with a virtual viewpoint at a predetermined position, the reconstructed image acquired is the display resolution corresponding to the selection control triggered by the user.

[0092] The embodiment of the present invention also provides another method for reconstructing an image.

[0093] Figure 3 FIG. 4 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention. Figure 3 As shown, the method may include the following steps:

[0094] Step S302: detecting that an interactive operation of controlling a virtual viewpoint to move from a first position to a second position occurs on the operation interface.

[0095] In the technical solution provided in the above step S302 of the present invention, the interactive operation can be an interaction performed by the user on the virtual viewpoint, wherein during the interactive operation on the operation interface, the virtual viewpoint will be displaced on the operation interface, and the virtual viewpoint can be controlled to move from a first position to a second position, and the second position can be the position where the virtual viewpoint stays after the displacement on the operation interface, for example, the position where the user's finger is lifted on the operation interface.

[0096] Step S304, controlling the virtual viewpoint to transition from the second position to the third position, wherein the third position is the position of the real viewpoint where the camera is deployed.

[0097] In the technical solution provided in the above step S304 of the present invention, after an interactive operation for controlling the virtual viewpoint to move from a first position to a second position is detected on the operation interface, the virtual viewpoint can be controlled to transition from the second position to a third position, and the virtual viewpoint can be transitioned from the second position where it stays along a certain path along the interactive inertia until it transitions to the third position. The third position can be deployed with a real viewpoint of the camera, which coincides with the spatial freedom position of the virtual viewpoint.

[0098] Optionally, the certain path in this embodiment may be a predefined smooth path. The spatial degree of freedom position may be a spatial three-degree-of-freedom position (x, y, z).

[0099] Step S306, reading the image of the camera located at the third position.

[0100] In the technical solution provided in the above step S306 of the present invention, after the virtual viewpoint is controlled to transition from the second position to the third position, the image of the camera located at the third position can be read.

[0101] In this embodiment, after the virtual viewpoint transitions to the third position, the image of the camera with the virtual viewpoint located at the third position is read.

[0102] Step S308: Process the image using a plane-based image deformation model to obtain a reconstructed image.

[0103] In the technical solution provided in the above step S308 of the present invention, after reading the image of the camera located at the third position, the image can be processed using a plane-based image deformation model to obtain a reconstructed image.

[0104] In this embodiment, after the virtual viewpoint transitions to the third position and the image of the camera at the third position is read, the image of the camera at the third position of the virtual viewpoint can be processed using a plane-based image deformation model in the image reconstruction scene to obtain a reconstructed image. The plane-based image deformation model is a two-dimensional image deformation model.

[0105] In this embodiment, the calculation process of the above-mentioned plane-based image deformation model is simple, so this embodiment is faster than the interpolation method of DIBR based on the depth map, thereby adapting to the computing resources of low-end machines, ensuring the timeliness when reconstructing images from virtual viewpoints, and the image quality in complex image reconstruction scenarios.

[0106] As an optional implementation, the real viewpoint position closest to the second position is selected as the third position.

[0107] In this embodiment, before controlling the virtual viewpoint to transition from the second position to the third position, at least one real viewpoint position can be acquired, and then the real viewpoint position closest to the second position is determined from the at least one real viewpoint position, and used as the third position to which the virtual viewpoint needs to be controlled to transition from the second position.

[0108] The embodiment of the present invention also provides another method for reconstructing an image.

[0109] Figure 4 FIG. 4 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention. Figure 4 As shown, the method may include the following steps:

[0110] Step S402: sensing the interactive operation of the operating object on the operating interface.

[0111] In the technical solution provided in step S402 of the present invention, the interactive operation may be an interaction with a virtual viewpoint performed by an operating object, and during the interactive operation on the operating interface, the virtual viewpoint may be displaced on the operating interface, wherein the operating object may be a user. This embodiment senses the interactive operation of the operating object on the operating interface.

[0112] Step S404, in response to the interactive operation, displaying the displacement of the virtual viewpoint on the operation interface based on the interactive operation, wherein the displacement is from a first position to a second position.

[0113] In the technical solution provided in the above step S404 of the present invention, the virtual viewpoint can be displaced on the operation interface based on the interactive operation. After sensing the interactive operation of the operation object on the operation interface, this embodiment can respond to the interactive operation and display the displacement of the virtual viewpoint on the operation interface based on the interactive operation, wherein the displacement is the displacement from the first position to the second position.

[0114] In this embodiment, in response to the above-mentioned interactive operation, the second position where the virtual viewpoint stays after being displaced from the first position on the operating interface can be obtained, and the displacement between the first position and the second position can be determined, which is also the displacement occurring on the operating interface due to the interactive operation, and then the displacement occurring on the operating interface due to the interactive operation is displayed.

[0115] Step S406, displaying the virtual viewpoint transitioning from the second position to the third position, wherein the third position is the position of the real viewpoint where the camera is deployed.

[0116] In the technical solution provided in the above step S406 of the present invention, after the displacement of the interactive operation on the operation interface is displayed, the virtual viewpoint is displayed to transition from the second position to the third position.

[0117] In this embodiment, the virtual viewpoint can be controlled to transition from the second position to the third position. The virtual viewpoint can be transitioned from the second position where it stays along a certain path along the interaction inertia until it transitions to the third position. The third position can be deployed with the real viewpoint of the camera, which coincides with the spatial freedom position of the virtual viewpoint, thereby displaying the result of the virtual viewpoint transitioning from the second position to the third position.

[0118] Optionally, the certain path in this embodiment may be a predefined smooth path. The spatial degree of freedom position may be a spatial three-degree-of-freedom position (x, y, z).

[0119] Step S408 , displaying a reconstructed image generated by processing a target image using a plane-based image deformation model, wherein the target image is an image captured by a camera located at a third position.

[0120] In the technical solution provided in the above step S408 of the present invention, after the display virtual viewpoint transitions from the second position to the third position, a reconstructed image generated by processing the target image using a plane-based image deformation model is displayed.

[0121] In this embodiment, after the virtual viewpoint transitions to the third position, the image of the camera with the virtual viewpoint located at the third position is read, and the target image captured by the camera located at the third position is read, and then the above target image is processed using a plane-based image deformation model to obtain and display the reconstructed image.

[0122] In this embodiment, the calculation process of the above-mentioned plane-based image deformation model is simple, so this embodiment is faster than the interpolation method of DIBR based on the depth map, thereby adapting to the computing resources of low-end machines, ensuring the timeliness when reconstructing images from virtual viewpoints, and the image quality in complex image reconstruction scenarios.

[0123] The embodiment of the present invention also provides another method for reconstructing an image.

[0124] Figure 5 FIG. 4 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention. Figure 5 As shown, the method may include the following steps:

[0125] Step S502: sensing the movement operation of the operation object on the operation interface.

[0126] In the technical solution provided in the above step S502 of the present invention, the moving operation may be a moving operation on the virtual viewpoint performed by the operating object. During the moving operation on the operating interface, the virtual viewpoint will be displaced on the operating interface corresponding to the trajectory of the moving operation, wherein the operating object may be a finger of the user, and thus the moving operation may be a sliding operation of the finger of the user. This embodiment senses the above moving operation of the operating object on the operating interface.

[0127] Step S504: if it is sensed that the moving operation is terminated, the stopping position of the operating object on the operating interface when the moving operation is terminated is obtained.

[0128] In the technical solution provided in the above step S504 of the present invention, after sensing the movement operation of the operation object on the operation interface, if it is sensed that the movement operation is terminated, the stop position of the operation object on the operation interface when the movement operation is terminated is obtained.

[0129] In this embodiment, the moving operation occurring on the operation interface is a continuous operation process. If the movement operation is sensed to be terminated, the stop position of the operation object on the operation interface when the movement operation is terminated can be obtained, that is, the movement operation is interrupted at the above-mentioned stop position.

[0130] Step S506, controlling the virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint.

[0131] In the technical solution provided in the above step S506 of the present invention, after obtaining the stop position of the operation object on the operation interface when the moving operation is terminated, the virtual viewpoint located at the stop position can be controlled to transition to a predetermined position.

[0132] In this embodiment, the virtual viewpoint moves with the moving operation. After obtaining the stop position of the operating object on the operating interface when the moving operation is terminated, the virtual viewpoint at the stop position can be triggered to control the transition to a predetermined position. The predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint, which can be a real camera position or a real viewpoint position.

[0133] Step S508: Use the plane-based image deformation model to process the image of the camera at the predetermined position to obtain a reconstructed image.

[0134] In the technical solution provided in the above step S508 of the present invention, after controlling the virtual viewpoint at the stop position to transition to the predetermined position, the image of the camera at the predetermined position can be processed using a plane-based image deformation model to obtain a reconstructed image.

[0135] In this embodiment, after the virtual viewpoint transitions to the predetermined position, the image of the camera with the virtual viewpoint located at the predetermined position is read, and the image captured by the camera located at the third position is read, and then the image is processed using a plane-based image deformation model to obtain a reconstructed image.

[0136] In this embodiment, the calculation process of the above-mentioned plane-based image deformation model is simple, so this embodiment is faster than the interpolation method of DIBR based on the depth map, thereby adapting to the computing resources of low-end machines, ensuring the timeliness when reconstructing images from virtual viewpoints, and the image quality in complex image reconstruction scenarios.

[0137] The embodiment of the present invention also provides another method for reconstructing an image.

[0138] Figure 6 FIG. 4 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention. Figure 6 As shown, the method may include the following steps:

[0139] Step S602: sensing the movement operation of the operation object on the operation interface.

[0140] In the technical solution provided in the above step S602 of the present invention, the moving operation may be a moving operation on the virtual viewpoint performed by the operating object. During the moving operation on the operating interface, the virtual viewpoint will be displaced on the operating interface corresponding to the trajectory of the moving operation, wherein the operating object may be a finger of the user, and the moving operation may be a sliding operation of the user. This embodiment senses the above moving operation of the operating object on the operating interface.

[0141] Step S604: if it is sensed that the moving operation is terminated, the stopping position of the operating object on the operating interface when the moving operation is terminated is obtained.

[0142] In the technical solution provided in the above step S604 of the present invention, after sensing the movement operation of the operation object on the operation interface, if it is sensed that the movement operation is terminated, the stop position of the operation object on the operation interface when the movement operation is terminated is obtained.

[0143] In this embodiment, the moving operation occurring on the operation interface is a continuous operation process. If the movement operation is sensed to be terminated, the stop position of the operation object on the operation interface when the movement operation is terminated can be obtained, that is, the movement operation is interrupted at the above-mentioned stop position.

[0144] Step S606, controlling the virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is the position of the real viewpoint where the camera is deployed.

[0145] In the technical solution provided in the above step S606 of the present invention, after obtaining the stop position of the operation object on the operation interface when the moving operation is terminated, the virtual viewpoint located at the stop position is controlled to transition to a predetermined position.

[0146] In this embodiment, the virtual viewpoint moves with the moving operation. After obtaining the stop position of the operating object in the operating interface when the moving operation is terminated, the virtual viewpoint at the stop position can be triggered to control the transition to a predetermined position, which is the position of the real viewpoint where the camera is deployed, and the camera position that can coincide with the spatial freedom position of the virtual viewpoint.

[0147] Step S608, reading the image of the camera located at the predetermined position.

[0148] In the technical solution provided in the above step S608 of the present invention, after the virtual viewpoint located at the stop position is controlled to transition to the predetermined position, the image of the camera located at the predetermined position is read.

[0149] In this embodiment, after the virtual viewpoint transitions to the predetermined position, the image of the camera with the virtual viewpoint located at the predetermined position is read.

[0150] Step S610: Process the image using a plane-based image deformation model to obtain a reconstructed image.

[0151] In the technical solution provided in the above step S610 of the present invention, after reading the image of the camera located at the predetermined position, the image is processed using a plane-based image deformation model to obtain a reconstructed image.

[0152] In this embodiment, after the virtual viewpoint transitions to the predetermined position and the image of the camera at the predetermined position is read, the image of the camera at the predetermined position of the virtual viewpoint can be processed using a plane-based image deformation model in the image reconstruction scene to obtain a reconstructed image.

[0153] An embodiment of the present invention also provides another method for reconstructing an image, which may include: during a live broadcast, detecting that a moving operation is received on the live broadcast screen; obtaining a target position of a virtual viewpoint after displacement on the live broadcast screen; controlling the virtual viewpoint to move from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; using an image deformation model to process an image where the virtual viewpoint is located at the predetermined position to obtain a reconstructed image.

[0154] The method for reconstructing an image in this embodiment can be applied in a live broadcast scene, for example, the live broadcast scene can be a transaction type live broadcast scene, which is not specifically limited here. The moving operation in this embodiment can be a moving operation of a virtual viewpoint performed by a terminal user on a live broadcast screen in a live broadcast scene, which can also be called user free viewpoint interaction, wherein, during the moving operation on the live broadcast screen, the virtual viewpoint will be displaced on the live broadcast screen, for example, the displacement corresponding to the moving operation starting from point A on the live broadcast screen.

[0155] This embodiment can detect interactive operations occurring on the live screen, and then obtain the target position where the virtual viewpoint stops after it is displaced on the live screen. For example, the target position is point B on the live screen, and the moving operation stops at point B. It can be an operation in which the user's finger is lifted up at the moment of reaching point B. In this case, the displacement of the virtual viewpoint on the live screen is the displacement between point A and point B.

[0156] After obtaining the target position of the virtual viewpoint after displacement on the live screen, the target position does not necessarily correspond to the real camera position, so that image reconstruction cannot be performed directly at the target position. This embodiment can control the virtual viewpoint to transition from the target position to the predetermined position, which can be to transition the virtual viewpoint from the target position along a certain path along the interaction inertia until the transitioned position coincides with the spatial freedom position of the virtual viewpoint, and the position coincident with the spatial freedom position of the virtual viewpoint is the predetermined position. Optionally, the predetermined position of this embodiment is the real viewpoint position closest to the target position.

[0157] In this embodiment, since the virtual viewpoint is no longer controlled by the interactive operation after the target position of the virtual viewpoint is obtained after displacement on the live screen, this embodiment controls the transition of the virtual viewpoint from the target position to the predetermined position. The interactive inertia involved can be understood as a rule for transitioning the virtual viewpoint.

[0158] Optionally, the certain path of this embodiment may be a smooth path predefined on the live screen. The spatial freedom position may be a spatial three-degree-of-freedom position (x, y, z).

[0159] In this embodiment, after the virtual viewpoint transitions to the predetermined position, there is an image with the virtual viewpoint located at the predetermined position. This embodiment can use an image deformation model to process the image with the virtual viewpoint located at the predetermined position to obtain a reconstructed image.

[0160] In this embodiment, the calculation process of the above-mentioned image deformation model is simple, so this embodiment is faster than the interpolation method of virtual viewpoint reconstruction based on depth map, adapts to the computing resources of low-end machines, ensures the timeliness (real-time) of virtual viewpoint reconstruction of images, and the image quality in complex image reconstruction scenarios.

[0161] In this embodiment, the calculation process of the above-mentioned plane-based image deformation model is simple, so this embodiment is faster than the interpolation method of DIBR based on the depth map, thereby adapting to the computing resources of low-end machines, ensuring the timeliness when reconstructing images from virtual viewpoints, and the image quality in complex image reconstruction scenarios.

[0162] This embodiment provides a DIBR method combined with a Warping function, which can support richer terminal interaction designs and can flexibly configure different interaction methods in combination with actual terminals and scenario conditions, thereby creating a good experience for users.

[0163] This embodiment uses a combination of inertial DIBR transition and real viewpoint position Warping to avoid the problem of low timeliness of calculation of low-end machines after staying at the virtual viewpoint position, and the problem of low image quality in related complex scenes. It can adapt to the computing resources of mid- and low-end machines and improve the image quality in complex scenes, thereby making the free viewpoint video experience more universal.

[0164] Example 2

[0165] The preferred implementation manner of the above method of this embodiment is further introduced below by way of examples.

[0166] Figure 7 Schematic diagram of a DIBR virtual viewpoint difference in the related art. Figure 7 As shown, Figure 8 As shown, the black thin solid arrow is used to indicate the real direction of the original camera, the hollow circle is used to indicate the original camera, and the black solid circle is used to indicate the virtual camera. This embodiment uses a camera array with sparse viewpoints to obtain a larger range of free viewpoints, wherein the black solid circle is used to indicate the position where the virtual viewpoint stays when performing virtual viewpoint interaction, and the hollow circle is used to indicate the position of the real camera determined by interpolation through the DIBR method. In the related art, the main interaction method is to determine the position of the virtual viewpoint where the user wants to stay through the gesture interaction of the terminal user, and interpolate through the DIBR method according to the 6DoF parameters of the virtual viewpoint to obtain the viewing image of the user at the virtual viewpoint position.

[0167] However, a single DIBR function cannot meet the diverse needs of users. This is mainly due to the following two reasons: on some mid- and low-end models, the terminal's computing resources are not abundant, which will lead to the high-complexity DIBR calculations being unable to achieve real-time purposes. Therefore, if the user stays on the virtual viewpoint for a long time to watch, the video will be stuck; in addition, if the scene is very complex, or the compression bit rate of the free viewpoint video is relatively low, the quality of the depth map cannot be guaranteed, which further causes the user to continue to stay in the virtual viewpoint position, and the subjective experience of the interpolated image quality is also seriously degraded compared to the original viewpoint.

[0168] Based on the above two reasons, this embodiment proposes that a method based on a simplified version of Warping needs to be added to the reference software of DIBR to meet the diverse and universal interaction requirements.

[0169] Figure 8 FIG. 1 is a schematic diagram of a virtual viewpoint interpolation based on Warping and DIBR according to an embodiment of the present invention. Figure 8As shown, the thin black solid arrow is used to indicate the real direction of the original camera, and the thick black solid arrow is used to indicate the direction of the virtual camera after the path is planned (all the black thick solid arrows are seen by the user). The hollow circle is used to indicate the original camera, and the solid black circle is used to indicate the virtual camera. Fig. 9 1 is a flowchart of a method for virtual viewpoint interpolation based on Warping and DIBR according to an embodiment of the present invention. Figure 8 and Fig. 9 It can be seen that in the virtual viewpoint interpolation based on Warping, when step S901 is executed, the user performs virtual viewpoint interaction (DIBR) and stays at the position of the black solid circle, the virtual viewpoint can be transitioned and interpolated along a smooth path along the interaction inertia by planning a path until the spatial three-degree-of-freedom position (x, y, z) of the virtual viewpoint coincides with the real camera position. For example, the virtual viewpoint moves from the second black circle to the position of the adjacent left hollow circle, and the position of the adjacent left hollow circle corresponds to the position of the real camera, thereby realizing step S902, inertial transition of the virtual viewpoint to the real viewpoint, and then executing step S903 to determine the real viewpoint position through Warping. After the spatial three-degree-of-freedom position (x, y, z) of the virtual viewpoint is overlapped with the real camera position, the difference between the interpolated planned 6DoF position and the real camera position is only the remaining three rotational degrees of freedom (the degrees of freedom for rotation around the x, y, and z axes). After executing step S903, re-execute S901. Figure 8 In the figure, the thin solid arrows correspond to the hollow circles, the thick solid arrows correspond to the black solid circles, and the solid dashed arrows correspond to the paths.

[0170] After the spatial three-degree-of-freedom position (x, y, z) of the virtual viewpoint is overlapped with the real camera position, the viewpoint reconstruction function can be updated through the original camera image according to the Warping operation of the three rotational degrees of freedom, so that the above two problems can be solved at the same time: the timeliness of the terminal reconstruction DIBR and the image quality problem in complex reconstruction scenes. At the same time, this embodiment can solve the smoothness problem when the user switches the free viewpoint by performing an inertial DIBR interpolation transition from the virtual viewpoint position of the user interaction to the real camera position.

[0171] Combination Figure 7 and Figure 8It can be seen that the black solid points (such as A1 and A2) can represent virtual viewpoints, and the white hollow points (such as B1, B2, B3, etc.) can represent the position of the real viewpoint where the real camera is located. After the moving operation occurs on the operation interface, the virtual viewpoint moves from point A1 to point A2. The virtual viewpoint at position A2 will select a real viewpoint where a real camera is located based on a predetermined control mode and move to this real viewpoint. The control mode at this time may include any of the following: selecting a real viewpoint that is A2 away from the virtual viewpoint; selecting a real viewpoint with idle computing resources and closest to the virtual viewpoint; in the event that any one or more of the following situations occurs: the camera where the selected real viewpoint is located is damaged, the waiting processing time exceeds the predetermined time, select the real viewpoint with the highest preset weight value.

[0172] In this embodiment, the above-mentioned Warping operation is faster than DIBR interpolation because of its simple calculation, only one projection is required, and no other pre- and post-processing operations are required. Considering the universality of the standard in different scenarios and models, this embodiment provides a DIBR method combined with the Warping function, which can support richer terminal interaction designs, and can flexibly configure different interaction methods in combination with actual terminals and scenario conditions to improve user experience.

[0173] In this embodiment, a combination of inertial DIBR transition and real viewpoint position Warping is used to avoid the problem of real-time calculation of low-end machines after staying at the virtual viewpoint position, as well as image quality problems in related complex scenes.

[0174] This embodiment provides a DIBR synthesis method with a Warping function support having low computational complexity. This method can simultaneously adapt to the computing resources of mid- and low-end machines and image quality issues in complex scenes, thereby making the free viewpoint video experience more universal.

[0175] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0177] Example 3

[0178] According to an embodiment of the present invention, a device for implementing the above-mentioned method for reconstructing an image is also provided. It should be noted that the device for reconstructing an image in this embodiment can be used to implement the embodiment of the present invention. Figure 2 The method of reconstructing the image is shown.

[0179] Fig.10 FIG. 1 is a schematic diagram of a device for reconstructing an image according to an embodiment of the present invention. Fig.10 As shown, the image reconstruction device 100 may include: a first acquisition module 101 , a first control module 102 , and a second processing module 103 .

[0180] The first acquisition module 101 is used to detect an interactive operation occurring on the operation interface and acquire a target position of a virtual viewpoint after displacement on the operation interface.

[0181] The first control module 102 is used to control the virtual viewpoint to transition from a target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint.

[0182] The second processing module 103 is used to process the image with the virtual viewpoint located at the predetermined position by using the image deformation model to obtain a reconstructed image.

[0183] It should be noted that the first acquisition module 101, the first control module 102 and the second processing module 103 correspond to steps S202 to S206 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0184] According to an embodiment of the present invention, another device for implementing the above-mentioned method for reconstructing an image is also provided. It should be noted that the device for reconstructing an image in this embodiment can be used to implement the embodiment of the present invention. Figure 3The method of reconstructing the image is shown.

[0185] According to an embodiment of the present invention, another device for implementing the above-mentioned method for reconstructing an image is also provided. It should be noted that the device for reconstructing an image in this embodiment can be used to implement the embodiment of the present invention. Figure 4 The method of reconstructing the image is shown.

[0186] Fig.11 FIG. 1 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention. Fig.11 As shown, the image reconstruction device 110 may include: a detection module 111 , a second control module 112 , a first reading module 113 and a second processing module 114 .

[0187] The detection module 111 is used to detect an interactive operation on the operation interface for controlling the virtual viewpoint to move from a first position to a second position.

[0188] The second control module 112 is used to control the virtual viewpoint to transition from the second position to the third position, wherein the third position is the position of the real viewpoint where the camera is deployed.

[0189] The first reading module 113 is used to read the image of the camera located at the third position.

[0190] The second processing module 114 is configured to process the image using a plane-based image deformation model to obtain a reconstructed image.

[0191] It should be noted that the detection module 111, the second control module 112, the first reading module 113 and the second processing module 114 correspond to steps S302 to S308 in Example 1, and the examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0192] According to an embodiment of the present invention, another device for implementing the above-mentioned method for reconstructing an image is also provided. It should be noted that the device for reconstructing an image in this embodiment can be used to implement the embodiment of the present invention. Figure 5 The method of reconstructing the image is shown.

[0193] Fig.12 FIG. 1 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention. Fig.12 As shown, the image reconstruction device 120 may include: a first sensing module 121 , a second acquisition module 122 , a third control module 123 and a third processing module 124 .

[0194] The first sensing module 121 is used to sense the movement operation of the operating object on the operating interface.

[0195] The second acquisition module 122 is used for acquiring the stopping position of the operation object on the operation interface when the moving operation is stopped, if the moving operation is sensed to be stopped.

[0196] The third control module 123 is used to control the virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint.

[0197] The third processing module 124 is configured to process the image of the camera at a predetermined position using a plane-based image deformation model to obtain a reconstructed image.

[0198] It should be noted that the first sensing module 121, the second acquisition module 122, the third control module 123 and the third processing module 124 correspond to steps S502 to S508 in Example 1, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0199] According to an embodiment of the present invention, another device for implementing the above-mentioned method for reconstructing an image is also provided. It should be noted that the device for reconstructing an image in this embodiment can be used to implement the embodiment of the present invention. Figure 6 The method of reconstructing the image is shown.

[0200] Fig.13 FIG. 1 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention. Fig.13 As shown, the image reconstruction device 130 may include: a second sensing module 131 , a third acquisition module 132 , a fourth control module 133 , a second reading module 134 and a fourth processing module 135 .

[0201] The second sensing module 131 is used to sense the movement operation of the operating object on the operating interface.

[0202] The third acquisition module 132 is used to acquire the stop position of the operation object on the operation interface when the movement operation is stopped, if the movement operation is sensed to be stopped.

[0203] The fourth control module 133 is used to control the virtual viewpoint located at the stop position to transition to a predetermined position, wherein the predetermined position is the position of the real viewpoint where the camera is deployed.

[0204] The second reading module 134 is used to read the image of the camera located at a predetermined position.

[0205] The fourth processing module 135 is used to process the image using a plane-based image deformation model to obtain a reconstructed image.

[0206] It should be noted that the second sensing module 131, the third acquisition module 132, the fourth control module 133, the second reading module 134 and the fourth processing module 135 correspond to steps S602 to S610 in Example 1, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0207] In the device for reconstructing images of this embodiment, the target position of the virtual viewpoint after displacement on the operation interface is obtained. When the virtual viewpoint transitions from the target position to the predetermined position, the image at the predetermined position of the virtual viewpoint can be processed using an image deformation model, thereby achieving the purpose of obtaining the reconstructed image. Since the calculation process of the image deformation model is simple, the calculation speed is fast, and it is adaptable to the computing resources of low-end machines, the technical problem of poor timeliness when reconstructing images from virtual viewpoints is solved, and the technical effect of improving the timeliness when reconstructing images from virtual viewpoints is achieved.

[0208] Example 4

[0209] The embodiment of the present invention may provide a system for reconstructing an image, and the system for reconstructing an image may include a computer terminal, and the computer terminal may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0210] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0211] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the method for reconstructing an image: detecting an interactive operation occurring on the operation interface, and obtaining a target position of the virtual viewpoint after displacement on the operation interface; controlling the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; and using an image deformation model to process an image in which the virtual viewpoint is located at the predetermined position to obtain a reconstructed image.

[0212] Optionally, Fig.14 is a structural block diagram of a computer terminal according to an embodiment of the present invention. Fig.14 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 142 , a memory 144 , and a transmission device 146 .

[0213] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the method and device for reconstructing images in the embodiments of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for reconstructing images. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0214] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: detect the occurrence of interactive operations on the operation interface, and obtain the target position of the virtual viewpoint after the displacement on the operation interface; control the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; use the image deformation model to process the image of the virtual viewpoint at the predetermined position to obtain a reconstructed image.

[0215] Optionally, the processor may further execute program code of the following steps: if it is detected that the interactive operation at the target position is interrupted, then starting a transition operation of executing the virtual viewpoint from the target position to a predetermined position.

[0216] Optionally, the processor may also execute the program code of the following steps: in the process of controlling the virtual viewpoint to transition from the target position to the predetermined position, using the DIBR virtual viewpoint interpolation algorithm to generate the interpolation of the transition image frame; and reconstructing the image during the transition process based on the interpolation of the transition image frame.

[0217] Optionally, the processor may further execute program codes of the following steps: after controlling the virtual viewpoint to transition from the target position to the predetermined position, calling the Warping image deformation algorithm and switching to use the image deformation model to perform image reconstruction.

[0218] Optionally, the processor may also execute program code of the following steps: based on a predetermined position of the virtual viewpoint, reading a corresponding image from the camera; and processing the read image using an image deformation model according to the rotational degrees of freedom in the spatial coordinate system where the virtual viewpoint is located.

[0219] Optionally, the processor may also execute the following program code: before controlling the virtual viewpoint to transition from the target position to the predetermined position, obtaining the distance between the target position where the virtual viewpoint is currently located and the predetermined position; if the distance exceeds a threshold, controlling the virtual viewpoint to move from the target position to the viewpoint position of the camera with the highest priority according to a preset camera priority; if the distance does not exceed the threshold, executing the step of transitioning the virtual viewpoint from the target position to the predetermined position.

[0220] Optionally, the processor may also execute program codes of the following steps: after detecting an interactive operation on the operation interface, popping up a prompt message indicating at least one selection control; and selecting a display resolution of the reconstructed image by triggering any one selection control.

[0221] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: detect an interactive operation on the operation interface to control the virtual viewpoint to move from a first position to a second position; control the virtual viewpoint to transition from the second position to a third position, wherein the third position is the position of the real viewpoint where the camera is deployed; read the image of the camera at the third position; process the image using a plane-based image deformation model to obtain a reconstructed image.

[0222] Optionally, the processor may further execute a program code of the following steps: selecting a real viewpoint position closest to the second position as the third position.

[0223] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: sensing the interactive operation of the operating object on the operating interface; in response to the interactive operation, displaying the displacement of the virtual viewpoint on the operating interface based on the interactive operation, wherein the displacement is from a first position to a second position; displaying the transition of the virtual viewpoint from the second position to a third position, wherein the third position is the position of the real viewpoint where the camera is deployed; displaying a reconstructed image generated by processing the target image using a plane-based image deformation model, wherein the target image is an image captured by the camera located at the third position.

[0224] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: sensing the movement operation of the operating object on the operating interface; if it is sensed that the movement operation is terminated, obtaining the stop position of the operating object on the operating interface when the movement operation is terminated; controlling the virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; using a plane-based image deformation model to process the image of the camera at the predetermined position to obtain a reconstructed image.

[0225] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: sensing the movement operation of the operating object on the operating interface; if it is sensed that the movement operation is terminated, obtaining the stop position of the operating object on the operating interface when the movement operation is terminated; controlling the virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is the position of the real viewpoint where the camera is deployed; reading the image of the camera at the predetermined position; using a plane-based image deformation model to process the image to obtain a reconstructed image.

[0226] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: during the live broadcast process, detect that a moving operation is received on the live broadcast screen; obtain the target position of the virtual viewpoint after the displacement on the live broadcast screen; control the virtual viewpoint to move from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; use the image deformation model to process the image where the virtual viewpoint is at the predetermined position to obtain a reconstructed image.

[0227] According to an embodiment of the present invention, a method for reconstructing an image is provided. By detecting an interactive operation on an operation interface, a target position of a virtual viewpoint after displacement on the operation interface is obtained; the virtual viewpoint is controlled to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; an image deformation model is used to process an image at the predetermined position of the virtual viewpoint to obtain a reconstructed image. In other words, the target position of the virtual viewpoint after displacement on the operation interface is obtained, and when the virtual viewpoint transitions from the target position to the predetermined position, the image deformation model can be used to process the image at the predetermined position of the virtual viewpoint, thereby achieving the purpose of obtaining a reconstructed image. Since the calculation process of the image deformation model is simple, the calculation speed is fast, and it is suitable for the computing resources of low-end machines, the technical problem of poor timeliness when reconstructing images from virtual viewpoints is solved, and the technical effect of improving timeliness when reconstructing images from virtual viewpoints is achieved.

[0228] It can be understood by those skilled in the art that Fig.14 The structure shown is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.14 It does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal A may also include Fig.14 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.14 Different configurations shown.

[0229] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0230] Example 5

[0231] The embodiment of the present invention further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for reconstructing an image provided in the first embodiment.

[0232] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0233] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: detecting an interactive operation occurring on an operating interface, and obtaining a target position of a virtual viewpoint after displacement on the operating interface; controlling the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; and using an image deformation model to process an image in which the virtual viewpoint is located at a predetermined position to obtain a reconstructed image.

[0234] Optionally, the computer-readable storage medium is further configured to store a program code for executing the following steps: if an interruption of the interactive operation at the target position is detected, initiating a transition operation of executing a transition of the virtual viewpoint from the target position to a predetermined position.

[0235] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: in the process of controlling the virtual viewpoint to transition from a target position to a predetermined position, using the DIBR virtual viewpoint interpolation algorithm to generate an interpolation of a transition image frame; and reconstructing the image during the transition process based on the interpolation of the transition image frame.

[0236] Optionally, the computer-readable storage medium is further configured to store program code for executing the following steps: after controlling the virtual viewpoint to transition from a target position to a predetermined position, calling a Warping image deformation algorithm and switching to using an image deformation model for image reconstruction.

[0237] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: reading a corresponding image from a camera based on a predetermined position of a virtual viewpoint; and processing the read image using an image deformation model according to the rotational degrees of freedom in a spatial coordinate system where the virtual viewpoint is located.

[0238] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: before controlling the virtual viewpoint to transition from the target position to the predetermined position, obtaining the distance between the target position where the virtual viewpoint is currently located and the predetermined position; if the distance exceeds a threshold, controlling the virtual viewpoint to move from the target position to the viewpoint position of the camera with the highest priority according to a preset camera priority; if the distance does not exceed the threshold, executing the step of transitioning the virtual viewpoint from the target position to the predetermined position.

[0239] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: after detecting an interactive operation on the operation interface, popping up a prompt message indicating at least one selection control; and selecting the display resolution of the reconstructed image by triggering any one of the selection controls.

[0240] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: detecting an interactive operation on an operation interface that controls a virtual viewpoint to move from a first position to a second position; controlling the virtual viewpoint to transition from the second position to a third position, wherein the third position is the position of a real viewpoint where a camera is deployed; reading an image from a camera at the third position; and processing the image using a plane-based image deformation model to obtain a reconstructed image.

[0241] Optionally, the computer-readable storage medium is further configured to store a program code for executing the following steps: selecting a real viewpoint position closest to the second position as the third position.

[0242] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: sensing an interactive operation of an operating object on an operating interface; in response to the interactive operation, displaying a displacement of a virtual viewpoint on the operating interface based on the interactive operation, wherein the displacement is from a first position to a second position; displaying a transition of the virtual viewpoint from the second position to a third position, wherein the third position is the position of a real viewpoint where a camera is deployed; and displaying a reconstructed image generated by processing a target image using a plane-based image deformation model, wherein the target image is an image captured by a camera located at the third position.

[0243] As an optional example, a computer-readable storage medium is configured to store program codes for executing the following steps: sensing a moving operation of an operating object on an operating interface; if it is sensed that the moving operation is terminated, obtaining a stop position of the operating object on the operating interface when the moving operation is terminated; controlling a virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is a camera position that coincides with a spatial degree of freedom position of the virtual viewpoint; and using a plane-based image deformation model to process an image of the camera at the predetermined position to obtain a reconstructed image.

[0244] As an optional example, a computer-readable storage medium is configured to store program codes for executing the following steps: sensing a moving operation of an operating object on an operating interface; if it is sensed that the moving operation is terminated, obtaining a stop position of the operating object on the operating interface when the moving operation is terminated; controlling a virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is a position of a real viewpoint where a camera is deployed; reading an image of a camera at the predetermined position; and processing the image using a plane-based image deformation model to obtain a reconstructed image.

[0245] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: during a live broadcast, detecting that a moving operation is received on the live broadcast screen; obtaining a target position of the virtual viewpoint after displacement on the live broadcast screen; controlling the virtual viewpoint to move from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the virtual viewpoint; using an image deformation model to process an image in which the virtual viewpoint is located at a predetermined position to obtain a reconstructed image.

[0246] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0247] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0248] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0249] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0250] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0251] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0252] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for reconstructing an image, comprising: Detecting an interactive operation on the operation interface, and obtaining a target position of a virtual viewpoint after displacement on the operation interface; If the distance between the target position where the virtual viewpoint is currently located and the predetermined position exceeds a threshold, the virtual viewpoint is controlled to move from the target position to the viewpoint position where the camera with the highest priority is located according to a preset camera priority, wherein the predetermined position is a camera position that coincides with the spatial degree of freedom position of the virtual viewpoint; If the distance does not exceed the threshold, controlling the virtual viewpoint to transition from the target position to the predetermined position according to a preset path; The image with the virtual viewpoint located at the predetermined position is processed using an image deformation model to obtain a reconstructed image.

2. The method according to claim 1, wherein: If it is detected that the interactive operation at the target position is interrupted, a transition operation of performing the transition of the virtual viewpoint from the target position to the predetermined position is initiated.

3. The method according to claim 1, wherein: In the process of controlling the virtual viewpoint to transition from the target position to the predetermined position according to the preset path, the method further includes: Use DIBR virtual viewpoint interpolation algorithm to generate interpolation of transition image frames; The image in the transition process is reconstructed based on the interpolation of the transition image frame.

4. The method according to claim 3, wherein: After controlling the virtual viewpoint to transition from the target position to the predetermined position according to the preset path, calling the Warping image deformation algorithm, switching to use the image deformation model to perform image reconstruction.

5. The method according to any one of claims 1 to 4, wherein: Using the image deformation model to process the image where the virtual viewpoint is located at the predetermined position includes: Based on the predetermined position of the virtual viewpoint, a corresponding image is read from the camera; The read image is processed using the image deformation model according to the rotational freedom in the space coordinate system where the virtual viewpoint is located.

6. The method according to claim 5, wherein: The spatial degree of freedom is the coordinate value of the virtual viewpoint on the spatial coordinate system, and the rotational degree of freedom is the degree of freedom of rotation around the coordinate axis of the spatial coordinate system.

7. The method according to claim 1, wherein: The method further comprises: The distance between the target position where the virtual viewpoint is currently located and the predetermined position is obtained.

8. The method according to claim 1, wherein: After detecting that an interactive operation occurs on the operation interface, the method further includes: Popping up a prompt message for indicating at least one selection control; By triggering any one of the selection controls, the display resolution of the reconstructed image is selected.

9. A method for reconstructing an image, comprising: During the live broadcast, a movement operation is detected on the live broadcast screen; Obtaining a target position of a virtual viewpoint after displacement on the live screen; If the distance between the target position where the virtual viewpoint is currently located and the predetermined position exceeds a threshold, the virtual viewpoint is controlled to move from the target position to the viewpoint position where the camera with the highest priority is located according to a preset camera priority, wherein the predetermined position is a camera position that coincides with the spatial degree of freedom position of the virtual viewpoint; If the distance does not exceed the threshold, controlling the virtual viewpoint to move from the target position to the predetermined position along a preset path; The image with the virtual viewpoint located at the predetermined position is processed using an image deformation model to obtain a reconstructed image.

10. A method for reconstructing an image, wherein: include: Detecting that an interactive operation of controlling a virtual viewpoint to move from a first position to a second position occurs on the operation interface; If the distance between the second position and the third position where the virtual viewpoint is currently located exceeds a threshold, controlling the virtual viewpoint to move from the second position to the viewpoint position of the camera with the highest priority according to a preset camera priority; If the distance does not exceed the threshold, controlling the virtual viewpoint to transition from the second position to a third position according to a preset path, wherein the third position is a position of a real viewpoint where a camera is deployed; reading an image from a camera located at the third position; The image is processed using a plane-based image deformation model to obtain a reconstructed image.

11. The method according to claim 10, wherein: The real viewpoint position closest to the second position is selected as the third position.

12. A method for reconstructing an image, wherein: include: Sensing interactive operations of the operation object on the operation interface; In response to the interactive operation, displaying a displacement of a virtual viewpoint on the operation interface based on the interactive operation, wherein the displacement is from a first position to a second position; Displaying the virtual viewpoint transitioning from the second position to a third position, wherein the third position is a position of a real viewpoint where a camera is deployed; displaying a reconstructed image generated by processing a target image using a plane-based image deformation model, wherein the target image is an image captured by a camera located at the third position; The method further comprises: if the distance between the second position and the third position where the virtual viewpoint is currently located exceeds a threshold, controlling the virtual viewpoint to move from the second position to the viewpoint position where the camera with the highest priority is located according to a preset camera priority; Displaying the virtual viewpoint transitioning from the second position to the third position includes: if the distance does not exceed the threshold, displaying the virtual viewpoint transitioning from the second position to the third position along a preset path.

13. A method for reconstructing an image, wherein: include: Sensing the movement of the operation object on the operation interface; If it is sensed that the moving operation is terminated, obtaining a stop position of the operating object on the operating interface when the moving operation is terminated; If the distance between the current stop position of the operation object and the predetermined position exceeds a threshold, the operation object is controlled to move from the stop position to the viewpoint position of the camera with the highest priority according to a preset camera priority, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the operation object; If the distance does not exceed the threshold, controlling the operating object located at the stop position to transition to the predetermined position along a preset path; The image of the camera at the predetermined position is processed using a plane-based image deformation model to obtain a reconstructed image.

14. A method for reconstructing an image, wherein: include: Sensing the movement of the operation object on the operation interface; If it is sensed that the moving operation is terminated, obtaining a stop position of the operating object on the operating interface when the moving operation is terminated; If the distance between the current stop position of the operation object and the predetermined position exceeds a threshold, the operation object is controlled to move from the stop position to the viewpoint position of the camera with the highest priority according to a preset camera priority, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the operation object; If the distance does not exceed the threshold, controlling the operation object at the stop position to transition to the predetermined position according to a preset path, wherein the predetermined position is a position of a real viewpoint where a camera is deployed; reading an image from a camera located at the predetermined position; The image is processed using a plane-based image deformation model to obtain a reconstructed image.

15. A device for reconstructing an image, wherein: include: A first acquisition module is used to detect an interactive operation on the operation interface and acquire a target position of a virtual viewpoint after displacement on the operation interface; The device is further configured to: if the distance between the target position where the virtual viewpoint is currently located and a predetermined position exceeds a threshold, control the virtual viewpoint to move from the target position to the viewpoint position where a camera with the highest priority is located according to a preset camera priority, wherein the predetermined position is a camera position that coincides with the spatial degree of freedom position of the virtual viewpoint; A first control module, configured to control the virtual viewpoint to transition from the target position to a predetermined position according to a preset path if the distance does not exceed the threshold, wherein the predetermined position is a camera position that coincides with a spatial degree of freedom position of the virtual viewpoint; The second processing module is used to process the image where the virtual viewpoint is located at the predetermined position using an image deformation model to obtain a reconstructed image.

16. A device for reconstructing an image, wherein: include: A detection module, used to detect an interactive operation of controlling a virtual viewpoint to move from a first position to a second position occurring on the operation interface; The device is further configured to: if the distance between the second position and the third position where the virtual viewpoint is currently located exceeds a threshold, control the virtual viewpoint to move from the second position to the viewpoint position where the camera with the highest priority is located according to a preset camera priority; A second control module is configured to control the virtual viewpoint to transition from the second position to a third position according to a preset path if the distance does not exceed the threshold, wherein the third position is a position of a real viewpoint where a camera is deployed; A first reading module, used for reading an image of a camera located at the third position; The second processing module is used to process the image using a plane-based image deformation model to obtain a reconstructed image.

17. A device for reconstructing an image, wherein: include: The first sensing module is used to sense the movement of the operating object on the operating interface; A second acquisition module is used for acquiring, if it is sensed that the moving operation is terminated, a stopping position of the operating object on the operation interface when the moving operation is terminated; The device is further configured to control the operation object to move from the stop position to the viewpoint position of the camera with the highest priority according to a preset camera priority if the distance between the stop position where the operation object is currently located and the predetermined position exceeds a threshold, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the operation object; A third control module is used to control the operation object at the stop position to transition to the predetermined position according to a preset path if the distance does not exceed the threshold, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the operation object; The third processing module is used to process the image of the camera at the predetermined position by using a plane-based image deformation model to obtain a reconstructed image.

18. A device for reconstructing an image, wherein: include: The second sensing module is used to sense the movement of the operating object on the operating interface; A third acquisition module is used for acquiring, if it is sensed that the moving operation is terminated, a stopping position of the operating object on the operation interface when the moving operation is terminated; The device is further configured to control the operation object to move from the stop position to the viewpoint position of the camera with the highest priority according to a preset camera priority if the distance between the stop position where the operation object is currently located and the predetermined position exceeds a threshold, wherein the predetermined position is a camera position that coincides with the spatial freedom position of the operation object; a fourth control module, configured to control the operating object at the stop position to transition to the predetermined position according to a preset path if the distance does not exceed the threshold, wherein the predetermined position is a position of a real viewpoint where a camera is deployed; A second reading module, used for reading an image of a camera located at the predetermined position; The fourth processing module is used to process the image using a plane-based image deformation model to obtain a reconstructed image.

19. A computer-readable storage medium, wherein: The computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 14.

20. A processor, wherein: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 14 when running.

21. A system for reconstructing an image, wherein: include: processor; A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: An interactive operation is detected on an operation interface, and a target position of a virtual viewpoint after displacement on the operation interface is obtained; if the distance between the target position where the virtual viewpoint is currently located and a predetermined position exceeds a threshold, the virtual viewpoint is controlled to move from the target position to the viewpoint position of a camera with the highest priority according to a preset camera priority, wherein the predetermined position is a camera position that coincides with the spatial degree of freedom position of the virtual viewpoint; if the distance does not exceed the threshold, the virtual viewpoint is controlled to transition from the target position to the predetermined position according to a preset path, wherein the predetermined position is a camera position that coincides with the spatial degree of freedom position of the virtual viewpoint; an image deformation model is used to process an image of the virtual viewpoint at the predetermined position to obtain a reconstructed image.

Images