Synthetic image method, image data transmission method, device, server and medium

Abstract

The server (SRV) uses different views (V) of the scene (S) to... i The corresponding image (IMG) i,j The target depth map (DM) is encoded to generate a stream (STR). The server (SRV) determines the target depth map (DM). i,j Depth blocks (DB) c,r i,j ), Target Depth Map (DM) i,j Depth blocks (DB) c,r i,j ) depth (dp) c,r i,j This can be achieved by comparing data included in another or the same depth map (DM). i,j DM i’,j’ Reference depth block (DB) in ) c’,r’ i,j DB c’,r’ i’,j’ ) depth (d c’,r’ i,j d c’,r’ i’,j’ ) Execute operation (op) c,r i,j The server generates a signal (SIG) that enables the device to recognize the exportable depth block (DB). c,r i,j The server sends streams (STRs) and signals (SIGs) to the device. The device (CLT) analyzes the signals to identify the derivative depth block (DB). c,r i,j ), and by using the reference depth block (DB) c’,r’ i,j DB c’,r’ i’,j’ ) depth (d c’,r’ i,j d c’,r’ i’,j’ ) Execute operation (op) c,r i,j To obtain the target depth map (DM) i,j The equipment uses a target depth map (DM). i,j ) and images obtained by decoding the stream (STR) (IMG*) i,j To synthesize new images (IMG) PV ).

Description

Technical Field

[0001] This invention relates to the general field of video communication. The invention can be applied to any type of video; however, it discovers specific but unrestricted applications, such as those provided in cloud environments, for interactive services (e.g., immersive video services and online gaming services). The invention is more specifically applied in the context of immersive video. Background Technology

[0002] In the context of immersive video, the observer can choose the viewpoint from which they want to observe the scene. The observer can change that viewpoint at any time, and the video will not stop. This is the principle of "free navigation."

[0003] Most of the viewpoints requested by the observer are not captured by the camera. These viewpoints can be synthesized using projection methods based on nearby captured view and depth information.

[0004] Depth calculation is a traditional and solved problem in computer vision. It is typically obtained through triangulation using epipolar geometry. This is theoretically straightforward, but far more complex in practice. It requires matching every point in one frame with a single point in another frame. This is possible because the camera captures the same scene. Such stereo matching is usually performed using very expensive optical flow methods.

[0005] The magnitude of an object's displacement between two cameras depends on the distance between the object and each camera. Knowing the displacement (stereo matching) allows for the calculation of the distance between the object and the cameras, i.e., its depth. This process typically includes the following steps:

[0006] (i) Image correction: To reduce the complexity of pixel matching, the epipolar lines of the two cameras are aligned horizontally. This can be achieved through linear transformations of the images, including rotation, translation, and tilting.

[0007] (ii) Stereo matching (or parallax calculation); and

[0008] (iii) Parallax is converted into depth by triangulation (polar geometry).

[0009] It should be noted that steps (i) and (iii) require both the inherent parameters of the camera (camera characteristics, such as focal length, resolution, etc.) and the non-inherent camera position (position, angle, etc.).

[0010] In the following description, the complex and expensive depth computing method described above will be referred to as the "conventional depth computing method".

[0011] Furthermore, the viewpoint requested by the observer may not be known to the server, so all captured views may be needed to synthesize the requested view.

[0012] The stream and associated depth map constitute the large amount of data to be transferred from the server to the client. On the client side, this content is decoded and then synthesized into the requested view.

[0013] This approach has two main problems: the data rate required to transmit this data is very high (bit rate), and the number of pixels the client needs to decode in a short period of time is very high (pixel rate). To partially solve this problem, solutions include not transmitting the depth map, but estimating the depth map on the client side based on the decoded texture view.

[0014] Because depth map computation is complex, a related problem with this solution is the computational complexity at the client end. This invention proposes different methods to reduce this complexity and improve the quality of depth maps. Summary of the Invention

[0015] The present invention aims to overcome at least some of the above-mentioned disadvantages.

[0016] Therefore, according to a first aspect, the present invention provides a method for synthesizing images, the method being performed by a device and comprising:

[0017] - Receive a stream from the server, which is obtained by encoding images corresponding to different views of the same scene captured by multiple cameras;

[0018] -Analyze the signal, the signal including at least one piece of information that enables the device to identify at least one derivable depth block of a target depth map corresponding to one of the images;

[0019] - Determine the operation to be used, the operation being used to derive the depth of the at least one derivable depth block based on the depth of at least one reference depth block included in a depth map corresponding to one of the images;

[0020] - Obtain the target depth map by performing at least one of the operations described above; and

[0021] - Use the target depth map and at least one image obtained by decoding the stream to synthesize a new image.

[0022] Relatedly, the present invention also relates to an apparatus comprising:

[0023] - A module for receiving a stream from a server, the stream being obtained by encoding images corresponding to different views of the same scene captured by multiple cameras;

[0024] - A module for analyzing signals, the signals including at least one piece of information that enables the device to identify at least one derivable depth block of a target depth map corresponding to one of the images;

[0025] - A module for determining an instruction to use an operation for deriving the depth of the at least one derivable depth block based on the depth of at least one reference depth block included in a depth map corresponding to one of the images;

[0026] Module, configured to obtain the target depth map by performing at least one of the operations; and

[0027] - A module for synthesizing a new image using the target depth map and at least one image obtained by decoding the stream.

[0028] In this document, the term "determination of operation" should be interpreted broadly. This means that the client can determine the derivation operation in a certain way. For example:

[0029] - The type of operation can be included in the signal;

[0030] - The servers and devices may have been previously exchanged to reach an agreement on operation;

[0031] - Operations can be predetermined and hard-coded in servers and / or devices, etc.

[0032] The device can be a client device of the server. The device can be embedded in the observer's (i.e., the user's) terminal, such as a smartphone, tablet, television, or personal computer.

[0033] The device can be embedded in a cloud network device accessible to the terminal. For example, an observer using the terminal can select a viewpoint, which is sent to the device from the user's terminal. The device can then perform a method to synthesize a new image of that viewpoint using a target depth map representing the depth of objects in the scene relative to a virtual camera defined by the viewpoint. The device can then send the new image to the terminal.

[0034] Therefore, in general, the present invention aims to simplify the calculation of depth maps by means of the device according to the invention, the depths of at least some blocks of these depth maps can be simply derived from the depths of reference depth blocks.

[0035] This invention enables improvements in the speed and / or quality of depth map computation during client-side depth map estimation, prior to synthesizing the view required by the observer.

[0036] One principle of this invention is to obtain and use information indicating that it is unnecessary to calculate the depth estimate of certain blocks on the client side according to conventional depth calculation methods, since the depth of these blocks can be more easily derived from the depth of other depth blocks.

[0037] In one embodiment, the operation for deriving the depth of the at least one derivable depth block does not use the camera's inherent or non-inherent parameters.

[0038] In one embodiment, the operation for deriving the depth of the at least one derivable depth block is a linear combination of the depths of the at least one reference depth block.

[0039] In one embodiment, the operation for deriving the depth of the at least one derivable depth block is to copy the depth of a reference depth block.

[0040] In one embodiment, the derivation of the depth of a derivable block is not a simple copy of the depth of a reference depth block, but rather a function of the depth of a specific number of blocks. For example, median, average, minimum (favorable to objects in the foreground), etc., can be used.

[0041] Therefore, in one embodiment, the operation for deriving the depth of the at least one derivable depth block includes calculating the average, median, or minimum value of a set of depths of the at least one reference depth block.

[0042] In one embodiment, the operation for deriving the depth of the at least one derivable depth block includes applying a prediction method to the at least one reference depth block.

[0043] In one embodiment, the signal includes an identifier for at least one non-essential depth block of the target depth map, and the method includes skipping the determination of the depth of the non-essential depth block before synthesizing the new image.

[0044] In practice, the image to be synthesized corresponding to the viewpoint requested by the observer does not necessarily need access to all depth information from all views of the scene. Rendering blocks of depth maps that are not needed for these views does not require estimation and is therefore unnecessary. The positions of these blocks can be obtained through a projection algorithm.

[0045] In one embodiment, a signal is received from a server.

[0046] In this embodiment, the algorithm can determine on the server side whether, for a given texture patch, the depth of the corresponding patch in the target depth map should be calculated, or whether the depth of the corresponding patch in the target depth map can be derived from the depths of one or more reference depth patches. When the server is aware of the viewpoint requested by the observer, the algorithm can also determine whether unnecessary depth patches exist.

[0047] Therefore, according to a second aspect, the present invention relates to a method executed in a server, the method comprising:

[0048] - Acquire multiple images corresponding to different views of the same scene captured by multiple cameras;

[0049] - The image is encoded to generate a stream;

[0050] - Determine at least one derivable depth block of a target depth map corresponding to one of the images, wherein the depth of the at least one derivable depth block is derived by performing an operation on the depth of at least one reference depth block included in the depth map corresponding to one of the images.

[0051] - Generate a signal that includes at least an indication of the operation; and

[0052] - The stream and the signal are sent to the device, the signal being sent according to a lossless protocol.

[0053] Relatedly, the present invention also relates to a server, comprising:

[0054] - A module for obtaining multiple images corresponding to different views of the same scene captured by multiple cameras;

[0055] An encoder for generating a stream by encoding the image;

[0056] - A module for determining at least one derivable depth block of a target depth map corresponding to one of the images, wherein the depth of the at least one derivable depth block is derived by performing an operation on the depth of at least one reference depth block included in the depth map corresponding to one of the images.

[0057] - A module for generating a signal, the signal including at least one piece of information enabling the device to identify the at least one derivable depth block; and

[0058] - A module for sending the stream and the signal to the device, wherein the signal is sent according to a lossless protocol.

[0059] According to the present invention, the signal sent from the server to the device must include the information necessary for the device to determine the derivative depth block.

[0060] In one embodiment, during the initial steps, the server indicates the size and shape of the block to the device. There are no particular restrictions on the size or shape of the block, even in the extreme case of a 1x1 block. In practice, depending on the context, a trade-off can be made between two possibilities regarding the block size:

[0061] - Smaller blocks to increase the number of derivative depth blocks, and

[0062] - Larger blocks to reduce the amount of information (signal size) sent from the server to the device.

[0063] In another embodiment, the signal also includes an identifier of the at least one reference depth block.

[0064] If the device knows the dimensions of the depth map (the number of blocks per row and column), the signal does not need to include identifiers of the derivable blocks.

[0065] If the device knows how to determine at least one reference depth block for deriving the depth of a given block and the operation to be used, then the signal does not need to include any of this information.

[0066] In one embodiment, the signal also includes an indication of the operation.

[0067] Therefore, in a particularly simple implementation, the signal includes only one bit for each depth block; for example, the bit is equal to 1 if the depth block is to be calculated, and equal to 0 if the depth block is to be derived.

[0068] In one embodiment, the server obtains a target viewpoint selected by an observer using a client device. The server determines at least one unnecessary depth block of the at least one target depth map, wherein the depth of the unnecessary depth block is not necessary for synthesizing a new image. The server includes information in the signal that enables the device to identify the at least one unnecessary depth block.

[0069] Therefore, in another particularly simple implementation, the signal includes a bit pair for each depth block, for example, the bit pair is equal to "01" if the depth block is to be computed, the bit pair is equal to "00" if the depth block is to be derived, and the bit pair is equal to "11" if the depth block is not needed.

[0070] Bits (or bit pairs) in a signal can be ordered according to a known order in the device. For example, if a depth map has C columns and R rows, the order of the bits can be as follows: bits corresponding to the depth block in row 1, column 1, ..., bits corresponding to the depth block in row 1, column C, ..., bits corresponding to the depth block in row 2, column 1, ..., bits corresponding to the depth block in row 2, column C, ..., bits corresponding to the depth block in row R, column 1, ..., bits corresponding to the depth block in row R, column C.

[0071] In another embodiment, the signal does not have bits (or bit pairs) for each block, but is explicitly for derived depth blocks (or unnecessary depth blocks).

[0072] In one embodiment, if a depth map is available to the server, the server can determine a reference depth block similar to a given depth block. For example, the server can perform motion estimation to identify the most suitable reference depth block.

[0073] In one variant, the server identifies a set of candidate reference depth blocks and selects the one from that set that is most similar to a given depth block. If motion estimation is validated by applying a difference (or proximity) criterion to the two blocks, then the two methods can be considered identical.

[0074] In practice, depth maps and textures often have relatively close or similar outlines.

[0075] Therefore, in a particular embodiment, the server determines a candidate reference depth block as a depth block corresponding to a candidate texture block, which is determined by the server through analysis of an image acquired by the camera that is not from a depth map (e.g., through motion estimation). This embodiment is particularly attractive when depth is not available on the server side.

[0076] For example, for a given derivable depth patch at a given location in the target depth map, the candidate texture patch could be:

[0077] - A texture patch in the image corresponding to the target depth map, located near a given position or at a position offset from the given position by a motion vector calculated for the texture or to derive the depth of the derivable depth patch; or

[0078] - Texture blocks in previous images of the same view that are either collocated or located at positions shifted by known motion vectors; or

[0079] - A texture block located at another position in an image of another view (e.g., with the parallax vector shifted).

[0080] Therefore, in one embodiment, the server obtains at least one reference depth block by means of the following method:

[0081] - Use a motion estimation method applied to at least one of the images; or

[0082] - Use a motion estimation method applied to at least one depth map corresponding to at least one of the images; or

[0083] - In a list of at least one candidate block determined for the at least one derivable depth block, wherein the candidate block is:

[0084] - A block in the neighborhood of the at least one derivable depth block in the target depth map; or

[0085] A block at the location of at least one derivable depth block in a depth map corresponding to another image.

[0086] Similarity criteria can be used to perform motion estimation, disparity estimation, or proximity checks on candidate blocks.

[0087] The similarity criterion can be SAD (sum of absolute differences) or SSD (sum of squared differences). The similarity criterion can be used or not used in minimizing the Lagrangian quantity that takes into account the cost of the additional information to be transmitted.

[0088] Machine learning algorithms can also be used to identify similar regions.

[0089] Alternatively, a machine learning algorithm can be used to directly select blocks to be discarded. This algorithm relies not only on similarity criteria between blocks but also uses features extracted from different blocks to find the blocks to be inferred. The algorithm is pre-trained in a supervised manner, where information indicating whether a block should be inferred is used as validation data.

[0090] In one embodiment, the server sends a residual depth block corresponding to the pixels between a given depth block and a reference depth block (found in a pre-selected candidate or by motion estimation).

[0091] Therefore, in one embodiment, the signal sent by the server also includes an identifier for at least one residual depth block, which can be added by the device to the at least one reference depth block to obtain the derivable depth block.

[0092] This signal indicates:

[0093] - Type of derivation. The derivation can be "direct," i.e., based on a reference depth block, or the derivation can be "through residuals," i.e., based on a residual reference depth block;

[0094] - At least one reference depth block used for derivation: the block can be identified explicitly or indirectly (left neighbor, top neighbor, juxtaposed, etc.).

[0095] - A view with at least one reference depth block: This view can be identified explicitly or indirectly (the left neighbor view, the view at the end of the scene, etc.).

[0096] - Frames in the view: The frame can be identified explicitly or indirectly (previous frame, current frame - n, ...).

[0097] When signals are sent from the server to the device, lossless protocols are used for transmission. Source coding (entropy coding, adaptive arithmetic coding, etc.) can be applied to reduce the amount of data to be transmitted.

[0098] Signals (or signaling) can be sent by the server in the form of standardized data streams, and their use is mandatory on the client side.

[0099] Signals / signaling can also be sent by the server in the form of Supplemental Enhancement Information (SEI) messages. Clients using this message can accelerate their processing; clients not using this message apply traditional processing.

[0100] In another embodiment, the signal is not received from the server, but calculated by the device. This embodiment can be particularly applied when the server cannot obtain a depth map, or when bandwidth between the server and the client is limited.

[0101] Therefore, in one embodiment, the method for synthesizing an image performed in the device includes:

[0102] - Decode the stream received from the server to obtain a decoded image;

[0103] - Determine at least one derivable depth block of a target depth map corresponding to a decoded image, wherein the depth of the at least one derivable depth block is derived by performing an operation on the depth of at least one reference depth block included in the depth map corresponding to a decoded image.

[0104] - Generate the signal, wherein the signal includes information capable of identifying the at least one derivable depth block; and

[0105] - A module that sends the signal to the device, the module being configured to obtain the target depth map by performing at least one of the operations.

[0106] The signals generated by the device may also include instructions for the operation and / or an identifier of the at least one reference depth block, regardless of whether the module is aware of this information.

[0107] In one embodiment, the device determines the at least one exportable depth block by parsing the stream.

[0108] For example, blocks encoded in skip or merge modes indicate that, within the same view, the same block changes little or no when moving from one image to another. Conversely, the use of intra-frame blocks strongly indicates the opposite phenomenon.

[0109] Therefore, in one embodiment, the device may consider a depth block corresponding to a texture block encoded in a skip mode or a merge mode as a derivable block.

[0110] In one embodiment, the device determines the at least one derivable depth block by analyzing the decoded image. The decoding step is performed after stream parsing to obtain the decoded pixels. Once these pixels are decoded, the same but different criteria described previously can be calculated, but this time applied to the decoded pixels.

[0111] Decoded pixels typically contain coded artifacts. One variation involves reducing these artifacts using conventional algorithms before performing calculations to obtain derivative or unnecessary depth blocks.

[0112] In one embodiment, motion vectors can also be reconstructed after the resolved stream. These motion vectors can be used directly because they show movement from one image to another within the same view. Similarly, parallax vectors reflect movement from one view to another for the same image.

[0113] In one embodiment, residual texture blocks can be used. For example, if the device determines that the texture residual between a given texture block and a reference texture block is low (at the texture pixels), the device can consider these texture blocks to be well matched, and the depth of the depth block corresponding to the given texture block can be derived from the depth of the reference texture block using texture motion vectors.

[0114] Therefore, in one embodiment of the synthesis method executed in the device, at least one reference depth block is obtained by the device through the following method:

[0115] - Use a motion estimation method applied to at least one of the decoded images; or

[0116] - Use a motion estimation method applied to at least one depth map corresponding to at least one of the decoded images; or

[0117] - In a list of at least one candidate block determined for the at least one derivable depth block, wherein the candidate block is:

[0118] - A block in the neighborhood of the at least one derivable depth block in the target depth map; or

[0119] The block at the location of at least one derivable depth block in the depth map corresponding to another of the decoded images.

[0120] The present invention also relates to a computer program including instructions configured to implement the steps of the method for synthesizing the image when the computer program is executed by a computer.

[0121] The present invention also relates to a computer program including instructions configured to implement the steps of the method executed in the server when the computer program is executed by a computer.

[0122] These programs can use any programming language, and they can take the form of source code, object code, or intermediate code between source code and object code, such as partially compiled code, or any other desired form.

[0123] The present invention also relates to a readable medium comprising at least one of the computer programs in this set of computer programs.

[0124] The recording medium can be any entity or device capable of storing programs. For example, it can include storage devices such as ROMs, like CD ROMs or microelectronic circuit ROMs, or it can include magnetic recording devices such as hard disks.

[0125] On the other hand, the recording medium can be a transmissible medium, such as electrical or optical signals, which can be carried by cables or optical fibers, radio waves, or other means. The program according to the invention can be downloaded, in particular, over Internet-type networks.

[0126] Alternatively, the recording medium may be an integrated circuit including a program, the integrated circuit being adapted to perform the method in question or the integrated circuit being used for the execution of the method in question. Attached Figure Description

[0127] Other features and advantages of the invention will become apparent from the following description given with reference to the accompanying drawings, which illustrate exemplary embodiments of the invention and are not limiting in any way. In the drawings:

[0128] Figure 1 The image and its associated depth map are shown.

[0129] Figure 2 A server and device according to a first embodiment of the present invention are shown.

[0130] Figure 3A flowchart illustrating the main steps of the method executed by the server and the main steps of the method executed by the device in the first embodiment of the present invention is shown.

[0131] Figure 4a illustrates a signal according to a first embodiment of the present invention.

[0132] Figure 4b illustrates a signal according to a second embodiment of the present invention.

[0133] Figure 5 Different possible operations are shown for deriving the depth of a block of depth map from the depth of at least one reference depth block.

[0134] Figure 6 A server and device according to a second embodiment of the present invention are shown.

[0135] Figure 7 A flowchart illustrating the main steps of the method executed by the server and the main steps of the method executed by the device in the second embodiment of the present invention is shown.

[0136] Figure 8 The hardware architecture of a server according to a specific embodiment of the present invention is shown.

[0137] Figure 9 The hardware architecture of a client device according to a specific embodiment of the present invention is shown. Specific Implementation

[0139] In this application embodiment, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject. First Embodiment

[0140] Figure 1 The IMG image and its associated DM depth map are shown. This demonstrates the outline of the IMG image that can be found on the DM depth map. We define the following symbols:

[0141] -IB c,r : The image block (or texture block) located in column c and row r; and

[0142] -DB c,r : The depth block located in column c and row r.

[0143] Figure 2 A server SRV and a device CLT according to a first embodiment of the present invention are illustrated. In this embodiment, the device CLT is embedded in the terminal of a user or observer. Alternatively, the device CLT can be implemented in a cloud-based network.

[0144] Figure 3A flowchart illustrating the main steps of the method executed by the server and the main steps of the method executed by the device in the first embodiment of the present invention is shown.

[0145] This first embodiment is in the context where the server SRV can generate a depth map but does not send the depth map to the device CLT.

[0146] The server SRV connects to the camera CAM via the image acquisition module IAM. i CAM camera i Configured to acquire the image IMG of scene S i,j Where i is the view V i The index is j, where j is the index of the frame.

[0147] In this first embodiment, each camera CAM is considered to be i Associated with a device not shown, and a camera CAM i Configure as an IMG for each image i,j Generate Depth Map (DM) i,j The server acquires these images during step S2.

[0148] The server SRV includes an encoder COD, which is configured to output data based on the image IMG. i,j Generate (step S4) video stream STR.

[0149] The server SRV includes a signal generation module SSGM, which is configured to generate (step S10) a signal SIG. The signal SIG includes features that enable the device CLT to recognize features included in the depth map DM. i,j Exportable depth block DB c,r i,j Information.

[0150] Figure 4a illustrates the signal SIG according to a first embodiment of the present invention. In this first embodiment, it is assumed that the device CLT is aware of:

[0151] - The dimensions of the depth map (number of blocks per row and column); and

[0152] – Used to derive an exportable block DB c,r i,j depth dp c,r i,j operation op c,r i,j .

[0153] For example, the operation could simply include the depth of the block to the left of the current depth block.

[0154] op c,r i,j :dpc,r i,j =dp c-1,r i,j

[0155] In fact, in this first embodiment, the signal SIG includes only one bit for each depth block. For example, if the depth block is to be calculated using conventional depth calculation methods, the bit is equal to 1, and if the depth block is to be derived, the bit is equal to 0.

[0156] Figure 4b illustrates the signal SIG according to a second embodiment of the present invention. The signal SIG is for two depth maps DM. i,j DM k,l As shown. In this second embodiment, for each depth block DB of each depth map. c,r i,j The SIG signal includes:

[0157] -Block Identifier DB c,r i,j ;

[0158] - Indicates the depth dp of the block c,r i,j Should it be one of the following attributes:

[0159] (i) is calculated using conventional depth calculation methods (TBC); or

[0160] (ii) is derived from (d);

[0161] (iii) When the depth of the depth block is considered unnecessary for synthesizing a new image, simply ignore or skip it (u); and

[0162] - For derivable blocks, derive the depth dp. c,r i,j Required operation op c,r i,j Instructions.

[0163] In another embodiment (not shown), the signal SIG may also include a reference depth block DB. c’,r’ i,j DB c’,r’ i’,j’ Identifier or residual depth block DBR c’,r’ i,j The identifiers are used to deduce the depth dp. c,r i,j Or a block of given depth.

[0164] Figure 5 The depth map DM is derived from the depth of at least one reference depth block shown by the dashed lines. i,jDM block c ,r i,j Different depths may allow for different operations.

[0165] For example, the reference depth block can be:

[0166] -Another depth map DM i’,j’ block DB c’,r’ i’,j’ ( Figure 5 a); or

[0167] - Same depth map DM i,j block DB c’,r’ i,j ( Figure 5 b).

[0168] The server can use different methods to identify (step S8) at least one reference depth block. For example, the server SRV obtains these reference depth blocks in the following way:

[0169] -Use applied to the image IMG i’,j’ Motion estimation methods; or

[0170] - Used for depth map DM i’,j’ Motion estimation methods;

[0171] - For a given derivative depth block DM c,r i,j The identified candidate block DM c’,r’ i’,j’ In the list, the candidate blocks are:

[0172] - Same depth map DM i,j The blocks in the neighborhood (c', r') of a given derivable depth block; or

[0173] -In conjunction with another image IMG i’,j’ (Usually the previous frame in the same view) corresponding to the depth map DM i’,j’ Given a block at position (c, r) where a depth block can be derived.

[0174] Reference block DB c’,r’ i’,j’ DB c’,r’ i,j The coordinates c' and r' can be close to the current depth block DB. c,r i,j The coordinates. If a more distant reference block corresponds better to the current depth block, a more distant reference block can also be selected.

[0175] exist Figure 2In one embodiment, the signal generation module SSGM includes a module SMDDB for determining derivable depth blocks and a module SMDUB for determining unnecessary depth blocks.

[0176] The server SRV includes a communication module COM for sending (step S12) the video stream STR and the signal SIG to the device CLT.

[0177] The SIG signal is sent according to a lossless protocol. The SIG signal can be sent as an SEI message.

[0178] The device CLT receives the stream STR and signal SIG via the communication module COM (step C2). The device CLT then decodes (step C4) the stream STR and retrieves the decoded image IMG*. i,j The decoder.

[0179] The device CLT includes a module DMGM, which is configured to generate a target depth map DM based on the received signal SIG (step C10). i,j .

[0180] Target Depth Map (DM) i,j This can be used to synthesize a new image IMG based on the viewpoint pv selected by the observer (step C12). pv In the embodiments described herein, the device CLT includes an I / O module through which a user can select the viewpoint.

[0181] The DMGM module is configured to analyze (step C6) the SIG signal and identify the target depth map DM. i,j The exportable depth block in the.

[0182] For example, if the client receives the signal SIG in Figure 4a, the client determines that each bit equal to 1 corresponds to a depth block that must be calculated using conventional depth calculation methods, and each bit equal to 0 corresponds to a derivable depth block.

[0183] In another example, if the client receives the SIG signal from Figure 4b, then for each depth block DB of the depth map... c,r i,j The client determines the block depth dp c,r i,j Should it be:

[0184] (i) is calculated using the conventional depth calculation method (TBC); or

[0185] (ii) is derived from (d);

[0186] (iii) When the depth of the depth block is considered to be unnecessary for synthesizing a new image, simply ignore or skip it (u).

[0187] For each exportable block, module DMGM also determines (step C8) how to derive the depth dp. c,r i,j Required operation op c,r i,j .

[0188] In one embodiment, the server SRV and the client initially agree on the operations to be performed to derive the depth of a given depth block based on the depth of one or more depth blocks.

[0189] Module DMGM performs at least one of the aforementioned operations. c,r i,j To obtain the target depth map DM i,j .

[0190] In this embodiment, the device CLT includes a module ISM for using the target depth map DM. i,j and at least one image IMG* obtained by decoding the stream STR i,j To synthesize new images (IMG) PV .

[0191] In the detailed example of this first embodiment, the server SRV sends an SEI message indicating an exportable block.

[0192] On the server side, an immersive video containing 30 views was captured by a multi-camera setup around the stadium. No depth maps were acquired or computed. Each block in each image within each view was examined sequentially; each block was a square with a size of 32x32. The SAD criterion was applied between the current block and its juxtaposed block (in the same view, the previous image). If the SAD value was below a predefined threshold, the block was considered to have no significant texture change. Then, it was assumed that the block's depth had no significant change. For a set of three consecutive blocks, the following string was created: 0210000101.

[0193] -0: Square block indicator

[0194] -2: Indicates a 32x32 size (1 represents 16x16, etc.)

[0195] -1: Indicator of the block to be derived

[0196] -0: Direct derivation (1 indicates residual derivation)

[0197] -0: Use juxtaposed blocks (1 for top, 2 for left, etc.).

[0198] -0: Block indicator to be computed

[0199] -0: Block indicator to be computed

[0200] -1: Indicator of the block to be derived

[0201] -0: Direct inheritance (1 indicates residual)

[0202] -1: Use top block

[0203] The string is losslessly encoded using Huffman coding, which converts it into a sequence of 0s and 1s. This string is then associated with the strings in other blocks and transmitted to the client via SEI messages.

[0204] At the client's CLT, the SEI message is read and decoded (reverse Huffman coding). The string 0210000101 is found. The client understands that for the first 32x32 square block, the depth should not be calculated, but must be derived directly from the juxtaposed blocks. Then, the client understands that for the next two blocks, the depth must be calculated, and the depth of the next block must be derived from the depth of the top block.

[0205] Second Embodiment

[0206] Figure 6 A server SRV and a device CLT according to a second embodiment of the present invention are shown.

[0207] Figure 7 A flowchart illustrating the main steps of the method performed by the device in this second embodiment is shown.

[0208] In this embodiment, the server SRV does not generate a depth map. The server SRV only sends the video stream STR to the device CLT (which receives the video stream STR in step C2).

[0209] In this embodiment, the client includes a module DSGM to generate (step C60) a signal SIG, which includes information capable of identifying a derivable depth block in the target depth map. In this embodiment, a derivable depth block refers to a block whose depth can be determined by analyzing information included in the decoded image IMG*. i,j IMG* i’,j’ Corresponding depth map DM i,j The depth block is derived by performing a depth operation on at least one reference depth block in the depth block.

[0210] In one embodiment, the DSGM module is configured to send a signal (step C70) to the DMGM module, and the DMGM module is configured to generate a target depth map DM based on the signal SIG. i,j The DMGM module can be similar to the one described above. Figure 2 The module with the same name as described.

[0211] In one embodiment, the derivable depth block can be determined by the submodule CMDDB (step C50), which is configured to parse the video stream STR and / or analyze the decoded image IMG*. i,j .

[0212] For example, the CMDDB submodule can parse the stream STR to identify texture blocks that have been encoded in Skip, Merge, or Intra mode, and can determine that the depth block corresponding to the texture block encoded in Skip or Merge mode is an exportable block.

[0213] In one embodiment, the DSGM module is configured to generate the SIG signal and obtain the reference depth block in the following manner:

[0214] -Use the IMG* applied to at least one of the decoded images. i’,j’ Motion estimation methods; or

[0215] - Using at least one depth map DM applied to at least one of the decoded images i’,j’ Motion estimation methods; or

[0216] - For a given derivative depth block DM c,r i,j A list of at least one candidate block, wherein the candidate block is:

[0217] - Blocks in the neighborhood of a given derivable depth block as described in the target depth map;

[0218] In comparison with another decoded image IMG i’,j’ Corresponding depth map DM i’,j’ The block at the location of at least one derivable depth block as described in the text.

[0219] In one embodiment, the module DSGM configured to generate the signal SIG further includes a submodule CMDUB, which is configured to identify its depth for synthesizing a new image IMG for a given viewpoint pv. pv Not a necessary depth block.

[0220] In a detailed example of this second embodiment, the client CLT reads syntax elements from the video stream to identify exportable depth blocks.

[0221] In this second example, there is no special processing on the server side. All views are encoded in a conventional way using a 2D or 3D encoder, and the video stream is transmitted to the client.

[0222] On the client side, for a given texture block, the pixel residuals are examined. It turns out that the average magnitude of the pixel residuals is very low, indicating that the encoder perfectly predicted the block. The stream resolution also indicates that the block was encoded in an inter-frame mode with a disparity vector of (+5, -10). Therefore, the client infers that, within the same frame, depth pixels located in another view can be reused in a block that has been moved by (+5, -10) pixels.

[0223] like Figure 8 As shown, the server SRV specifically includes processor 1. SRV Random Access Memory 3 SRV Read-only memory 2 SRV Non-volatile flash memory 4 SRV .

[0224] Read-only memory 2 SRV The recording medium constituting the present invention can be made by processor 1 SRV Reading Read-Only Memory 2 SRV and in read-only memory 2 SRV The above records the computer program PG according to the present invention. SRV .

[0225] Computer program PG SRV Define the functional (software) modules of server 100.

[0226] exist Figure 2 In some embodiments, these functional modules include:

[0227] - Module IAM is used to acquire multiple images (IMG). i,j ;

[0228] - Encoder COD, used to generate the video stream STR;

[0229] - Module SSGM, used to determine exportable depth blocks and generate the SIG signal; and

[0230] - Communication module, used to send video stream STR and signal SIG to the client CLT.

[0231] like Figure 9 As shown, the device CLT specifically includes a processor 1 CLT Random Access Memory 3 CLT Read-only memory 2 CLT Non-volatile flash memory 4 CLT .

[0232] Read-only memory 2 CLT The recording medium constituting the present invention can be made by processor 1 CLT Read, and in read-only memory 2 CLTThe above records the computer program PG according to the present invention. CLT .

[0233] Computer program PG CLT Define the functional (software) modules of the device CLT.

[0234] exist Figure 2 In some embodiments, these functional modules include:

[0235] -Decoder DEC;

[0236] - Module DMGM, configured to analyze the SIG signal received from server SRV and obtain a target depth map; and

[0237] - Module ISM is used to synthesize new images.

[0238] exist Figure 6 In some embodiments, these functional modules include:

[0239] -Decoder DEC;

[0240] - Module DSGM is used to determine the exportable depth block and to generate the signal SIG;

[0241] - Module DMGM, configured to analyze the signal SIG received from module DSGM and obtain a target depth map; and

[0242] - Module ISM is used to synthesize new images.

Claims

1. A composite image (IMG) PV The method, characterized in that, The method is performed by a device (CLT), and the method includes: - Receive (C2) stream (STR) from server (SRV), the stream being transmitted via connections to multiple cameras (CAM). i Different views (V) of the same scene (S) obtained i The corresponding image (IMG) i,j It is obtained by encoding; - Analyze the (C6) signal (SIG), the signal (SIG) including enabling the device to recognize an image (IMG) i,j The corresponding target depth map (DM) i,j At least one exportable depth block (DB) c,r i,j The signal includes at least one piece of information, and further includes bit pairs for each of the derivative depth blocks, the values ​​of different bit pairs indicating different operations for the derivative depth blocks; - Based on the value of the bit pair, determine (C8) the operation (op) to be used for each of the derived depth blocks. c,r i,j The operation includes at least one of the following: calculating the derived depth block, deriving the derived depth block, ignoring the derived depth block, and, in the case where the value of the bit pair is used to indicate the derivation of the derived depth block, the operation (op) c ,r i,j ) used according to in a said image (IMG) i,j IMG i’,j’ The corresponding depth map (DM) i,j DM i’,j’ At least one reference depth block (DB) included in ) c’,r’ i,j DB c’,r’ i’,j’ ) depth (d c’,r’ i,j d c’,r’ i’,j’ ), to derive the exportable depth block (DB) c,r i,j ) depth (dp) c,r i,j ); - By performing at least one of the aforementioned operations (op) c,r i,j To obtain the target depth map (DM) described in (C10) i,j );as well as - Using the target depth map (DM) i,j and at least one image (IMG*) obtained by decoding the stream (STR). i,j To synthesize (C12) a new image (IMG) PV ).

2. The method for synthesizing images according to claim 1, characterized in that, The signal (SIG) is received from the server (SRV).

3. The method for synthesizing images according to claim 2, characterized in that, The signal (SIG) is transmitted in the supplemental enhancement information message.

4. The method for synthesizing images according to claim 1, characterized in that, The method includes: - Decode the stream (STR) (C4) to obtain a decoded image (IMG*). i,j ); - Determine with one of the decoded images (IMG*) i,j The corresponding target depth map (DM) i,j The at least one derivative depth block (DB) of the above. c,r i,j ), whereby, by comparing with one of the decoded images (IMG* i,j IMG* i’,j’ The corresponding depth map (DM) i,j DM i’,j’ At least one reference depth block (DB) included in ) c’,r’ i,j DB c’,r’ i’,j’ ) depth (d c ’,r’ i,j d c’,r’ i’,j’ ) Execute operation (op) c,r i,j To derive the at least one exportable depth block (DB) c,r i,j ) depth (dp) c,r i,j ); - Generate (C50) the signal (SIG), wherein the signal includes the ability to identify the at least one derivative depth block (DB). c,r i,j ) a piece of information; and - The module (DMGM) of the device sends the signal (SIG) (C54) to obtain the target depth map by performing at least one of the operations.

5. The method for synthesizing images according to claim 4, characterized in that, The at least one exportable depth block (DB) c,r i,j () is obtained by parsing the stream (STR).

6. The method for synthesizing images according to claim 4 or 5, characterized in that, The at least one exportable depth block (DB) c,r i,j ) is achieved by analyzing the decoded image (IMG* i,j It is certain.

7. The method for synthesizing images according to claim 4 or 5, characterized in that, The at least one reference depth block (DB) c’,r’ i’,j’ The CLT is obtained by the device as follows: - Use the method applied to at least one of the decoded images (IMG*) i’,j’ Motion estimation methods; or - Use the application applied to at least one of the decoded images (IMG*) i’,j’ At least one corresponding depth map (DM) i’,j’ Motion estimation methods; - For the at least one derivative depth block (DM) c,r i,j At least one candidate block (DM) determined c’,r’ i’,j’ In the list, the candidate block is: -In the target depth map (DM) i,j The at least one derivative depth block (DM) in c,r i,j The blocks in the neighborhood of ) or - In conjunction with another decoded image (IMG) i’,j’ The corresponding depth map (DM) i’,j’ At least one derivative depth block (DM) as described in the document c,r i,j The block at position (c, r).

8. The method according to claim 1, characterized in that, The signal also indicates the corresponding operation for deriving each of the derivable depth blocks.

9. The method according to claim 1, characterized in that, The signal also includes a corresponding identifier for deriving at least one of the reference depth blocks for each of the derivable depth blocks.

10. The method according to claim 1, characterized in that, The operation used to derive the depth of the at least one derivable depth block does not use the camera's inherent or non-inherent parameters.

11. The method according to claim 1, characterized in that, The operation for deriving the depth of the at least one derivable depth block is a linear combination of the depths of the at least one reference depth block.

12. A method for transmitting image data, characterized in that, The method is executed by the server, and the method includes: - Obtain (S2) and by multiple cameras (CAM) i Different views (V) of the same scene (S) obtained i Multiple corresponding images (IMG) i,j ); - By analyzing the image (IMG) i,j Encode the (S4) stream (STR) to generate the stream. - Determine (S8) with one of the images (IMG) i,j The corresponding target depth map (DM) i,j At least one exportable depth block (DB) c,r i,j ), wherein, by means of in a said image (IMG) i,j IMG i’,j’ The corresponding depth map (DM) i,j DM i’,j’ At least one reference depth block (DB) included in ) c’,r’ i,j DB c’,r’ i’,j’ ) depth (d c’,r’ i,j d c’,r’ i’,j’ ) Execute operation (op) c,r i,j To derive the at least one exportable depth block (DB) c,r i,j ) depth (dp) c,r i,j ); - Generate (S10) signal (SIG), the signal including enabling the device (CLT) to recognize the at least one exportable depth block (DB). c,r i,j The signal includes at least one piece of information, and further includes bit pairs for each of the derived depth blocks, the values ​​of different bit pairs indicating different operations for the derived depth blocks, the operations including at least one of: calculating the derived depth block, deriving the derived depth block, and ignoring the derived depth block; The stream (STR) and the signal (SIG) are sent to the device (CLT) in S12, the signal (SIG) being sent according to a lossless protocol.

13. The method according to claim 12, characterized in that, The at least one reference depth block (DB) c’,r’ i’,j’ The server (SRV) obtains this information through the following method: - Use applied to at least one of the images (IMG) i’,j’ Motion estimation methods; or - Used with at least one of the images (IMG) i’,j’ At least one corresponding depth map (DM) i’,j’ Motion estimation methods; or - For the at least one derivative depth block (DM) c,r i,j At least one candidate block (DM) determined c’,r’ i’,j’ In the list, the candidate block is: -In the target depth map (DM) i,j The at least one derivative depth block (DM) in c,r i,j The blocks in the neighborhood of ) or - In conjunction with another image (IMG) i’,j’ The corresponding depth map (DM) i’,j’ At least one derivative depth block (DM) as described in the document c,r i,j The block at position (c, r).

14. The method according to claim 12, characterized in that, The method includes: - Obtain the target viewpoint (pv) selected by the observer; - Determine at least one of the target depth maps (DM) i,j At least one non-essential depth block, wherein the depth of the non-essential depth block is relevant to the synthesis of the new image (IMG). pv ) is not necessary; and - The signal includes elements that enable the device to identify the at least one unnecessary depth block (DB). c,r i,j (This is a piece of information.) 15. The method according to any one of claims 12 to 14, characterized in that, The signal (SIG) also includes the operation (op). c,r i,j (instructions).

16. The method according to any one of claims 12 to 14, characterized in that, The signal (SIG) also includes the at least one reference depth block (DB). c’,r’ i,j DB c’,r’ i’,j’ The identifier of ).

17. The method according to any one of claims 12 to 14, characterized in that, Used to derive the at least one exportable depth block (DB) c,r i,j The depth (dp) c,r i,j The operation (op) c,r i,j ) Not using the aforementioned camera (CAM) i (Inherent parameters or non-inherent parameters) 18. The method according to any one of claims 12 to 14, characterized in that, Used to derive the at least one exportable depth block (DB) c,r i,j The depth (dp) c,r i,j The operation (op) c,r i,j ) is the at least one reference depth block (DB) c’,r’ i,j DB c’,r’ i’,j’ The depth (d) c’,r’ i,j d c’,r’ i’,j’ A linear combination of ).

19. The method according to claim 18, characterized in that, Used to derive the at least one exportable depth block (DB) c,r i,j The depth (dp) c,r i,j The operation (op) c,r i,j ) is to copy a reference depth block (DB) c’,r’ i,j DB c’,r’ i’,j’ ) depth (d c’,r’ i,j ).

20. The method according to any one of claims 12 to 14, characterized in that, Used to derive the at least one exportable depth block (DB) c,r i,j The depth (dp) c,r i,j The operation (op) c,r i,j This includes calculating the at least one reference depth block (DB). c’,r’ i’,j’ (a set of average, median, or minimum values ​​of the depths).

21. The method according to any one of claims 12 to 14, characterized in that, Used to derive the at least one exportable depth block (DB) c,r i,j The depth (dp) c,r i,j The operation (op) c,r i,j This includes applying the prediction method to the at least one reference depth block (DB). c’,r’ i’,j’ ).

22. The method according to any one of claims 12 to 14, characterized in that, The signal (SIG) includes at least one of the target depth maps (DM). i,j The method includes identifying at least one non-essential depth block of a synthetic new image (IMG). pv Prior to this, the determination of the depth of the unnecessary depth block is skipped.

23. The method according to any one of claims 12 to 14, characterized in that, The signal (SIG) includes at least one residual depth block (RDB). c,r i,j The identifier of the at least one residual depth block, which is added by the device (CLT) to the at least one reference depth block (DB). c’,r’ i’,j’ ) to obtain the exportable depth block (DB) c,r i,j ).

24. A device, characterized in that, include: - Module (COM) for receiving streams (STR) from a server (SRV), the streams (STR) being transmitted via connections to multiple cameras (CAM). i Different views (V) of the same scene (S) obtained i The corresponding image (IMG) i,j It is obtained by encoding; - Module (DMGM) for analyzing signals (SIG), the signals (SIG) including those enabling the device to identify an image (IMG) i,j The corresponding target depth map (DM) i,j At least one exportable depth block (DB) c,r i,j The signal includes at least one piece of information, and further includes bit pairs for each of the derivative depth blocks, the values ​​of different bit pairs indicating different operations for the derivative depth blocks; - Module (DMGM) is used to determine the operation to be used for each of the derived depth blocks based on the values ​​of the bit pairs. c,r i,j The operation includes at least one of the following: calculating the derived depth block, deriving the derived depth block, ignoring the derived depth block, and, in the case where the value of the bit pair is used to indicate the derivation of the derived depth block, the operation (op) c,r i,j ) used according to in a said image (IMG) i,j IMG i’,j’ The corresponding depth map (DM) i,j DM i’,j’ At least one reference depth block (DB) included in ) c’,r’ i,j DB c’,r’ i’,j’ ) depth (d c’,r’ i,j d c’,r’ i’,j’ To derive the exportable depth block (DB) c,r i,j ) depth (dp) c,r i,j ); - Module (DMGM) for performing the operation (op) for each of the said exportable depth blocks c,r i,j To obtain the target depth map (DM) i,j );as well as - Module (ISM) for using the target depth map (DM) i,j and at least one image (IMG*) obtained by decoding the stream (STR). i,j To synthesize new images (IMG) PV ).

25. A server (SRV), characterized in that, include: - Module (IAM) for acquiring data from multiple cameras (CAM) i Different views (V) of the same scene (S) obtained i Multiple corresponding images (IMG) i,j ); - Encoder (COD) for encoding the image (IMG) i,j Encode the data to generate a stream (STR); - Module (SSGM) is used to determine the relationship between the image (IMG) and the image. i,j The corresponding target depth map (DM) i,j At least one exportable depth block (DB) c,r i,j ), wherein, by comparing with one of the images (IMG) i,j IMG i’,j’ The corresponding depth map (DM) i,j DM i’,j’ At least one reference depth block (DB) included in ) c’,r’ i,j DB c’,r’ i’,j’ ) depth (d c’,r’ i,j d c ’,r’ i’,j’ ) Execute operation (op) c,r i,j To derive the at least one exportable depth block (DB) c,r i,j ) depth (dp) c ,r i,j ); - Module (SSGM) for generating a signal (SIG) that enables the device to recognize the at least one exportable depth block (DB). c,r i,j The signal includes at least one piece of information, and further includes bit pairs for each of the derived depth blocks, the values ​​of different bit pairs indicating different operations for the derived depth blocks, the operations including at least one of: calculating the derived depth block, deriving the derived depth block, and ignoring the derived depth block; as well as - Module (COM) for sending the stream (STR) and the signal (SIG) to the device (CLT), the signal (SIG) being sent according to a lossless protocol.

26. A device, characterized in that, include: A processor and a memory for storing instructions, wherein the processor is configured to execute instructions stored in the memory to implement the method according to any one of claims 1 to 11.

27. A server, characterized in that, include: A processor and a memory for storing instructions, wherein the processor is configured to execute instructions stored in the memory to implement the method according to any one of claims 12 to 23.

28. A computer-readable storage medium, characterized in that, The method includes a computer program comprising instructions that, when executed by a device, implement the method according to any one of claims 1 to 23.

Images