Image and video coding with multi-sensor collaboration and frequency adaptive processing

By leveraging multi-sensor collaboration and frequency adaptive processing, and utilizing the complementarity of color and depth images, a machine learning model is used to reconstruct high-quality depth images. This solves the problems of low compression efficiency and block artifacts in existing technologies, achieving more efficient video compression and quality preservation.

CN118901079APending Publication Date: 2024-11-05GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280093869.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-03-25
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

Existing video coding technologies suffer from low compression efficiency and are unable to effectively handle block artifacts under a large number of parameters when processing multi-sensor data, especially depth maps and color maps. Furthermore, traditional methods struggle to balance video quality and bit rate.

Method used

By employing multi-sensor collaborative technology, frequency adaptive processing and guided reconstruction are used to leverage the complementarity of color and depth images. Machine learning models are used to process different frequency components separately to reconstruct high-quality depth images and reduce the number of bits in the bitstream.

Benefits of technology

It significantly improves video compression efficiency, reduces block artifacts, maintains or improves video quality, and saves 50% to 90% of the bit rate at the same bit rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118901079A_ABST
    Figure CN118901079A_ABST
Patent Text Reader

Abstract

In certain aspects, a method is disclosed. The processor receives a first image and a second image of the same scene. The first image and the second image are acquired by two different sensors. The processor estimates first information at the first frequency of the first image based at least in part on a first component at the first frequency of the second image. The processor estimates second information at the second frequency of the first image based at least in part on a second component at the second frequency of the second image. The processor reconstructs the first image based on the first information and the second information.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Embodiments of the present application relate to image and video coding.

[0002] In recent years, the storage and transmission of video data have become more and more common, and massive amounts of video data are constantly being generated. Therefore, the effective compression of video data is becoming more and more important. Video coding technology has made a meaningful contribution to the compression of video data. The earliest research on video compression can be traced back to 1929, when inter-frame compression was first proposed. After years of research and development, mature video compression codec standards have gradually been formed, such as audio video interleave (AVI), moving picture expert group (MPEG), advanced video coding (H.264 / AVC), and high-efficiency video coding (H.265 / HEVC). The latest versatile video coding (H.266 / VVC) standard was officially released in 2020, representing the most advanced video coding technology at present. Although the structure of VVC is still based on the traditional hybrid video coding mode, its compression rate has been increased by about one-fold. Summary of the invention

[0003] According to one aspect of the present application, a method is disclosed. A processor receives a first image and a second image of the same scene. The first image and the second image are acquired by two different sensors. The processor estimates first information at a first frequency of the first image based at least in part on a first component at a first frequency of the second image. The processor estimates second information at a second frequency of the first image based at least in part on a second component at a second frequency of the second image. The processor reconstructs the first image based on the first information and the second information.

[0004] According to another aspect of the present application, a system includes a memory for storing instructions and a processor coupled to the memory. The processor is used to receive a first image and a second image of the same scene when executing the instructions. The first image and the second image are acquired by two different sensors. The processor is also used to estimate the first information at the first frequency of the first image based at least in part on the first component at the first frequency of the second image when executing the instructions, and to estimate the second information at the second frequency of the first image based at least in part on the second component at the second frequency of the second image. The processor is also used to reconstruct the first image based on the first information and the second information when executing the instructions.

[0005] According to another aspect of the present application, a method is disclosed. A first sensor acquires a first image of a scene. A second sensor acquires a second image of the scene. A first processor estimates first information at a first frequency of the first image based at least in part on a first component at a first frequency of the second image. The first processor estimates second information at a second frequency of the first image based at least in part on a second component at a second frequency of the second image. The first processor reconstructs the first image based on the first information and the second information.

[0006] According to another aspect of the present application, a system includes an encoding subsystem and a decoding subsystem. The encoding subsystem includes a first sensor, a second sensor, a first memory for storing instructions, and a first processor coupled to the first memory and the first sensor and the second sensor. The first sensor is used to acquire a first image of a scene. The second sensor is used to acquire a second image of the scene. The decoding subsystem includes a second memory for storing instructions and a second processor coupled to the second memory. The second processor is used to estimate the first information at the first frequency of the first image based at least in part on the first component at the first frequency of the second image when executing the instructions, and to estimate the second information at the second frequency of the first image based at least in part on the second component at the second frequency of the second image. The second processor is also used to reconstruct the first image based on the first information and the second information when executing the instructions.

[0007] These illustrative embodiments are mentioned not to limit or define the present application, but to provide examples to aid understanding of the present application. Other embodiments are described in the detailed description, and further description is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings, which are incorporated herein and constitute a part of the specification, illustrate embodiments of the present application and, together with the description, further explain the principles of the present application and enable technicians in the relevant field to make and use the present application.

[0009] Figure 1 A block diagram of an exemplary encoding system according to some embodiments of the present application is shown.

[0010] Figure 2 A block diagram of an exemplary decoding system according to some embodiments of the present application is shown.

[0011] Figure 3 A detailed block diagram of an exemplary encoding and decoding system according to some embodiments of the present application is shown.

[0012] Figure 4 A detailed block diagram of an exemplary compression module according to some embodiments of the present application is shown.

[0013] Figure 5A detailed block diagram of an exemplary guided reconstruction module according to some embodiments of the present application is shown.

[0014] Fig. 6A A detailed block diagram of an exemplary guided reconstruction module using a machine learning model according to some embodiments of the present application is shown.

[0015] Figure 6B A detailed block diagram of another exemplary guided reconstruction module using a machine learning model according to some embodiments of the present application is shown.

[0016] Figure 6C A detailed block diagram of yet another exemplary guided reconstruction module using a machine learning model according to some embodiments of the present application is shown.

[0017] Figure 7 The anchor image and the anchor image under different quantization parameters (QP) according to some embodiments of the present application are shown. Figures 6A to 6C Visual comparison between various output depth images of the guided reconstruction module in.

[0018] Figure 8 A flowchart of an exemplary encoding method according to some embodiments of the present application is shown.

[0019] Fig. 9 A flowchart of an exemplary decoding method according to some embodiments of the present application is shown.

[0020] Fig. 10A A detailed flowchart of an exemplary image reconstruction method according to some embodiments of the present application is shown.

[0021] Fig. 10B A detailed flowchart of another exemplary image reconstruction method according to some embodiments of the present application is shown.

[0022] Fig.11 A block diagram of an exemplary model training system according to some embodiments of the present application is shown.

[0023] Fig.12 An exemplary model training scheme according to some embodiments of the present application is shown.

[0024] The embodiments of the present application will be described in conjunction with the accompanying drawings. DETAILED DESCRIPTION

[0025] Although some configurations and arrangements are discussed, it is understood that this is for illustration only. Those skilled in the art will recognize that other configurations and arrangements may be used without departing from the spirit and scope of the present application. Those skilled in the art will appreciate that the present application may also be used in a variety of other applications.

[0026] It should be noted that the phrases "one embodiment", "embodiment", "example embodiment", "some embodiments", "certain embodiments", etc. used in this specification indicate that the embodiment may include specific features, structures, or characteristics, but not every embodiment necessarily includes the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an embodiment, whether or not it is explicitly described, those skilled in the art may implement such feature, structure, or characteristic in conjunction with other embodiments.

[0027] In general, terms may be understood at least in part based on contextual usage. For example, the term "one or more" as used herein may be used to describe any feature, structure, or characteristic in a singular sense, or may be used to describe a combination of features, structures, or characteristics in a plural sense, depending at least in part on the context. Similarly, terms such as "one," "this," or "the" may be understood to indicate singular usage or to indicate plural usage, depending at least in part on the context. In addition, the term "based on" may be understood to be not intended to express a set of exclusive factors, but to allow for the presence of additional factors that are not necessarily explicitly described, depending at least in part on the context.

[0028] Various aspects of image and video coding systems are now described with reference to various devices and methods. These devices and methods will be described in detail below and illustrated in the accompanying drawings by various modules, components, circuits, steps, operations, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, firmware, computer software, or any combination thereof. Whether these elements are implemented as hardware, firmware, or software depends on the specific application and the design constraints on the overall system.

[0029] The techniques described herein can be used in various image and video coding applications. As described herein, image and video coding includes encoding and decoding a video, a video frame, or a still image (also referred to as a picture). For ease of description, this application refers to a video, a frame, or an image; unless otherwise specified, in either case, a video, a video frame, and a still image are included.

[0030] The three-dimensional (3D) extension of HEVC (3D-HEVC) is a 3D video coding standard developed by the joint collaborative team on 3D video coding extension development (JCT-3V). The goal of 3D-HEVC is to improve the coding technology based on HEVC to efficiently compress multiple viewpoints and their corresponding depth data. 3D-HEVC includes all the key technologies of HEVC and adds technologies that are beneficial to multi-view video coding to improve the efficiency of 3D video encoding and decoding. Compared with two-dimensional (2D) video coding, 3D video coding sends depth maps to facilitate the synthesis of virtual viewpoints at the decoding end, but there are certain differences in color and depth. Existing video coding tools are not suitable for depth map coding, so the research on 3D-HEVC began. However, existing methods are committed to modifying the internal modules of 3D-HEVC to improve performance, without considering the characteristics and correlation of depth maps and color maps.

[0031] On the other hand, the advancement of sensor technology, especially the popularity of multi-sensor data, provides new opportunities to reform and improve compression efficiency by utilizing multi-sensor collaboration. Due to the unique properties of each sensor, multi-sensor data has obvious advantages over single-sensor data. Multi-sensor collaboration, such as color images and depth images, can significantly improve coding efficiency. However, traditional video codecs, including 3D-HEVC, only save bits by removing redundancy, without considering saving bits through multi-sensor collaboration. In addition, traditional 3D-HEVC has low compression efficiency for depth data, and obvious block artifacts will appear when the quantization parameter (QP) is large. Although most existing deep learning-based methods can accelerate the prediction mode decision for coding unit / prediction unit (CU / PU), they cannot handle the block artifacts in the case of large QP caused by 3D-HEVC. In addition, existing methods focus on improving video quality or reducing bit rate, but not both.

[0032] In order to save bits in the bitstream and improve the video / image quality, the present application provides various schemes for image and video encoding based on guided reconstruction using multi-sensor collaboration (e.g., color images and depth images) and frequency adaptive processing. As described in detail below, the present application can be implemented for compression of various multi-sensor data, such as color / depth, color / near-infrared (NIR), visible light (VIS) / infrared (IR), or color / light detection and ranging (LiDAR), and various video compression standards (also known as codecs) can be used, such as HEVC, VVC, audio video standard (AVS), etc. In some embodiments, the color image acquired by the color sensor represents the color and texture of the scene, while the depth image acquired by the depth sensor represents the 3D geometry of the scene. These two types of sensor data can be complementary, and the color image can help reconstruct the depth image corresponding to it.

[0033] In some embodiments, in order to save bits in the bitstream in multi-sensor collaboration, the original depth image can be downsampled (i.e., the downsampling factor is greater than 1) on the encoding side to make it a low resolution (LR) depth image, and the downsampled LR depth image and the corresponding color image can be compressed, for example, by 3D-HEVC, respectively, into the bitstream to be sent to the decoding side. In some embodiments, in order to maintain the best video / image quality, the original depth image may not be downsampled (i.e., the downsampling factor is equal to 1) on the encoding side, so that the depth image to be compressed and the corresponding color image have the same resolution (i.e., size). The depth image and the color image can be compressed into the bitstream separately by 3D-HEVC to be sent to the decoding side.

[0034] On the decoding side, guided reconstruction can combine the color information and depth information from the color image and the depth image, respectively. If the depth image has been downsampled, the guided reconstruction can use the above color information and depth information to reconstruct and restore the high resolution (HR) depth image. If the depth image has not been downsampled, the guided reconstruction can use the above color information and depth information to improve the quality of the depth image. For example, quality enhancement (e.g., removing block artifacts caused by 3D-HEVC encoding and decoding) and bit reduction can be balanced and optimized under different upsampling factors as needed. Consistent with the scope of the present application, guided reconstruction uses a frequency adaptive processing scheme to process different frequency components of the input image using different machine learning models designed and trained for different frequency components (e.g., convolutional neural networks (CNNs)) to obtain more accurate results. In some embodiments, a high frequency (HF) learning model is used to estimate rapid transitions in depth, such as edges and details of scenes, while a low frequency (LF) learning model is used to estimate slow changes in depth, such as smooth areas of scenes. The final depth image can be reconstructed by combining and upsampling features estimated from learning models for different frequency components.

[0035] In some embodiments, the design of the learning model is not limited by the downsampling factor value, and therefore has the flexibility to balance efficiency and performance without considering a specific downsampling factor. That is, the design of the learning model can be independent of different downsampling factors and remain essentially unchanged under different downsampling factors. In some embodiments, a discrete wavelet transform (DWT) is used to separate subband components of different frequencies from the input image before the learning model is used, and the size / resolution relationship of the input image is adjusted to ensure the consistency of the learning model under different upsampling factors.

[0036] Figure 1 A block diagram of an exemplary encoding system 100 according to some embodiments of the present application is shown. Figure 2A block diagram of an exemplary decoding system 200 according to some embodiments of the present application is shown. System 100 or system 200 can each be applied to or integrated into various systems and devices capable of data processing, such as computers and wireless communication devices. For example, system 100 or system 200 can be all or part of a mobile phone, a desktop computer, a laptop computer, a tablet computer, a car computer, a game console, a printer, a positioning device, a wearable electronic device, a smart sensor, a virtual reality (VR) device, an argument reality (AR) device, or any other suitable electronic device with data processing capabilities. Figure 1 and Figure 2 As shown, system 100 or system 200 may include a processor 102, a memory 104, and an interface 106. These components are shown as being interconnected via a bus, but other types of connections may also be provided. It is understood that system 100 or system 200 may include any other suitable components for performing the functions described herein.

[0037] Processor 102 may include a microprocessor, such as a graphics processing unit (GPU), an image signal processor (ISP), a central processing unit (CPU), a digital signal processor (DSP), a tensor processing unit (TPU), a vision processing unit (VPU), a neural processing unit (NPU), a synergistic processing unit (SPU), or a physical processing unit (PPU), a microcontroller unit (MCU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device (PLD), a state machine, gated logic, discrete hardware circuits, and other suitable hardware for performing the various functions described in the present application. Although Figure 1 and Figure 2Only one processor is shown, but it is understood that multiple processors may be included. Processor 102 may be a hardware device having one or more processing cores. Processor 102 may execute software. Whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise, software may be broadly understood as instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, processes, functions, and the like. Software may include computer instructions written in an interpreted language, a compiled language, or machine code. Under the broad category of software, other techniques for indicating hardware may be used.

[0038] Memory 104 may broadly include memory (also known as main / system memory) and storage (also known as auxiliary memory). For example, memory 104 may include random-access memory (RAM), read-only memory (ROM), static RAM (SRAM), dynamic RAM (DRAM), ferroelectric RAM (FRAM), electrically erasable programmable ROM (EEPROM), compact disc read-only memory (CD-ROM), or other optical storage, hard disk drive (HDD) (such as magnetic disk storage or other magnetic storage device), flash drive, solid-state drive (SSD), or any other medium that can be used to carry or store the desired program code in the form of instructions that can be accessed and executed by processor 102. In a broad sense, memory 104 can be embodied as any computer-readable medium, such as a non-transitory computer-readable medium. Although Figure 1 and Figure 2 Only one memory is shown, but it can be understood that multiple memories may be included.

[0039] The interface 106 may broadly include a data interface and a communication interface for receiving and sending signals in the process of receiving and sending information with other external network elements. For example, the interface 106 may include an input / output (I / O) device and a wired or wireless transceiver. Figure 1 and Figure 2 Only one memory is shown, but it can be understood that multiple interfaces may be included.

[0040] The processor 102, the memory 104, and the interface 106 can be implemented in various forms in the system 100 or the system 200 for performing video encoding functions. In some embodiments, the processor 102, the memory 104, and the interface 106 of the system 100 or the system 200 are implemented (e.g., integrated) on one or more systems-on-chip (SoC). In one example, the processor 102, the memory 104, and the interface 106 can be integrated on an application processor (application processor, AP) SoC, which controls application processing in an operating system (operating system, OS) environment, including running video encoding and decoding applications. In another example, the processor 102, the memory 104, and the interface 106 can be integrated on a dedicated processor chip for video encoding, such as a GPU or ISP chip dedicated to image and video processing in a real-time operating system (real-time operating system, RTOS).

[0041] like Figure 1 As shown, in the encoding system 100, the processor 102 may include one or more modules, such as an encoder 101. Figure 1 The encoder 101 is shown as being located within a processor 102, but it is understood that the encoder 101 may include one or more submodules that may be implemented on different processors that are close to or far from each other. The encoder 101 (and any corresponding submodules or subunits) may be a hardware unit (e.g., part of an integrated circuit) of the processor 102, designed to be used with other components or software units implemented by the processor 102 by executing at least a portion of a program (i.e., instructions). The instructions of the program may be stored on a computer-readable medium, such as a memory 104, and when executed by the processor 102, the instructions may perform a process having one or more functions related to image and video encoding, such as downsampling, image partitioning, inter-frame prediction, intra-frame prediction, transforms, quantization, filtering, entropy coding, etc., described in detail below.

[0042] Similarly, if Figure 2 As shown, in the decoding system 200, the processor 102 may include one or more modules, such as a decoder 201. Figure 2The decoder 201 is shown to be located within a processor 102, but it is understood that the decoder 201 may include one or more submodules that may be implemented on different processors that are close to or far from each other. The decoder 201 (and any corresponding submodules or subunits) may be a hardware unit (e.g., part of an integrated circuit) of the processor 102, designed to be used with other components or software units implemented by the processor 102 by executing at least a portion of a program (i.e., instructions). The instructions of the program may be stored on a computer-readable medium, such as a memory 104, and when executed by the processor 102, the instructions may perform a process having one or more functions related to image and video decoding, such as entropy decoding, upsampling, inverse quantization, inverse transform, inter-frame prediction, intra-frame prediction, filtering, as described in detail below.

[0043] In accordance with the scope of the present application, for multi-sensor cooperative applications, at least the encoding system 100 also includes a plurality of sensors 108 coupled to the processor 102, the memory 104, and the interface 106 via a bus. The sensor 108 may include a first sensor 108 and a second sensor 108, the first sensor 108 being used to acquire a first image of a scene (e.g., including one or more objects, also referred to as scene objects), and the second sensor 108 being used to acquire a second image of the same scene. That is, the first image and the second image may be acquired by two different sensors 108. In some embodiments, the first image and the second image of the scene are different types of images, but are complementary to each other with respect to the scene. In other words, the first sensor 108 and the second sensor 108 may acquire different types of image information of the scene, which, when combined together, provide a comprehensive visual representation of the scene. The first image and the second image may also have related features, i.e., may be related images of the same scene. In some embodiments, the first image and the second image may both reflect the edge of the scene (including the edge of an object in the scene). For example, the first image acquired by the first sensor 108 is a depth image, and the second image acquired by the second sensor 108 is a color image. The depth image and the color image may be associated with each other because the two images may represent the edge of the same scene. The depth image and color image may also complement each other with respect to a scene (eg, a 3D scene) because the depth image and color image when combined may provide a comprehensive visual representation of the scene.

[0044] In some embodiments, the first sensor 108 is a depth sensor, and the depth image represents the 3D geometry of the scene. For example, the depth sensor may include any 3D rangefinder that obtains multi-point distance information over a wide field of view (field-of-view, FoV), such as a LiDAR distance sensor, a time-of-flight (time-of-flight, ToF) camera, or a light field camera. In some embodiments, the second sensor 108 is a color sensor, and the color image represents the color of the scene. It is understood that the "color" referred to herein may also include texture and grayscale. For example, the color sensor may include any sensor that detects the color of light reflected from an object in any suitable spectrum, such as a VIS sensor, an IR sensor, a VIS-IR sensor, a NIR sensor, or a red-green-blue / NIR (RGB-NIR) sensor. It is understood that various types of multi-sensor images (data) are not limited to color images and depth images, and in other examples may be any other suitable type of image of the same scene. It is also understood that the number of sensors 108 and the type of multi-sensor images are not limited to two, and in other examples may be more than two. It is also understood that the sensor 108 can be used to obtain different types of videos, such as color videos and depth videos, each video including multiple frames, such as color frames and depth frames. Figure 2 Not shown, but in some examples, decoding system 200 may also include sensor 108 like encoding system 100 .

[0045] Figure 3 Detailed block diagram of an exemplary encoding and decoding system 300 according to some embodiments of the present application is shown. The system 300 may be a combination of the encoding system 100 and the decoding system 200 described above, or any part of the combination. Figure 3As shown, the system 300 may include a first sensor 108 and a second sensor 108, an encoder 101, and a decoder 201. The first sensor 108 may be used to obtain a depth image D of a scene, and the second sensor 108 may be used to obtain a color image I of the same scene. As described above, for ease of description, the color image and the depth image are used as examples of two complementary images of the same scene, and any other suitable type of complementary image of the same scene may also be used. As described above, the color image and the depth image may be a still image (image) or a video frame captured by the sensor 108. It is understood that in some examples, the depth image D and / or the color image I may not be obtained by the first sensor 108 and / or the second sensor 108, but by any other suitable means. In one example, the depth image D and / or the color image I may be derived from other images of the scene or otherwise obtained using any suitable image analysis or processing technology. In another example, the depth image D and / or the color image I may be obtained by a third party and sent from the third party to the encoder 101.

[0046] like Figure 3 As shown, the encoder 101 (e.g., a part of the encoding system 100 or the encoding and decoding system 300) may include a downsampling module 302 and a compression module 304. According to some embodiments, the downsampling module 302 is used to downsample the original depth image D to generate a downsampled depth image D s For example, the original depth image D can be a HR depth image, and the downsampled depth image D s It can be a downsampled LR depth image. It is understood that unless otherwise specified, LR images and downsampled images can be used interchangeably in this application. The downsampling module 302 can downsample the original depth image D to reduce its size (resolution) by any suitable downsampling technique, including but not limited to interpolation, uniform sampling, machine learning model, or any combination thereof. In this way, the downsampled depth image D s The amount of data (eg, the number of bits) of may become smaller than the amount of data representing the original depth image D. In contrast, in some embodiments, the original color image I does not pass through the downsampling module 302 and is therefore not downsampled.

[0047] It is understood that in some embodiments, the original depth image D is not downsampled. That is, the downsampling module 302 can be bypassed so that the original depth image D (eg, HR depth image) can be used instead of the downsampled depth image D. s(e.g., LR depth image) is provided to the compression module 304. In some embodiments, assuming that the size of the original color image I and the depth image D are both (W, H, C), and the depth video downsampling can be performed by the downsampling module 302 using a uniform sampling operation with a downsampling factor of s, the depth image D or D s The height and width of may become (W / s, H / s, C), where C represents the number of channels (e.g., for an RGB color image I, C=3). When the downsampling factor s is set to 1, the downsampling module 302 may be bypassed (the original depth image D is not downsampled), so that the size of the original depth image D remains unchanged, that is, the same as the size of the original color image I. When the downsampling factor is set to be greater than 1 (e.g., 2 or 4), the downsampling module 302 may downsample the original depth image D to generate a downsampled depth image D s , so that the downsampled depth image D s The size of is reduced, that is, it becomes smaller than the size of the original color image I. It is understandable that the value of the downsampling factor is not limited to 1, 2, 4, and can be any suitable value, such as 2.5, 3, 6, 8, etc. It is understandable that in this application, as long as the height (H) and width (W) of two images of the same scene are the same, the size (resolution) of the two images can be considered to be the same, regardless of the number of channels. That is to say, in this application, the size (resolution) of the two images can be compared by comparing the height and width of the two images, rather than comparing the size of the two images by comparing the number of channels.

[0048] In some embodiments, the compression module 304 is used to convert the original color image I and the downsampled depth image D s (or the original depth image D) are compressed (encoded) into a bitstream respectively. The original color image I and the downsampled / original depth image D can be s / D independently performs compression. The compression module 304 can use any suitable compression technology (codec), including but not limited to 3D-HEVC, VVC, AVS, etc. to perform compression. For example, Figure 4 Detailed block diagram of an exemplary compression module 304 for implementing 3D-HEVC encoding and decoding according to some embodiments of the present application is shown. Figure 4 As shown, the compression module 304 implementing the 3D-HEVC codec may include a video encoder 402 conforming to HEVC and a depth map encoder 404 for respectively encoding color frames and depth frames in independent views of the 3D video. The compression module 304 may also include multiple video encoders 406 and depth encoders 408, each of which is used to encode a corresponding color frame and a corresponding depth frame in a corresponding view of the N related views of the 3D video. Each depth frame may be Figure 3The downsampled / original depth image D in s / D corresponds, and each color frame can be Figure 3 The compression module 304 may further include a multiplexer (MUX) 410 to sequentially select encoded data from each encoder 402, 404, 406, or 406 to form a bitstream. That is, the compression module 304 implementing the 3D-HEVC codec may compress multiple viewpoints and their corresponding depth data into a bitstream by sending a depth map to facilitate the synthesis of virtual viewpoints during decoding.

[0049] Refer again Figure 3 , the decoder 201 (eg, a part of the decoding system 200 or the encoding and decoding system 300) may include a decompression module 306 and a guided reconstruction module 308. In some embodiments, the compressed original color image I and the downsampled / original depth image D are included. s The bit stream of / D is received from encoder 101 (e.g., via Figure 1 The interface 106 of the encoding system 100 in the decoder 201) is sent to the decoder 201. That is, the decompression module 306 of the decoder 201 can (for example, by Figure 2 The interface 106 of the decoding system 200 in FIG. 1 receives the compressed original color image I and the downsampled / original depth image D s In some embodiments, the size (resolution) of the depth image is reduced from the original depth image D to the downsampled depth image D s (e.g., s>1), the number of bits sent in the bitstream can be reduced to improve the throughput of the system 300. The decompression module 306 can be used to decompress the compressed original color image I and the downsampled / original depth image D from the bitstream using the same compression technology (codec) implemented by the compression module 304, including but not limited to 3D-HEVC, VVC, AVS, etc. s / D, to reconstruct the color image I' and the downsampled depth image D respectively s '. The compressed color image I and the downsampled / original depth image D can be s In some embodiments, the decompression module 306 implements 3D-HEVC codec to obtain the color image I' and the downsampled depth image D respectively. s '(or depth image D'). It can be understood that in some examples, the depth image D sThe depth image D and / or the color image I' may not be obtained by the decoder 201 from the decompressed bitstream, but may be obtained by any other suitable means. In one example, the depth image D may be derived or otherwise obtained from other images of the scene using any suitable image analysis or processing technique. s In another example, the depth image D may be acquired by a sensor directly coupled to the decoder 201 (on the decoder side, not shown). s ' / D' and / or color image I'.

[0050] It is understood that, depending on the codec used by the compression module 304 and the decompression module 306, the decompression module 306 may not be able to perfectly restore the original color image I and depth image D from the bitstream due to information loss, for example. s ' / D'. In other words, the color image I' may be different from the color image I, and the depth image D s ' / D' may be related to the depth image D s / D is different. It is understandable that in the present application, the compression and decompression processes respectively performed by the compression module 304 and the decompression module 306 may sometimes be collectively referred to as a compression process. Therefore, the color image I' and the depth image D outputted from the decompression module 306 are different. s ' / D' can also sometimes be referred to as the compressed color image I' and the compressed depth image D s ' / D'. Depending on the compression efficiency of the depth data and / or the QP used by the compression module 304 and the decompression module 306 during the compression / decompression process, the depth image D s The edges of the scene on the ' / D' and color image I' may show block artifacts and information loss, resulting in blur or other distortion. For example, when using the 3D-HEVC codec with relatively low depth data compression efficiency, the larger the QP, the smaller the depth image D s The edges of the scene may be more blurred on ' / D'.

[0051] The guided reconstruction module 308 can be used to obtain the depth image D s ' / D' reconstructs a depth image D" (e.g., HR depth image). However, the traditional LR depth upsampling process may cause blurring due to the lack of high-frequency components, especially at the edges of the scene. On the other hand, the color image I' has many clear and complete edges that can be used to reconstruct the depth image D". In addition, the color image I' may also contain edges that are unnecessary for reconstructing the depth image D", thereby causing texture copying artifacts after reconstructing the depth image D". Although the edges in the color image I' can provide important guidance for reconstructing the depth image D", they cannot be directly used for upsampling.

[0052] In keeping with the scope of the present application, the guided reconstruction module 308 may be used to reconstruct the image D from the depth image D under the guidance of the color image I'. s ' / D' reconstructs the depth image D". According to some embodiments, because the original color image I and the original depth image D are related, the compressed color image I' and the depth image D s ' / D' is also related. Therefore, the color image I' and the depth image D s The relevant features in ' / D' (e.g., at the edge of the scene) and their information can be combined in the guided reconstruction module 308 to be used to restore the depth image D after upsampling. Depending on whether the input depth image of the reconstruction module 308 is downsampled (e.g., s>1) or not downsampled (e.g., s=1), the guided reconstruction module 308 can upsample and restore the downsampled depth image D s ', or improve the quality of the depth image D'.

[0053] In some embodiments, the guided reconstruction module 308 separates sub-band components of the color image I' at different frequencies (e.g., one or more high-frequency components and one or more low-frequency components), and combines these sub-band components with sub-band components of the depth image D' at different frequencies, or with the downsampled depth image D s As described below, for example, DWT and / or downsampling can be used to extract the color image I' and / or the depth image D s ' / D' is preprocessed to separate high-frequency components and low-frequency components and / or to adjust and match the color image I' and the depth image D s The size of the initial feature of ' / D'. In some embodiments, the guided reconstruction module 308 then uses different machine learning models designed and trained for different frequency components (e.g., high-frequency component CNN and low-frequency component CNN) to process the combined initial features at different frequencies to estimate the features at different frequencies, respectively. In some embodiments, the guided reconstruction module 308 then combines and upsamples the estimated features at different frequencies to reconstruct the depth image D". For example, the combined features can be upsampled using a machine learning model (e.g., an upsampling CNN) to reconstruct the depth image D".

[0054] As described above, it is understood that when the system 300 uses multi-sensor collaboration for image and video encoding, the number and / or type of images that can be applied are not subject to the above-mentioned Figure 3 The limitations of the examples described may vary in other examples. It is understood that Figure 3Each element shown is independently shown to represent the characteristic functions different from each other in system 300, and does not mean that each assembly is formed by the configuration unit of independent hardware or single software.That is to say, each element is included to be listed as an element for the convenience of explanation, and at least two elements can be combined to form a single element, or an element can be divided into multiple elements to perform a function.It can also be understood that some elements are not the necessary elements for performing the functions described in the present application, but can be optional elements for improving performance.It can also be understood that these elements can be realized using electronic hardware, firmware, computer software or any combination thereof.Whether these elements are implemented as hardware, firmware or software depends on the specific application and design constraints applied on system 300.

[0055] Figure 5 Detailed block diagram of an exemplary guided reconstruction module 308 according to some embodiments of the present application is shown. Figure 5 As shown, the guided reconstruction module 308 may include a preprocessing unit 504, a preprocessing unit 505, a HF estimation unit 506, a LF estimation unit 507, and a reconstruction unit 508. The preprocessing unit 504 may be used to receive a guided input 502 (e.g., a color image I'), and extract components of the guided input 502 at different frequencies from the guided input 502. In some embodiments, the preprocessing unit 504 may be used to separate one or more first components of the guided input 502 at a first frequency from the guided input 502, and to separate one or more second components of the guided input 502 at a second frequency from the guided input 502. For example, the first frequency may be higher than the second frequency. For ease of description, the first frequency may be referred to as HF herein, and the second frequency may be referred to as LF. In an example where the guided input 502 is a color image I', the HF component of the color image I' may indicate a rapid transition of color or texture, such as edges and details, while the LF component of the color image I' may indicate a slow change of color or texture, such as a smooth and continuous area. In some embodiments, the preprocessing unit 504 may also be used to extract initial HF features of the guide input 502 from the HF component and extract initial LF features of the guide input 502 from the LF component. That is, the preprocessing unit 504 may extract initial features of corresponding frequencies from each frequency component.

[0056] In the target input 503 (eg, the downsampled depth image D sIn an embodiment where the size of the target input 503 is smaller than the size of the guide input 502 (e.g., the color image I'), the preprocessing unit 504 may also be used to downsample the guide input 502 based on the size of the target input 503. For example, the preprocessing unit 504 may downsample the extracted HF initial features and LF initial features so that the resolution of the HF initial features and the LF initial features becomes equal to the resolution of the extracted HF initial features and the LF initial features from the target input 503 (e.g., the downsampled depth image D s '). It can be understood that in some cases, for example, when the target input 503 (e.g., the downsampled depth image D s ') is equal to the size of the guide input 502 (e.g., color image I'), and / or when the resolution of the HF initial features and the LF initial features of the guide input 502 is equal to the resolution of the initial features of the target input 503, the downsampling function of the preprocessing unit 504 may not be performed.

[0057] Similarly, in some embodiments, the preprocessing unit 505 may be used to receive a target input 503 (e.g., a depth image D'), and to extract components of the target input 503 at different frequencies from the target input 503. In some embodiments, the preprocessing unit 505 may be used to separate one or more first components of the target input 503 at a first frequency (e.g., HF) from the target input 503, and to separate one or more second components of the target input 503 at a second frequency (e.g., LF) from the target input 503. In an example where the target input 503 is a depth image D', the HF component of the depth image D' may indicate a rapid transition in depth, such as an edge, while the LF component of the depth image D' may indicate a slow change in depth, such as a smooth area. In some embodiments, the preprocessing unit 505 may also be used to extract initial HF features of the target input 503 from the HF component, and to extract initial LF features of the target input 503 from the LF component. That is, the preprocessing unit 505 may extract initial features of the corresponding frequencies from each frequency component, respectively.

[0058] It is understood that in some cases, for example, depending on the size of the target input 503, such as when the target input 503 is a downsampled depth image D s ', the frequency component separation function of the preprocessing unit 505 may not be performed. In these cases, the preprocessing unit 505 may be used to extract the initial features of the target input 503 from the target input 503. That is, the preprocessing unit 505 does not extract the initial features of the corresponding frequencies from each frequency component separately, but can directly extract the initial features of the target input 503 (for example, the downsampled depth image D s ') to extract initial features without frequency component separation.

[0059] In some embodiments, the pre-processing unit 504 may also be used to combine the initial HF features of the guide input 502 with the initial features of the target input 503 from the pre-processing unit 505. Depending on whether the pre-processing unit 505 performs a frequency component separation function, the pre-processing unit 504 may combine the initial HF features of the guide input 502 with the initial HF features of the target input 503, or combine the initial HF features of the guide input 502 with the initial features of the target input 503. Similarly, in some embodiments, the pre-processing unit 505 may also be used to combine the initial LF features of the guide input 502 from the pre-processing unit 504 with the initial features of the target input 503. Depending on whether the pre-processing unit 505 performs a frequency component separation function, the pre-processing unit 505 may combine the initial LF features of the guide input 502 with the initial LF features of the target input 503, or combine the initial LF features of the guide input 502 with the initial features of the target input 503.

[0060] That is, because different frequency components represent different information types of the guide input 502 and the target input 503, the preprocessing unit 504 and the preprocessing unit 505 can work together to combine the initial features at different frequencies (e.g., HF and LF) from the guide input 502 and the target input 503 to improve the estimation accuracy. In addition, when the sizes of the guide input 502 and the target input 503 do not match, the preprocessing unit 504 and / or the preprocessing unit 505 can also adjust the size of the guide input 502 and / or the target input 503 so that the initial features from the guide input 502 and the target input 503 can be combined.

[0061] In some embodiments, the HF estimation unit 506 may be configured to receive a combined initial feature (e.g., HF initial feature) at a first frequency (e.g., the combined HF initial features of the guide input 502 and the target input 503, or the combined HF initial features of the guide input 502 and the initial features of the target input 503) from the preprocessing unit 504, and estimate first information (e.g., HF information) at the first frequency of the target input 503 based on the combined initial features at the first frequency. Similarly, in some embodiments, the LF estimation unit 507 may be configured to receive a combined initial feature (e.g., LF initial feature) at a second frequency (e.g., the combined LF initial features of the guide input 502 and the target input 503, or the combined LF initial features of the guide input 502 and the initial features of the target input 503) from the preprocessing unit 505, and estimate second information (e.g., LF information) at a second frequency of the target input 503 based on the combined initial features at the second frequency. In some embodiments, the depth image D sThe first information of ' / D' includes features indicating the edge of the scene, the depth image D s The second information of ' / D' includes features indicating smooth regions of the scene. In some embodiments, the estimation functions of the HF estimation unit 506 and the LF estimation unit 507 may be performed using machine learning models designed and trained for corresponding frequency components.

[0062] In some embodiments, the reconstruction unit 508 may be used to reconstruct the target input 503 based on the estimated first information and the second information to generate a target output 510 (e.g., a depth image D ″). In order to reconstruct the target input 503, the reconstruction unit 508 may be used to combine the estimated first information and the second information of the target input 503, and upsample the target input 503 based on the combined first information and the second information. That is, the HF feature indicating the edge of the scene may be used as a depth image D ″ in which the reconstructed edge becomes blurred or distorted due to the lack of the HF component. s In some embodiments, the upsampling function of the reconstruction unit 508 may be performed using a machine learning model designed and trained for upsampling.

[0063] Consistent with the scope of this application, for example, FIG. 6A to FIG. 6C As shown, in some embodiments, the HF estimation unit 506 , the LF estimation unit 507 , and the reconstruction unit 508 use a machine learning model such as a CNN to improve the efficiency and effectiveness of the guided reconstruction module 308 . FIG. 6A to FIG. 6C Will combine Figure 5 The following description is provided to describe the guided reconstruction module 308 that implements the machine learning model under different downsampling factors (e.g., 1, 2, and 4). It is understood that in some examples, the guided reconstruction module 308 can implement any other suitable machine learning model, such as regression, support vector machine (SVM), decision tree, Bayesian network, etc.

[0064] like Figure 5 and Fig. 6AAs shown, the guide input 502 may be an input color image I', and the target input 503 may be an input depth image D'. That is, in this example, the downsampling factor of the input depth image D' may be 1, and the size of the input color image I' may be the same as the size of the input depth image D'. The preprocessing unit 504 may include a DWT unit 602, a feature extraction unit 604, and a splicing (C) layer 606. DWT is a transform that decomposes a given signal into multiple sets, each of which is a time series of coefficients describing the temporal evolution of the signal in a corresponding frequency band. The DWT unit 602 may perform a DWT on the input color image I' to separate the input color image I' into four sub-band components: three HF components (HH, HL, LH) obtained by scanning the input color image I' in three different directions (height, width, diagonal), and one LF component (LL). In addition to the frequency component separation, the DWT unit 602 may also reduce the size of the input color image I' by half, i.e., downsample the input color image I' using a downsampling factor of 2. It is understood that in some examples, the DWT unit 602 can be replaced by any suitable unit that can perform frequency component separation and downsampling on the input color image I', such as any other suitable transform or CNN. The feature extraction unit 604 may include a convolution layer (e.g., a 3×3 convolution layer) for extracting initial HF features from three HF components (HH, HL, LH), and a convolution layer for extracting initial LF features from one LF component (LL). It is understood that, as Fig. 6A As shown, the number of convolutional layers used by the feature extraction unit 604 is not limited to one, and can be any suitable number.

[0065] Similarly, the preprocessing unit 505 may include a DWT unit 603, a feature extraction unit 605, and a splicing layer 607. The DWT unit 603 may perform a DWT on the input depth image D' to separate the input depth image D' into four sub-band components: three HF components (HH, HL, LH) obtained by scanning the input depth image D' in three different directions (height, width, diagonal), and one LF component (LL). In addition to the frequency component separation, the DWT unit 603 may also reduce the size of the input depth image D' by half, that is, downsample the input depth image D' using a downsampling factor of 2. It can be understood that in some examples, the DWT unit 603 may be replaced by any suitable unit that can perform frequency component separation and downsampling on the input depth image D', such as any other suitable transform or CNN. The feature extraction unit 605 may include a convolution layer (e.g., a 3×3 convolution layer) for extracting initial HF features from the three HF components (HH, HL, LH), and a convolution layer for extracting initial LF features from one LF component (LL). It is understandable that if Fig. 6A As shown, the number of convolutional layers used by the feature extraction unit 605 is not limited to one, and can be any suitable number.

[0066] The stitching layer 606 of the preprocessing unit 504 may stitch the initial HF features of the input color image I' with the initial HF features of the input depth image D' to form a stitched initial HF feature of the input color image I' and the depth image D'. Similarly, the stitching layer 607 of the preprocessing unit 505 may stitch the initial LF features of the input color image I' with the initial LF features of the input depth image D' to form a stitched initial LF feature of the input color image I' and the depth image D'. In order to perform the stitching operation, the sizes (resolutions) of the two feature maps need to be the same. Therefore, the size of the extracted HF features of the input color image I' may be the same as the size of the extracted HF features of the input depth image D', and the size of the extracted LF features of the input color image I' may be the same as the size of the extracted LF features of the input depth image D'.

[0067] The HF estimation unit 506 may include a HF machine learning model having one or more sub-models 608 for HF components and features. The HF machine learning model may be a CNN, and each sub-model 608 may be a CNN block. The sub-model 608 may include multiple levels, such as one or more dense blocks, 1×1 convolutional layers, and additional convolutional layers (e.g., 3×3 convolutional layers) to gradually extract HF features from the initial HF features and obtain deep semantic information with multi-scale HF features. In one example, each dense block may include four convolutional layers and one concatenation layer. For example, semantic information with multi-scale HF features from one or more dense blocks of the sub-model 608 may be used to distinguish valid edges from invalid edges. In one example, a 1×1 convolutional layer may be used to reduce the number of channels of the extracted HF features to reduce the computational complexity of the HF machine learning model. In some embodiments, for example, the initial HF features (i.e., the HF features) may be added together by an addition operation. Fig. 6A The input of the first sub-model 608 shown in FIG. 5 ) and / or any intermediate HF features (not shown) between the sub-models 608 are added to the output of the last sub-model 608 to obtain the residual between the two, thereby further reducing the computational complexity of the HF machine learning model. For example, pixel-level element-by-element addition can be performed. Finally, the estimated HF features can be extracted by the last 3×3 convolution layer, and the estimated HF features are provided as the output HF information of the HF estimation unit 506.

[0068] Similarly, the LF estimation unit 507 may include an LF machine learning model having one or more sub-models 609 for LF components and features. The LF machine learning model may be a CNN, and each sub-model 609 may be a CNN block. The sub-model 609 may include multiple levels, such as one or more residual blocks (ResBlocks), 1×1 convolutional layers, and additional convolutional layers (e.g., 3×3 convolutional layers) to gradually extract LF features from the initial LF features and obtain deep semantic information with multi-scale LF features. In one example, since LF features are simpler and more predictable than HF features, the residual block may include fewer convolutional layers (e.g., two convolutional layers) than the dense block. Each residual block may be replaced by a dense block. For example, semantic information with multi-scale LF features from one or more residual blocks of the sub-model 609 can be used to distinguish between valid smooth areas and invalid smooth areas. In one example, a 1×1 convolutional layer can be used to reduce the number of channels of the extracted LF features to reduce the computational complexity of the LF machine learning model. In some embodiments, for example, the initial LF features (i.e., Fig. 6A The input of the first sub-model 609 shown in the figure) and / or any intermediate LF features (not shown) between the sub-models 609 are added to the output of the last sub-model 609 to obtain the residual between the two, thereby further reducing the computational complexity of the LF machine learning model. For example, pixel-level element-by-element addition can be performed. Finally, the estimated LF features can be extracted by the last 3×3 convolutional layer and provided as the output LF information of the LF estimation unit 507.

[0069] like Fig. 6AAs shown, the reconstruction unit 508 may include a splicing layer 610 and an upsampling unit 612. The splicing layer 610 may splice the estimated HF features from the HF estimation unit 506 with the estimated LF features from the LF estimation unit 507. As described above, due to the loss of edge information (i.e., lack of HF components) caused by compression / decompression, the edges of the scene in the depth image D' may become blurred or distorted, which can be compensated by the HF features estimated from the HF components of the color image I' and the depth image D'. The upsampling unit 612 may use an upsampling machine learning model (e.g., CNN) to upsample the input depth image D' (downsampled to half by the DWT unit 603) based on the spliced ​​HF and LF features to reconstruct an output depth image D'' of the same size as the input depth image D'. The upsampling machine learning model may include one or more upsampling layers to learn and restore the output depth image D''. It will be appreciated that any other suitable upsampling method, such as interpolation or inverse DWT (IDWT), may be used to replace the upsampling machine learning model, provided that the four subband components (HH, HL, LH, and LL) for IDWT can be obtained from the concatenated HF and LF features.

[0070] like Figure 6B As shown in another example, the guide input 502 may still be the input color image I', and the target input 503 may become a downsampled input depth image D having a size half that of the original depth image D. s '. That is, in this example, the input depth image D s The downsampling factor of ' can be 2, and the size of the input color image I' can be the size of the input depth image D s ' twice. Fig. 6A The example is different. Figure 6B The preprocessing unit 505 in the embodiment may not include the DWT unit 603, so that the convolution layer of the feature extraction unit 605 can directly extract the input depth image D without frequency component separation. s ', rather than extracting the initial HF features and LF features. On the one hand, since the input depth image D s 'After downsampling, using DWT for frequency component separation may not be able to obtain useful HF or LF features from the downsampled image. On the other hand, DWT may further s 'Downsampling is performed, resulting in the initial HF / LF feature size of the input color image I' being the same as the input depth image D s 'The initial HF / LF feature sizes do not match. That is, when the downsampling factor is 2, due to the input depth image D sThe size of ' is half of the input color image I', so DWT can be performed only on the input color image I'.

[0071] like Figure 6B As shown, Fig. 6A Different from the example of FIG. 5 , the concatenation layer 606 of the pre-processing unit 504 can combine the initial HF features of the input color image I′ with the input depth image D s The initial features of the input color image I' are spliced ​​together by the splicing layer 607 of the pre-processing unit 505. The initial LF features of the input color image I' are spliced ​​together with the input depth image D s In this way, the size of the extracted HF features of the input color image I' can still be consistent with the input depth image D s The size of the extracted features of the input color image I' can still be the same as that of the input depth image D s The size of the extracted features of the concatenated input color image I' and the input depth image D s The initial features of the input color image I' and the input depth image D can be provided as the input of the HF estimation unit 506, the initial LF features of the concatenated input color image I' and the input depth image D s The initial features of ' can be provided as input to the LF estimation unit 507. In addition to the pre-processing units 504 and 505, Fig. 6A and Figure 6B The remaining elements may remain the same and thus will not be described in detail.

[0072] like Figure 6C As shown in another example, the guide input 502 may still be the input color image I', and the target input 503 may become a downsampled input depth image D with a size of one quarter of the original depth image D. s '. That is, in this example, the input depth image D s The downsampling factor of ' can be 4, and the size of the input color image I' can be the size of the input depth image D s Four times of '. Figure 6B The example is different. Figure 6C The preprocessing unit 504 in the embodiment may further include a downsampling unit 614 for downsampling the initial HF feature and the initial LF feature using a downsampling factor of 2, so that the size of the extracted HF feature of the input color image I' after passing through the downsampling unit 614 can still be the same as the input depth image D s The size of the extracted features of the input color image I' is the same as that of the input depth image D after the downsampling unit 614. sIt is understood that in some examples, downsampling can be performed before the feature extraction unit 604 or the DWT unit 602, as long as the size of the initial HF / LF features of the input color image I is the same as that of the input depth image D s The size of the initial features of ' can be matched at the concatenated layers 606 and 607. That is, the downsampling unit 614 of the preprocessing unit 504 can be used to downsample the input color image I', the separated HF components (HH, HL, LH) and LF components (LL) of the input color image I', or the extracted HF features and LF features of the input color image I'.

[0073] like Figure 6C As shown, Figure 6B The input depth image D s Compared with the example in which the downsampling factor is 2, the upsampling machine learning model of the upsampling unit 612 in the reconstruction unit 508 may have more layers (for example, two) to reconstruct the input depth image D with a downsampling factor of 4. s 'Up-sampling. In addition to the pre-processing units 504 and 505 and the reconstruction unit 508, Fig. 6A and Figure 6C The remaining elements may remain the same and thus will not be described in detail.

[0074] It can be understood that the number of sub-models 608 and 609 in the HF estimation unit 506 and the LF estimation unit 507, and the number of levels in each sub-model 608 or 609 are not limited to FIG. 6A to FIG. 6C , but may be any suitable number based on the efficiency and accuracy requirements of the guided reconstruction module 308. It is also understood that the number of sub-models 608 and 609 in the HF estimation unit 506 and the LF estimation unit 507, and the number of levels in each sub-model 608 or 609 are not affected by the input depth image D' / D s ', and therefore, can be optimized based on the efficiency and accuracy requirements of the guided reconstruction module 308, without being affected by the input depth image D' / D s '. It can also be understood that the input depth image D' / D s The downsampling factor is not limited to FIG. 6A to FIG. 6C The design of the guided reconstruction unit 308 can be similarly adjusted by introducing or removing the DWT unit 603, introducing or removing the downsampling unit 614, and / or adjusting the downsampling factor of the downsampling unit 614, so that the size of the initial HF / LF features of the input color image I' is the same as that of the input depth image D s ' is the same size as the initial features.

[0075] Figure 7 The anchor image and the anchor image at different QPs according to some embodiments of the present application are shown. Figures 6A to 6C Visual comparison between the various output depth images of the guided reconstruction module 308. Figure 7 As shown in the first column of , as QP increases, the results of the 3D-HEVC anchor point contain obvious blocky artifacts. Figure 7 In the second column, Fig. 6A The output depth image D" (eg, downsampling factor = 1) of the guided reconstruction module 308 in FIG. 3 has clear depth edges and significantly reduced block artifacts. As QP increases, Figure 7 In the third column, Figure 6B Some small edges of the output depth image D (e.g., downsampling factor = 2) of the guided reconstruction module 308 in the image are blurred, resulting in local distortion, but no obvious block artifacts. As QP increases, Figure 7 In the fourth column, Figure 6C The output depth image D of the guided reconstruction module 308 in (eg, downsampling factor = 4) becomes more blurred and distorted, but without severe block artifacts. The results show that FIG. 6A to FIG. 6C The guided reconstruction module 308 in the embodiment can effectively utilize multi-sensor collaboration for video compression and successfully suppress block artifacts.

[0076] In addition, compared with the 3D-HEVC anchor point, Figures 6A-6C The guided reconstruction module 308 in the 3D-HEVC can significantly save bits in terms of Bjontegaard Delta (BD) rate while maintaining depth quality. For example, when the downsampling factor is 1, the bit rate is the same as that of 3D-HEVC, but its peak signal-to-noise ratio (PSNR) is higher than the 3D-HEVC anchor point, which means Fig. 6A The guided reconstruction module 308 in can effectively improve the depth quality at the same bit rate; when the downsampling factor is 2, Figure 6B The guided reconstruction module 308 in can reduce the bit rate by 50% to 75% while maintaining the depth quality; when the downsampling factor is 4, compared with the 3D-HEVC anchor, Figure 6C The guided reconstruction module 308 in the example can further save about 60% to 90% of the bit rate. Although most of the depth information may be lost when the downsampling factor is 4, Figure 6C The guided reconstruction module 308 in can still maintain the depth quality to a certain extent.

[0077] Figure 8800 according to some embodiments of the present application. The method 800 may be performed by the encoder 101 of the encoding system 100, the encoding and decoding system 300, or any other suitable image and video encoding system. As described below, the method 800 may include operations 802, 804, 806, and 808. It is understood that some operations (e.g., operation 806) may be optional, and some operations may be performed simultaneously or in parallel. Figure 8 The order shown in the figure is different from the order in which it is executed.

[0078] In operation 802, a first sensor acquires a first image of a scene. In some embodiments, the first image is a depth image. Figure 3 As shown, the first sensor 108 can be used to acquire a depth image. In operation 804, the second sensor acquires a second image of the scene. In some embodiments, the first image and the second image are complementary to each other with respect to the scene. In some embodiments, the second image is a color image. Figure 3 As shown, the second sensor 108 can be used to obtain a color image. In optional operation 806, the first processor downsamples the first image. In some embodiments, in order to downsample the first image, at least one of interpolation, uniform sampling, or a machine learning model is used. Figure 1 and Figure 3 As shown, the downsampling module 302 of the encoder 101 in the processor 102 can be used to downsample the depth image. In operation 808, the first processor compresses the first image and the second image into a bit stream. Figure 1 and Figure 3 As shown, the compression module 304 of the encoder 101 in the processor 102 can be used to compress the depth image and the color image into an output bitstream (for example, using 3D-HEVC codec). It will be appreciated that operations 802, 804, 806, and 808 can be repeated for each frame of the video to encode the video (for example, a 3D video). It will also be appreciated that in some examples, the first image and / or the second image may not be acquired by the first sensor and / or the second sensor, but by any other suitable means. In one example, the first image and / or the second image may be derived from other images of the scene or otherwise acquired using any suitable image analysis or processing technique. In another example, the first image and / or the second image may be acquired by a third party and sent from the third party to the encoder.

[0079] Fig. 91 shows a flowchart of an exemplary decoding method 900 according to some embodiments of the present application. The method 900 may be performed by the decoding system 200, the encoding and decoding system 300, or the decoder 201 of any other suitable image and video decoding system. As described below, the method 900 may include operation 902, operation 904, operation 906, and operation 908. It is understood that some operations may be optional, and some operations may be performed simultaneously or in parallel. Fig. 9 The order shown in the figure is different from the order in which it is executed.

[0080] In operation 902, a second processor decompresses a bit stream to receive a first image and a second image. Figure 2 and Figure 3 As shown, the decompression module 306 of the decoder 201 in the processor 102 can be used to decompress the input bit stream (for example, using 3D-HEVC codec) to receive the depth image and the color image. It will be appreciated that in some examples, the first image and / or the second image may not be obtained by the decoder from the decompressed bit stream, but may be obtained by any other suitable means. In one example, the first image and / or the second image may be derived or otherwise obtained from other images of the scene using any suitable image analysis or processing technique. In another example, the first image and / or the second image may be obtained by a sensor directly coupled to the decoder. It will also be appreciated that whether the first image (for example, the depth image) obtained from the bit stream is a downsampled first image depends on whether the optional operation 806 is performed.

[0081] In operation 904, the second processor estimates first information at a first frequency of the first image based at least in part on a first component of the second image at the first frequency. The first information may include features indicating an edge of a scene. In operation 906, the second processor estimates second information at a second frequency of the first image based at least in part on a second component of the second frequency of the second image. The first frequency may be higher than the second frequency. In some embodiments, the first information may include features indicating an edge of a scene, and the second information may include features indicating a smooth area of ​​the scene. Figure 2 and Figure 3 As shown, the guided reconstruction module 308 of the decoder 201 in the processor 102 can be used to estimate the first information and the second information.

[0082] refer to Fig. 10A and Fig. 10BIn some embodiments, in operation 1002, a first component and a second component of the second image are separated from the second image. In some embodiments, the first component and the second component are separated using a DWT. In some embodiments, in operation 1004, an initial feature of the second image at a first frequency is extracted from the first component, and in operation 1006, an initial feature of the second image at a second frequency is extracted from the second component. Figure 5 and FIG. 6A to FIG. 6C As shown, the pre-processing unit 504 of the guided reconstruction module 308 can be used to separate the HF component and the LF component from the input color image I', and extract the initial HF feature and the initial LF feature from the HF component and the LF component respectively.

[0083] In some embodiments where the size of the first image is the same as the size of the second image, the first information is estimated based on the first component of the second image and the first component at the first frequency of the first image, and the second information is estimated based on the second component of the second image and the second component at the second frequency of the first image.

[0084] refer to Fig. 10A In some embodiments, in operation 1008, a first component and a second component of the first image are separated from the first image. In some embodiments, the first component and the second component are separated using a DWT. In some embodiments, in operation 1010, an initial feature of the first image at a first frequency is extracted from the first component, and in operation 1012, an initial feature of the first image at a second frequency is extracted from the second component. Figure 5 and Fig. 6A As shown, the preprocessing unit 505 of the guided reconstruction module 308 can be used to separate the HF component and the LF component from the input depth image D′ using a downsampling factor equal to 1, and extract initial HF features and initial LF features from the HF component and the LF component, respectively.

[0085] refer to Fig. 10A In some embodiments, in operation 1014, the initial features of the first image at the first frequency are combined with the initial features of the second image at the first frequency. Figure 5 and Fig. 6A As shown, the pre-processing unit 504 of the guided reconstruction module 308 may combine the initial HF features of the input color image I′ and the initial HF features of the input depth image D′.

[0086] refer to Fig. 10A In some embodiments, in operation 1016, a first learning model for the first frequency is used to estimate features at the first frequency based on initial features of a combination of the first image and the second image at the first frequency. Figure 5 and Fig. 6AAs shown, the HF estimation unit 506 of the guided reconstruction module 308 may be configured to estimate the HF features of the input depth image D' based on the initial HF features of the combined input color image I' and the input depth image D' using the HF machine learning model.

[0087] refer to Fig. 10A In some embodiments, in operation 1018, the initial features of the first image at the second frequency are combined with the initial features of the second image at the second frequency. Figure 5 and Fig. 6A As shown, the pre-processing unit 505 of the guided reconstruction module 308 may be used to combine the initial LF features of the input color image I' and the initial LF features of the input depth image D'.

[0088] refer to Fig. 10A In some embodiments, in operation 1020, a second learning model for the second frequency is used to estimate features of the second frequency based on initial features of a combination of the first image and the second image at the second frequency. Figure 5 and Fig. 6A As shown, the LF estimation unit 507 of the guided reconstruction module 308 may be configured to estimate the LF features of the input depth image D' based on the initial LF features of the combined input color image I' and the input depth image D' using a LF machine learning model.

[0089] In some embodiments where the size of the first image is smaller than the size of the second image, the first information is estimated based on the first component of the second image and the first image, and the second information is estimated based on the second component of the second image and the first image.

[0090] refer to Fig. 10B In some embodiments, in operation 1009, initial features of the first image are extracted. Figure 5 , Figure 6B ,as well as Figure 6C As shown, the pre-processing unit 505 of the guided reconstruction module 308 can be used to obtain the down-sampled input depth image D from the down-sampled factor equal to 2 or 4. s ' Extract the initial features.

[0091] refer to Fig. 10B In some embodiments, in operation 1011, the initial features of the second image at the first frequency are combined with the initial features of the first image. Figure 5 , Figure 6B ,as well as Figure 6C As shown, the pre-processing unit 504 of the guided reconstruction module 308 can be used to combine the initial HF features of the input color image I' with the input depth image D s ' is combined with the initial features.

[0092] refer to Fig. 10B In some embodiments, in operation 1013, a first learning model for the first frequency is used to estimate features at the first frequency based on initial features of a combination of the first image and the second image. Figure 5 , Figure 6B ,as well as Figure 6C As shown, the HF estimation unit 506 of the guided reconstruction module 308 can be used to use the HF machine learning model based on the initial HF features of the combined input color image I' and the input depth image D s 'The initial features of the estimated input depth image D s 'HF characteristics.

[0093] refer to Fig. 10B In some embodiments, in operation 1015, the initial features of the second image at the second frequency are combined with the initial features of the first image. Figure 5 , Figure 6B ,as well as Figure 6C As shown, the pre-processing unit 505 of the guided reconstruction module 308 can be used to combine the initial LF features of the input color image I' with the input depth image D s ' is combined with the initial features.

[0094] refer to Fig. 10B In some embodiments, in operation 1017, a second learning model for the second frequency is used to estimate features at the second frequency based on initial features of a combination of the first image and the second image. Figure 5 , Figure 6B ,as well as Figure 6C As shown, the LF estimation unit 507 of the guided reconstruction module 308 can be used to use the LF machine learning model based on the initial LF features of the combined input color image I' and the input depth image D s 'The initial features of the estimated input depth image D s 'LF feature.

[0095] Optionally, in some embodiments, before combining the initial features in operations 1011 and 1015, for example, in operation 1007, the second image, the first component and the second component of the second image, or the initial features of the second image are downsampled based on the size of the first image. Figure 5 and Figure 6C As shown, the pre-processing unit 504 of the guided reconstruction module 308 can be used to downsample the input depth image D s When the size of ' is one-fourth of the size of the input color image I', the input color image I' is downsampled so that the size of the initial HF / LF feature of the input color image I' becomes the same as the size of the input depth image Ds It is understandable that the preprocessing unit 504 of the guided reconstruction module 308 can be used to downsample the input color image I', the separated HF components (HH, HL, LH) and LF components (LL) of the input color image I', or the extracted HF features and LF features of the input color image I'.

[0096] Reference again Fig. 9 In operation 908, the second processor reconstructs the first image based on the first information and the second information. Figure 2 and Figure 3 As shown, the guided reconstruction module 308 of the decoder 201 in the processor 102 may be used to reconstruct the first image based on the first information and the second information.

[0097] refer to Fig. 10A and Fig. 10B In some embodiments, to reconstruct the first image, in operation 1022, the estimated first information and the estimated second information are combined, and in operation 1024, the first image is upsampled based on the combined first information and second information (e.g., using a third machine learning model for upsampling). Figure 5 and FIG. 6A to FIG. 6C As shown, the reconstruction unit 508 of the guided reconstruction module 308 can be used to transform the input depth image D' / D s The estimated HF features and LF features of ' are combined, and based on the combined HF and LF features, an upsampling machine learning model is used to upsample the input depth image D' / D s 'Upsamples to produce an output depth image D' of the same size as the input color image I'.

[0098] Fig.11 1 shows a block diagram of an exemplary model training system 1100 according to some embodiments of the present application. The system 1100 can be used to train various machine learning models described herein, such as the above Figures 6A to 6C The HF machine learning model, LF machine learning model, and upsampling machine learning model described in the embodiment of the present invention are described in the embodiment of the present invention. In different examples, the system 1100 can jointly train each machine learning model as a single model, or can train each machine learning model as an independent model. The system 1100 can be implemented by the encoding system 100, the decoding system 200, or a separate computing system.

[0099] like Fig.11As shown, the system 1100 may include a model training module 1102, which is used to train each CNN model 1101 through a set of training samples 1104 using a training algorithm 1108 based on a loss function 1106. The CNN model 1101 may include the above Figures 6A to 6C The HF machine learning model, LF machine learning model, and upsampling machine learning model are described in detail.

[0100] In some embodiments, Fig.12 As shown, each training sample 1104 includes an input color image 1202 of a scene, an input depth image 1204 of the scene, and a ground truth (GT) depth image 1206 of the scene. The training may be supervised training using the GT depth image 1206 in each training sample 1104. In some embodiments, the input color image 1202 and the input depth image 1204 in each training sample 1104 are compressed. In other words, the input color image 1202 and the input depth image 1204 may be compressed and decompressed using the same codec used by the encoding / decoding system that will use the CNN model 1101. In some embodiments, the input color image 1202 and the input depth image 1204 in each training sample 1104 are compressed based on a QP. In other words, the input color image 1202 and the input depth image 1204 may be compressed and decompressed based on the same QP or similar QP (e.g., in the same set of QPs) used by the encoding / decoding system that will use the CNN model 1101.

[0101] Reference again Fig.11 , the CNN model 1101 may include multiple parameters, which may be jointly adjusted by the model training module 1102 when the training sample 1104 is input. The model training module 1102 may use a training algorithm 1108 to jointly adjust the parameters of the CNN model 1101 through the training sample 1104 to minimize the loss function 1106. The training algorithm 1108 may be any iterative optimization algorithm suitable for finding the minimum value of the loss function 1106, including a gradient descent algorithm (e.g., a stochastic gradient descent algorithm).

[0102] like Fig.12As shown, in some embodiments, for each training sample 1104, the model training module 1102 is used to use the CNN model 1101 to upsample the input depth image 1204 under the guidance of the HF component from the input color image 1202 to reconstruct the output depth image 1208. In some embodiments, for each training sample 1104, the model training module 1102 is also used to train the CNN model 1101 based on the difference between each output depth image 1208 and the corresponding GT depth image 1206 using a loss function 1106. That is, the loss between the output depth image 1208 and the GT depth image 1206 can be calculated. For example, the loss function 1106 can be as follows, subtracting the L1 loss L1 and the structural similarity index (SSIM) loss L SSIM Combined with different weights to keep as much structure and detail as possible: Loss r (x,y)=L1(x,y)+w×L SSIM (x,y) (2), x and y are the output depth image 1208 and the corresponding GT depth image 1206, respectively, and w is the weight of the SSIM loss. L1 and L2 can be defined as follows: SSIM : L1(x,y)=|xy| (3), L SSIM (x,y)=[l(x,y)] α [c(x,y)] β [s(x,y)] γ (4) As shown below, SSIM compares brightness, contrast, and structure simultaneously: Brightness part: Contrast section: Structural part: μ x and μ y are the means of x and y respectively; and is the variance of x and y; σ xy is the covariance of x and y; c1=(k1L) 2 and c2=(k2L) 2is a constant, and c3=c2 / 2, L is the range of pixel values. In some embodiments, since pixel values ​​are more important for the residual map, L1 loss can be assigned a greater weight than SSIM loss. For example, the weight w of SSIM loss can be less than 1, such as 0.05, k1 can be equal to 0.01, and k2 can be equal to 0.03.

[0103] In various aspects of the present application, the functions described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as instructions on a non-transitory computer-readable medium. Computer-readable media include computer storage media. Storage media may be a computer program that can be executed by a processor (e.g., Figure 1 and Figure 2 Any available medium accessed by the processor 102 in the computer. As an example and not limitation, such computer-readable media may include random-access memory (RAM), read-only memory (ROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), or other optical disc storage, hard disk drive (HDD) (such as disk storage or other magnetic storage device), flash drive, solid-state disk (SSD), or any other medium that can be used to carry or store the required program code in the form of instructions or data structures and can be accessed by a processing system (such as a mobile device or computer). Disks and optical discs used in this article include CDs, laser optical discs, optical discs, digital video discs (DVD), and floppy disks, wherein disks generally reproduce data magnetically, while optical discs use lasers to reproduce data optically. Combinations of the above should also be included in the scope of computer-readable media.

[0104] According to one aspect of the present application, a method is disclosed. A processor receives a first image and a second image of the same scene. The first image and the second image are acquired by two different sensors. The processor estimates first information at a first frequency of the first image based at least in part on a first component at a first frequency of the second image. The processor estimates second information at a second frequency of the first image based at least in part on a second component at a second frequency of the second image. The processor reconstructs the first image based on the first information and the second information.

[0105] In some embodiments, the first image and the second image are complementary to each other with respect to the scene.

[0106] In some embodiments, the first image is a depth image and the second image is a color image.

[0107] In some embodiments, the first frequency is higher than the second frequency. In some embodiments, the first information includes features indicating an edge of the scene and the second information includes features indicating a smooth area of ​​the scene.

[0108] In some embodiments, a first component and a second component of the second image are separated from the second image, an initial feature of the second image at a first frequency is extracted from the first component, and an initial feature of the second image at a second frequency is extracted from the second component.

[0109] In some embodiments, the first component and the second component are extracted using a DWT.

[0110] In some embodiments, the size of the first image is the same as the size of the second image. In some embodiments, to estimate the first information, the first information is estimated based on the first component of the second image and the first component at the first frequency of the first image. In some embodiments, to estimate the second information, the second information is estimated based on the second component of the second image and the second component at the second frequency of the first image.

[0111] In some embodiments, a first component and a second component of the first image are separated from the first image, initial features of the first image at a first frequency are extracted from the first component, the initial features of the first image at the first frequency and the initial features of the second image at the first frequency are combined, the initial features of the first image at the second frequency are extracted from the first component, and the initial features of the first image at the second frequency and the initial features of the second image at the second frequency are combined.

[0112] In some embodiments, to estimate the first information, a first learning model for the first frequency is used to estimate the features at the first frequency based on the initial features of the combination of the first image and the second image at the first frequency. In some embodiments, to estimate the second information, a first learning model for the first frequency is used to estimate the features at the second frequency based on the initial features of the combination of the first image and the second image at the second frequency.

[0113] In some embodiments, the size of the first image is smaller than the size of the second image. In some embodiments, to estimate the first information, the first information is estimated based on the first component of the second image and the first image. In some embodiments, to estimate the second information, the second information is estimated based on the second component of the second image and the first image.

[0114] In some embodiments, initial features of the first image are extracted, initial features of the second image at a first frequency are combined with initial features of the first image, and initial features of the second image at a second frequency are combined with initial features of the first image.

[0115] In some embodiments, before the initial features are combined, the second image, the first component and the second component of the second image, or the initial features of the second image are downsampled based on the size of the first image.

[0116] In some embodiments, to estimate the first information, a first learning model for the first frequency is used to estimate the features at the first frequency based on initial features of the combination of the first image and the second image. In some embodiments, to estimate the second information, a second learning model for the second frequency is used to estimate the features at the second frequency based on initial features of the combination of the first image and the second image.

[0117] In some embodiments, to reconstruct the first image, the estimated first information is combined with the estimated second information, and the first image is upsampled based on the combined first information and second information.

[0118] In some embodiments, to upsample the first image, the first image is upsampled based on the combined first information and second information using a third learning model for upsampling.

[0119] According to another aspect of the present application, a system includes a memory for storing instructions and a processor coupled to the memory. The processor is used to receive a first image and a second image of the same scene when executing the instructions. The first image and the second image are acquired by two different sensors. The processor is also used to estimate the first information at the first frequency of the first image based at least in part on the first component at the first frequency of the second image when executing the instructions, and to estimate the second information at the second frequency of the first image based at least in part on the second component at the second frequency of the second image when executing the instructions. The processor is also used to reconstruct the first image based on the first information and the second information when executing the instructions.

[0120] In some embodiments, the first image and the second image are complementary to each other with respect to the scene.

[0121] In some embodiments, the first image is a depth image and the second image is a color image.

[0122] In some embodiments, the first frequency is higher than the second frequency. In some embodiments, the first information includes features indicating an edge of the scene and the second information includes features indicating a smooth area of ​​the scene.

[0123] In some embodiments, the processor is further used to separate a first component and a second component of the second image from the second image, extract initial features of the second image at a first frequency from the first component, and extract initial features of the second image at a second frequency from the second component.

[0124] In some embodiments, the first component and the second component are separated using a DWT.

[0125] In some embodiments, the size of the first image is the same as the size of the second image. In some embodiments, to estimate the first information, the processor is further configured to estimate the first information based on the first component of the second image and the first component at the first frequency of the first image. In some embodiments, to estimate the second information, the processor is further configured to estimate the second information based on the second component of the second image and the second component at the second frequency of the first image.

[0126] In some embodiments, the processor is also used to separate a first component and a second component of the first image from the first image, extract initial features of the first image at a first frequency from the first component, combine the initial features of the first image at the first frequency with the initial features of the second image at the first frequency, extract the initial features of the first image at the second frequency from the first component, and combine the initial features of the first image at the second frequency with the initial features of the second image at the second frequency.

[0127] In some embodiments, in order to estimate the first information, the processor is further used to use a first learning model for the first frequency to estimate the features at the first frequency based on the initial features of the combination of the first image and the second image at the first frequency. In some embodiments, in order to estimate the second information, the processor is further used to use a second learning model for the second frequency to estimate the features at the second frequency based on the initial features of the combination of the first image and the second image at the second frequency.

[0128] In some embodiments, the size of the first image is smaller than the size of the second image. In some embodiments, to estimate the first information, the processor is further configured to estimate the first information based on the first component of the second image and the first image. In some embodiments, to estimate the second information, the processor is further configured to estimate the second information based on the second component of the second image and the first image.

[0129] In some embodiments, the processor is further used to extract initial features of the first image, combine the initial features of the second image at the first frequency with the initial features of the first image, and combine the initial features of the second image at the second frequency with the initial features of the first image.

[0130] In some embodiments, the processor is further configured to downsample the second image, the first component and the second component of the second image, or the initial features of the second image based on a size of the second image before combining the initial features.

[0131] In some embodiments, in order to estimate the first information, the processor is further used to use a first learning model for the first frequency to estimate the features at the first frequency based on the initial features of the combination of the first image and the second image. In some embodiments, in order to estimate the second information, the processor is further used to use a second learning model for the second frequency to estimate the features at the second frequency based on the initial features of the combination of the first image and the second image.

[0132] In some embodiments, in order to reconstruct the first image, the processor is further configured to combine the estimated first information with the estimated second information, and upsample the first image based on the combined first information and second information.

[0133] In some embodiments, to upsample the first image, the processor is further configured to use a third learning model for upsampling to upsample the first image based on the combined first information and second information.

[0134] According to another aspect of the present application, a method is disclosed. A first sensor acquires a first image of a scene. A second sensor acquires a second image of the scene. A first processor estimates first information at a first frequency of the first image based at least in part on a first component at a first frequency of the second image. The first processor estimates second information at a second frequency of the first image based at least in part on a second component at a second frequency of the second image. The first processor reconstructs the first image based on the first information and the second information.

[0135] In some embodiments, the second processor compresses the first image and the second image into a bitstream, sends the bitstream from the second processor to the first processor, and the first processor decompresses the bitstream to receive the first image and the second image.

[0136] In some embodiments, the size of the first image is the same as the size of the second image. In some embodiments, to estimate the first information, the first information is estimated based on the first component of the second image and the first image. In some embodiments, to estimate the second information, the second information is estimated based on the second component of the second image and the first image.

[0137] In some embodiments, the second processor downsamples the first image prior to compression such that a size of the first image becomes smaller than a size of the second image.

[0138] In some embodiments, to estimate the first information, the first information is estimated based on the first component of the second image and the first image. In some embodiments, to estimate the second information, the second information is estimated based on the second component of the second image and the first image.

[0139] According to another aspect of the present application, a system includes an encoding subsystem and a decoding subsystem. The encoding subsystem includes a first sensor, a second sensor, a first memory for storing instructions, and a first processor coupled to the first memory and the first sensor and the second sensor. The first sensor is used to acquire a first image of a scene. The second sensor is used to acquire a second image of the scene. The decoding subsystem includes a second memory for storing instructions and a second processor coupled to the second memory. The second processor is used to estimate the first information at the first frequency of the first image based at least in part on the first component at the first frequency of the second image when executing the instructions, and to estimate the second information at the second frequency of the first image based at least in part on the second component at the second frequency of the second image. The second processor is also used to reconstruct the first image based on the first information and the second information when executing the instructions.

[0140] In some embodiments, the first processor of the encoding subsystem is used to compress the first image and the second image into a bitstream. In some embodiments, the encoding subsystem also includes a first interface, and the first interface is used to send the bitstream to the decoding subsystem. In some embodiments, the decoding subsystem also includes a second interface, and the second interface is used to receive the bitstream from the encoding subsystem. In some embodiments, the second processor of the decoding subsystem is also used to decompress the bitstream to receive the first image and the second image.

[0141] In some embodiments, the size of the first image is the same as the size of the second image. In some embodiments, to estimate the first information, the second processor is further configured to estimate the first information based on the first component of the second image and the first component at the first frequency of the first image. In some embodiments, to estimate the second information, the second processor is further configured to estimate the second information based on the second component of the second image and the second component at the second frequency of the first image.

[0142] In some embodiments, the first processor is further configured to downsample the first image before compression so that a size of the first image becomes smaller than a size of the second image.

[0143] In some embodiments, to estimate the first information, the second processor is further configured to estimate the first information based on the first component of the second image and the first image. In some embodiments, to estimate the second information, the second processor is further configured to estimate the second information based on the second component of the second image and the first image.

[0144] The description of the above-mentioned embodiments discloses the general nature of the present application, and others can easily modify and / or adjust these embodiments for various applications without departing from the general concept of the present application by applying knowledge within the technical scope of the art, without excessive experimentation. Therefore, based on the teachings and guidance provided herein, such adjustments and modifications belong to the meaning and scope of the equivalents of the embodiments of the present application. It is understood that the wording or terminology herein is for description rather than for limitation, and therefore, those skilled in the art should interpret the terms or wording of this specification according to the teachings and guidance.

[0145] The above describes the embodiments of the present application with the help of functional building blocks, which illustrate the implementation of the specific functions and relationships of the embodiments of the present application. For the convenience of description, the boundaries of these building blocks are arbitrarily defined herein. As long as its specific functions and relationships are properly performed, other boundaries may also be defined.

[0146] The Summary and Abstract sections may set forth one or more but not all exemplary embodiments of the present application as contemplated by the inventor(s), and thus, are not intended to limit the present application and the appended claims in any way.

[0147] Various functional blocks, modules, and steps are disclosed above. The arrangement provided is illustrative and not restrictive. Therefore, the functional blocks, modules, and steps can be reordered or combined in a manner different from the examples provided above. Likewise, some embodiments include only a subset of functional blocks, modules, and steps, and any such subset may exist.

[0148] The breadth and scope of the present application should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A method comprising: The processor receives a first image and a second image of the same scene, wherein the first image and the second image are acquired by two different sensors; The processor estimates first information of the first image at the first frequency based at least in part on a first component of the second image at the first frequency; The processor estimates second information of the first image at a second frequency based at least in part on a second component of the second image at the second frequency; and The processor reconstructs the first image based on the first information and the second information.

2. The method according to claim 1, wherein: The first image and the second image are complementary to each other with respect to the scene.

3. The method according to claim 1, wherein: The first image is a depth image, and the second image is a color image.

4. The method according to claim 3, wherein The first frequency is higher than the second frequency; and The first information includes features indicating edges of the scene, and the second information includes features indicating smooth areas of the scene.

5. The method according to claim 1, further comprising: separating the first component and the second component of the second image from the second image; extracting initial features of the second image at the first frequency from the first component; as well as An initial feature of the second image at the second frequency is extracted from the second component.

6. The method according to claim 5, wherein: The first component and the second component are separated using a discrete wavelet transform (DWT).

7. The method according to claim 5, wherein The size of the first image is the same as the size of the second image; Estimating the first information comprises estimating the first information based on the first component of the second image and a first component of the first image at the first frequency; and Estimating the second information includes estimating the second information based on the second component of the second image and a second component at the second frequency of the first image.

8. The method according to claim 7, further comprising: separating the first component and the second component of the first image from the first image; extracting initial features of the first image at the first frequency from the first component; combining the initial features of the first image at the first frequency with the initial features of the second image at the first frequency; extracting initial features of the first image at the second frequency from the first component; as well as The initial features of the first image at the second frequency and the initial features of the second image at the second frequency are combined.

9. The method according to claim 8, wherein Estimating the first information includes estimating features at the first frequency based on the initial features of a combination of the first image and the second image at the first frequency using a first learning model for the first frequency; and Estimating the second information includes estimating features at the second frequency based on the initial features of a combination of the first image and the second image at the second frequency using a second learning model for the second frequency.

10. The method according to claim 5, wherein The size of the first image is smaller than the size of the second image; estimating the first information comprises estimating the first information based on the first component of the second image and the first image; and Estimating the second information includes estimating the second information based on the second component of the second image and the first image.

11. The method according to claim 10, further comprising: extracting initial features of the first image; combining the initial features of the second image at the first frequency with the initial features of the first image; as well as The initial features of the second image at the second frequency are combined with the initial features of the first image.

12. The method according to claim 11, further comprising: Prior to combining the initial features, the second image, the first component and the second component of the second image, or the initial features of the second image are downsampled based on the size of the first image.

13. The method according to claim 11, wherein Estimating the first information includes estimating features at the first frequency based on the initial features of a combination of the first image and the second image using a first learning model for the first frequency; and Estimating the second information includes estimating features at the second frequency based on the initial features of a combination of the first image and the second image using a second learning model for the second frequency.

14. The method according to claim 1, wherein: Reconstructing the first image comprises: combining the estimated first information with the estimated second information; and The first image is upsampled based on the combined first information and the second information.

15. The method according to claim 14, wherein: Upsampling the first image includes upsampling the first image based on the combined first information and the second information using a third learned model for upsampling.

16. A system comprising: A memory for storing instructions; as well as A processor, coupled to the memory, and configured to, when executing the instructions: Receiving a first image and a second image of the same scene, wherein the first image and the second image are acquired by two different sensors; estimating first information of the first image at the first frequency based at least in part on a first component of the second image at the first frequency; estimating second information of the first image at the second frequency based at least in part on a second component of the second image at the second frequency; and The first image is reconstructed based on the first information and the second information.

17. The system of claim 16, wherein: The first image and the second image are complementary to each other with respect to the scene.

18. The system of claim 16, wherein: The first image is a depth image, and the second image is a color image.

19. The system of claim 18, wherein The first frequency is higher than the second frequency; and The first information includes features indicating edges of the scene, and the second information includes features indicating smooth areas of the scene.

20. The system of claim 16, wherein: The processor is further configured to: separating the first component and the second component of the second image from the second image; extracting initial features of the second image at the first frequency from the first component; as well as An initial feature of the second image at the second frequency is extracted from the second component.

21. The system of claim 20, wherein: The first component and the second component are separated using a discrete wavelet transform (DWT).

22. The system of claim 20, wherein The size of the first image is the same as the size of the second image; To estimate the first information, the processor is further configured to estimate the first information based on the first component of the second image and a first component of the first image at the first frequency; and In order to estimate the second information, the processor is further configured to estimate the second information based on the second component of the second image and the second component of the first image at the second frequency.

23. The system of claim 22, wherein: The processor is further configured to: separating the first component and the second component of the first image from the first image; extracting initial features of the first image at the first frequency from the first component; combining the initial features of the first image at the first frequency with the initial features of the second image at the first frequency; extracting initial features of the first image at the second frequency from the first component; as well as The initial features of the first image at the second frequency and the initial features of the second image at the second frequency are combined.

24. The system of claim 23, wherein To estimate the first information, the processor is further configured to estimate features at the first frequency based on the initial features of a combination of the first image and the second image at the first frequency using a first learning model for the first frequency; and In order to estimate the second information, the processor is further configured to estimate features at the second frequency based on the initial features of a combination of the first image and the second image at the second frequency using a second learning model for the second frequency.

25. The system of claim 20, wherein The size of the first image is smaller than the size of the second image; To estimate the first information, the processor is further configured to estimate the first information based on the first component of the second image and the first image; and To estimate the second information, the processor is further configured to estimate the second information based on the second component of the second image and the first image.

26. The system of claim 25, wherein: The processor is further configured to: extracting initial features of the first image; combining the initial features of the second image at the first frequency with the initial features of the first image; as well as The initial features of the second image at the second frequency are combined with the initial features of the first image.

27. The system of claim 26, wherein: The processor is further configured to downsample the second image, the first component and the second component of the second image, or the initial features of the second image based on the size of the first image before combining the initial features.

28. The system of claim 26, wherein To estimate the first information, the processor is further configured to estimate features at the first frequency based on the initial features of a combination of the first image and the second image using a first learning model for the first frequency; and In order to estimate the second information, the processor is further configured to estimate features at the second frequency based on the initial features of a combination of the first image and the second image using a second learning model for the second frequency.

29. The system of claim 16, wherein: In order to reconstruct the first image, the processor is further configured to: combining the estimated first information with the estimated second information; and The first image is upsampled based on the combined first information and the second information.

30. The system of claim 29, wherein: In order to upsample the first image, the processor is further configured to upsample the first image based on the combined first information and the second information using a third learning model for upsampling.

31. A method comprising: The first sensor acquires a first image of a scene; The second sensor acquires a second image of the scene; A first processor estimates first information of the first image at the first frequency based at least in part on a first component of the second image at the first frequency; The first processor estimates second information of the first image at the second frequency based at least in part on a second component of the second image at the second frequency; as well as The first processor reconstructs the first image based on the first information and the second information.

32. The method of claim 31 further comprising: a second processor compressing the first image and the second image into a bitstream; sending the bitstream from the second processor to the first processor; as well as The first processor decompresses the bit stream to receive the first image and the second image.

33. The method of claim 32, wherein The size of the first image is the same as the size of the second image; Estimating the first information comprises estimating the first information based on the first component of the second image and a first component of the first image at the first frequency; and Estimating the second information includes estimating the second information based on the second component of the second image and a second component at the second frequency of the first image.

34. The method of claim 32, further comprising: The second processor downsamples the first image before compression so that a size of the first image becomes smaller than a size of the second image.

35. The method of claim 34, wherein estimating the first information comprises estimating the first information based on the first component of the second image and the first image; and Estimating the second information includes estimating the second information based on the second component of the second image and the first image.

36. A system comprising: Coding subsystem, including: A first sensor, configured to acquire a first image of a scene; a second sensor, configured to acquire a second image of the scene; A first memory for storing instructions; and a first processor coupled to the first memory and the first sensor and the second sensor; Decoding subsystem, including: A second memory for storing instructions; and a second processor, coupled to the second memory, and configured to, when executing the instructions: estimating first information of the first image at the first frequency based at least in part on a first component of the second image at the first frequency; estimating second information of the first image at the second frequency based at least in part on a second component of the second image at the second frequency; and The first image is reconstructed based on the first information and the second information.

37. The system of claim 36, wherein The first processor of the encoding subsystem is used to compress the first image and the second image into a bitstream; The encoding subsystem further comprises a first interface, wherein the first interface is used to send the bit stream to the decoding subsystem; The decoding subsystem further comprises a second interface for receiving the bit stream from the encoding subsystem; and The second processor of the decoding subsystem is further configured to decompress the bitstream to receive the first image and the second image.

38. The system of claim 37, wherein The size of the first image is the same as the size of the second image; To estimate the first information, the second processor is further configured to estimate the first information based on the first component of the second image and a first component of the first image at the first frequency; and In order to estimate the second information, the second processor is further configured to estimate the second information based on the second component of the second image and the second component of the first image at the second frequency.

39. The system of claim 37, wherein: The first processor is further configured to downsample the first image before compression so that a size of the first image becomes smaller than a size of the second image.

40. The system of claim 39, wherein To estimate the first information, the second processor is further configured to estimate the first information based on the first component of the second image and the first image; and To estimate the second information, the second processor is further configured to estimate the second information based on the second component of the second image and the first image.