Checking joint demosaicking and denoising for single, four and nine bayer patterns

Abstract

The present disclosure provides methods, devices, systems, and computer-readable media for de-mosaicking images by a device. A method includes obtaining a plurality of mosaicked images from a plurality of sensors of a device, concatenating each image of the plurality of mosaicked images with an encoded embedding of a respective mosaicking pattern of a sensor of the plurality of sensors that captured the each image of the plurality of mosaicked images, providing the concatenated plurality of mosaicked images to a machine learning model, and obtaining a plurality of de-mosaicked images corresponding to the plurality of mosaicked images from the machine learning model. Each sensor of the plurality of sensors has a respective mosaicking pattern from a plurality of mosaicking patterns.

Description

Technical Field

[0001] This disclosure relates generally to image processing, and more specifically to methods, apparatus, systems, and non-transitory computer-readable media for joint demosaicing and denoising of mosaic images captured using a mosaic layout of multiple sensors. Background Technology

[0002] Recent advancements in both camera hardware and image processing software have transformed modern mobile devices (e.g., smartphones, cellular phones, tablets, digital cameras, personal digital assistants (PDAs), wearable devices, etc.) into powerful and portable image and / or video capture and / or recording devices. Therefore, mobile devices can incorporate multiple imaging sensors (e.g., still cameras, action cameras, camcorders, etc.). Furthermore, depending on design constraints and / or desired functionality, each imaging sensor mounted on a mobile device can be manufactured with a different hardware configuration. For example, multiple imaging sensors in a mobile device can have a color filter array (CFA) located in front of the imaging sensors, where each pixel of the imaging sensor is placed behind a single color filter. In such an example, the color filters of the CFA can be arranged in different mosaic (e.g., repeating) layout patterns.

[0003] To obtain a panchromatic image (e.g., a red-green-blue (RGB) image) from an imaging sensor, the mobile device may need to de-mosaic the mosaic image (e.g., raw image data) output by the imaging sensor. De-mosaic can refer to de-mosaicing based on the color filter mosaic layout of the imaging sensor (e.g., Bayer pattern, etc.). Image processing steps to estimate panchromatic images (e.g., RGB images) using X-trans patterns, etc. That is, a mosaic image provided by an imaging sensor may not include the values ​​of all colors (e.g., red, green, and blue) for each pixel of the imaging sensor. Therefore, a mobile device may need to estimate the missing color information based on the CFA mosaic layout of the imaging sensor. For example, a relevant mobile device could employ a simple algorithm to estimate (e.g., interpolate) the missing color values ​​based on the color values ​​of neighboring pixels. As another example, a relevant mobile device could employ a machine learning model to generate a panchromatic image from the mosaic image.

[0004] However, such methods may be specific to a particular mosaic pattern and therefore may require multiple algorithms and / or models to support more than one mosaic pattern. Consequently, these methods may not scale easily, and the resource overhead (e.g., memory footprint, processing load, etc.) incurred by having custom algorithms and / or models for each mosaic pattern, as well as the time and / or processing overhead for switching between different algorithms and / or models, may be prohibitively high, so these methods may not be able to support multiple imaging sensors with different mosaic patterns.

[0005] Therefore, since the need to perform demosaicing and / or denoising on mosaic images captured using multiple mosaic layouts can be constrained by resources, time, and / or processing overhead, there is a need for further improvements to image processing techniques. This paper proposes improvements. These improvements are also applicable to other image processing techniques and image processing standards employing these techniques. Summary of the Invention

[0006] Technical solution The following presents a simplified overview of one or more embodiments of this disclosure to provide a basic understanding of these embodiments. This summary is not a broad overview of all contemplated embodiments and is neither intended to identify key or essential elements of all embodiments nor to define the scope of any or all embodiments. The sole purpose of this summary is to present, in a simplified form, some ideas of one or more embodiments of this disclosure as a prelude to the more detailed description that follows.

[0007] This disclosure discloses methods, apparatus, systems, and non-transitory computer-readable media for jointly de-mosaicing and denoising mosaic images captured using a mosaic layout of multiple sensors.

[0008] According to one aspect of this disclosure, a method for demosaicing an image by a device may include: obtaining a plurality of mosaic images from a plurality of sensors of the device. The method may include: concatenating each of the plurality of mosaic images with an encoded embedding of a corresponding mosaic pattern from a sensor among the plurality of sensors that captured each of the plurality of mosaic images. The method may include: providing the concatenated plurality of mosaic images to a machine learning model. The method may include: obtaining a plurality of demosaic images corresponding to the plurality of mosaic images from the machine learning model. Each of the plurality of sensors has a corresponding mosaic pattern from a plurality of mosaic patterns.

[0009] According to one aspect of this disclosure, an apparatus for demosaicing an image may include: a plurality of sensors; a memory storing instructions; and one or more processors communicatively coupled to the plurality of sensors and the memory. Each of the plurality of sensors has a corresponding mosaic pattern from a plurality of mosaic patterns. The one or more processors may be configured to execute the instructions to obtain a plurality of mosaic images from the plurality of sensors. The one or more processors may be configured to concatenate each of the plurality of mosaic images with an encoded embedding of the corresponding mosaic pattern of the sensor among the plurality of sensors that captured each of the plurality of mosaic images. The one or more processors may be configured to provide the concatenated plurality of mosaic images to a machine learning model. The one or more processors may be configured to obtain a plurality of demosaic images corresponding to the plurality of mosaic images from the machine learning model.

[0010] According to one aspect of this disclosure, an apparatus for demosaicing an image includes: means for acquiring a plurality of mosaic images from a plurality of sensors of the apparatus; means for concatenating each of the plurality of mosaic images with encoded embeddings of a corresponding mosaic pattern from a sensor among the plurality of sensors that captured each of the plurality of mosaic images; means for providing the concatenated plurality of mosaic images to a machine learning model; and means for acquiring a plurality of demosaic images corresponding to the plurality of mosaic images from the machine learning model. Each of the plurality of sensors has a corresponding mosaic pattern from the plurality of mosaic patterns.

[0011] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided for storing computer-executable instructions for demosaicing images by a device. When executed by at least one processor of the device, the computer-executable instructions cause the device to acquire a plurality of mosaic images from a plurality of sensors of the device. When executed by the at least one processor of the device, the computer-executable instructions cause the device to concatenate each of the plurality of mosaic images with an encoded embedding of a corresponding mosaic pattern from a sensor among the plurality of sensors that captured each of the plurality of mosaic images. When executed by the at least one processor of the device, the computer-executable instructions cause the device to provide the concatenated plurality of mosaic images to a machine learning model. When executed by the at least one processor of the device, the computer-executable instructions cause the device to acquire a plurality of demosaic images corresponding to the plurality of mosaic images from the machine learning model. Each of the plurality of sensors has a corresponding mosaic pattern from a plurality of mosaic patterns.

[0012] According to one aspect of this disclosure, a computer-readable storage medium for storing instructions is provided. When executed by at least one processor, the instructions cause the at least one processor to perform a corresponding method.

[0013] Other aspects are set forth in part in the description which follows, and will be apparent in part from the description, or may be learned by practicing the embodiments presented in this disclosure. Attached Figure Description

[0014] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, wherein: Figure 1 Examples of apparatuses that can be used to implement one or more aspects of this disclosure are shown; Figure 2A An example depicting a color filter array (CFA) according to one aspect of this disclosure; Figure 2B An example of demosaic processing according to one aspect of this disclosure is shown; Figure 2C An example depicting a quad-Bayer pattern of a CFA according to one aspect of this disclosure; Figure 2D An example of a Nona-Bayer pattern for a CFA according to one aspect of this disclosure is shown; Figure 3 An example of a mobile device having multiple cameras according to one aspect of this disclosure is depicted; Figure 4 An example of a mobile device having multiple cameras according to one aspect of this disclosure is shown; Figure 5 A flowchart is depicted for selecting a demosaic model according to one aspect of this disclosure; Figure 6 A flowchart illustrating an example training phase of a unified demosaicing model according to one aspect of this disclosure is shown. Figure 7 A flowchart depicting an example reasoning phase of a unified demosaic model according to one aspect of this disclosure; Figure 8 A flowchart illustrating an example training phase of a unified demosaic model with mosaic masking according to one aspect of this disclosure is shown. Figure 9 A flowchart depicting an example inference stage of a unified demosaic model with mosaic masking according to one aspect of this disclosure; Figure 10A and Figure 10B An example of a training dataset for training a demosaic model is shown according to one aspect of this disclosure; Figure 11A An example depicting a nine-nona shuffling block according to a unified demosaic model based on one aspect of this disclosure; Figure 11B An example of a quad shuffling block according to a unified demosaic model is shown; Figure 11C An example of a color extraction head for a unified demosaic model according to one aspect of this disclosure is shown; Figure 11D An example of a modified color extraction head for a unified demosaic model according to one aspect of this disclosure is shown; Figure 12 An example of a re-mosaic unified model (SRUM) based on one aspect of this disclosure is described; Figure 13 An example of a latent spatial unification model (LSUM) according to one aspect of this disclosure is shown; Figure 14 An example of a modified joint denoising and demosaic model (M-JDNDM) according to one aspect of this disclosure is described; Figure 15A An example of a mobile device with mosaic concealment according to one aspect of this disclosure is shown; Figure 15B Examples depicting pattern information with mosaic masking according to one aspect of this disclosure; Figure 16 A block diagram of an example device for demosaicing an image is shown according to one aspect of this disclosure; Figure 17 A flowchart illustrating an example method for demosaicing an image according to one aspect of this disclosure is shown. Detailed Implementation

[0015] The detailed description that follows, taken in conjunction with the accompanying drawings, is intended to describe various configurations and not to represent configurations in which only the concepts described herein may be practiced. For the purpose of providing a thorough understanding of the various concepts, the detailed description includes specific details. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts. In the following description, the same parts are labeled with the same reference numerals throughout the specification and drawings.

[0016] The following description provides examples and does not limit the scope, applicability, or embodiments set forth in the claims. Changes may be made to the function and / or arrangement of the elements discussed without departing from the scope of this disclosure. Various procedures or components may be omitted, substituted, or added as appropriate in the various examples. For example, the described methods may be performed in a different order than that described, and various steps may be added, omitted, and / or combined. Optionally or additionally, features described with reference to some examples may be combined in other examples.

[0017] Various aspects and / or features may be presented according to the system, which may include multiple devices, components, modules, etc. It should be understood and recognized that various systems may include additional devices, components, modules, etc., and / or may exclude all devices, components, modules, etc. discussed in conjunction with the figures. Combinations of these methods may also be used.

[0018] As an overview of the topics described below in more detail, the aspects described herein relate to apparatus, methods, systems, and non-transitory computer-readable media for performing joint demosaicing and denoising on mosaic images captured using a mosaic layout of multiple sensors. Additionally, the aspects described herein can provide methods for correcting images for dead pixels in imaging sensors.

[0019] The aspects presented in this paper provide a machine learning model that can be jointly trained on an image dataset (e.g., raw and / or unprocessed image data) of mosaic images captured using multiple sensor mosaic layouts. The mosaic images can be cascaded with coded embedding information of the sensor mosaic layouts that indicate the mosaic images. The machine learning model can be configured to provide de-mosaicing and / or denoising images corresponding to the input mosaic image and the corresponding sensor mosaic layout. Advantageously, the memory footprint of a unified machine learning model can be significantly reduced by using a unified machine learning model to de-mosaic and denoise mosaic images from multiple sensor mosaic layouts, compared to the memory resource footprint required to configure a separate machine learning model for each sensor mosaic layout. Furthermore, the unified machine learning model can be trained to correct dead pixels in the sensor mosaic layouts, and thus provides improved de-mosaicing and / or denoising images compared to related de-mosaicing methods.

[0020] Although this disclosure describes machine learning models trained and / or configured to perform de-mosaicing, denoising, and / or dead pixel correction using mosaic images based on the red-green-blue (RGB) color space, this disclosure is not limited in this respect. For example, the ideas described herein can be applied to other color spaces (such as, but not limited to, standard RGB (sRGB), luminance-chromaticity (YCbCr), hue saturation value (HSV), International Commission on Illumination (CIE) 1931 RGB, CIE 1931 XYZ, etc.). As another example, without departing from the scope of this disclosure, the ideas described herein can be applied to electronic devices having imaging sensors utilizing more than three (3) different sensor mosaic layouts (e.g., four (4) or more mosaic layouts). As yet another example, the ideas described herein can be applied to having mosaic layouts (such as, but not limited to, 1×1 single-Bayer patterns, 2×2 four-Bayer patterns, 3×3 nine-Bayer patterns, Q×Q Bayer patterns, etc.). Imaging sensors (such as X-Trans patterns).

[0021] It is worth noting that the aspects presented in this paper can be applied to perform demosaicing, denoising, and / or dead pixel correction on mosaic images that have been captured using multiple sensor mosaic layouts, without requiring different custom methods to be configured for each sensor mosaic layout.

[0022] As mentioned above, this article discusses specific embodiments involving demosaicing of images. However, before discussing these concepts in more detail, [the following section discusses specific implementations related to image demosaicing]. Figure 1 Examples of computing devices that can be used to implement and / or otherwise provide aspects of this disclosure are discussed.

[0023] Figure 1Examples of apparatus 100 that can be used to implement one or more aspects of this disclosure are depicted according to one or more illustrative aspects discussed herein. For example, in some instances, apparatus 100 may implement one or more aspects of this disclosure by reading and / or executing instructions and accordingly performing one or more actions. In one or more arrangements, apparatus 100 may represent, be incorporated into and / or include a processor, a personal computer (PC), a printed circuit board (PCB) including a computing device, a minicomputer, a mainframe computer, a microcomputer, a telephone computing device, a wired / wireless computing device (e.g., a smartphone, a personal digital assistant (PDA)), a laptop computer, a tablet computer, a smart device, a wearable device or any other similar functional device. In embodiments, device 100 may represent, incorporate, and / or include desktop computers, computer servers, virtual machines, networked appliances, mobile devices (e.g., user devices (UEs), laptop computers, tablet computers, personal digital assistants (PDAs), smartphones, any other type of mobile computing device, etc.), cameras, wearable devices (e.g., smartwatches, headsets, headsets, etc.), smart devices (e.g., voice-controlled virtual assistants, set-top boxes (STBs), refrigerators, air conditioners, microwave ovens, televisions (TVs), etc.), Internet of Things (IoT) devices, and / or any other type of data processing device.

[0024] For example, device 100 may include a processor, a personal computer (PC), a printed circuit board (PCB) including a computing device, a minicomputer, a mainframe computer, a microcomputer, a telephone computing device (e.g., a cellular phone, a smartphone, a Session Initiation Protocol (SIP) phone), a wired / wireless computing device (e.g., a smartphone, a PDA), a laptop computer, a tablet computer, a smart device, a wearable device, or any other similar functional device.

[0025] In an embodiment, such as Figure 1 As shown, device 100 may include a collection of components such as processor 120, memory 130, storage component 140, input component 150, output component 160, communication interface 170, and demosaicing component 180. The collection of components of device 100 may be communicatively coupled via bus 110.

[0026] Bus 110 may include one or more components that enable communication between a set of components of device 100. For example, bus 110 may be a communication bus, crossbar, network, etc. Although bus 110 is in Figure 1 The bus 110 is shown as a single line, but it can be implemented using multiple (e.g., two (2) or more) connections between sets of components of the device 100. This disclosure is not limited in this respect.

[0027] Device 100 may include one or more processors, such as processor 120. Processor 120 may be implemented as hardware, firmware, and / or a combination of hardware and software. For example, processor 120 may include a central processing unit (CPU), application processor (AP), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), image signal processor (ISP), neural processing unit (NPU), sensor hub processor, communication processor (CP), artificial intelligence (AI) dedicated processor designed to have a hardware architecture specified for processing AI models, general-purpose single-chip and / or multi-chip processors, or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. General-purpose processors may include microprocessors, or any conventional processor, controller, microcontroller, or state machine.

[0028] Processor 120 may also be implemented as a combination of computing devices (such as a combination of a DSP and a microprocessor, a combination of a main processor and an auxiliary processor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). In embodiments, specific processing and methods may be performed by circuitry for a given function. In embodiments, the auxiliary processor may be configured to consume less power than the main processor. In embodiments, one or more processors may be implemented individually (e.g., as several different chips) and / or may be combined into a single form.

[0029] The processor 120 can control the overall operation of the device 100 and / or the set of components of the device 100 (e.g., memory 130, storage component 140, input component 150, output component 160, communication interface 170, and demosaic component 180).

[0030] Device 100 may include memory 130. In embodiments, memory 130 may include volatile memory (such as, but not limited to, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), etc.). In embodiments, memory 130 may include non-volatile memory (such as, but not limited to, read-only memory (ROM), electrically erasable programmable ROM (EEPROM), NAND flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), magnetic memory, optical memory, etc.). However, this disclosure is not limited in this respect, and memory 130 may include other types of dynamic and / or static memory storage. In embodiments, memory 130 may store information and / or instructions used (e.g., executed) by processor 120.

[0031] The storage component 140 of device 100 may store information and / or computer-readable instructions and / or code relating to the operation and use of device 100. For example, storage component 140 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact discs (CDs), digital universal discs (DVDs), universal serial bus (USB) flash drives, PCMCIA cards, floppy disks, cartridges, magnetic tapes, and / or other types of non-transitory computer-readable media, and corresponding drives.

[0032] Device 100 may include an input component 150. Input component 150 may include one or more components that allow device 100 to receive information, such as via user input (e.g., touchscreen, keyboard, keypad, mouse, stylus, button, switch, microphone, camera, virtual reality (VR) headset, haptic glove, etc.). In embodiments, input component 150 may include one or more sensors for sensing information (e.g., Global Positioning System (GPS) components, accelerometers, gyroscopes, actuators, transducers, contact sensors, proximity sensors, ranging devices, cameras, video cameras, depth cameras, time-of-flight (TOF) cameras, stereo cameras, etc.). In embodiments, input component 150 may include more than one of the same sensor type (e.g., multiple cameras). Additionally, multiple sensors of the same type may have multiple configurations that may differ from each other. For example, input component 150 may include multiple color filter array mosaic layouts that may have different color filters from each other (e.g., Bayer patterns, etc.). Multiple imaging sensors (e.g., cameras) for X-Trans patterns, etc.

[0033] The output component 160 of device 100 may include one or more components (e.g., display, liquid crystal display (LCD), light-emitting diode (LED), organic light-emitting diode (OLED), haptic feedback device, speaker, buzzer, alarm, etc.) that can provide output information from device 100.

[0034] Device 100 may include a communication interface 170. Communication interface 170 may include a receiver component, a transmitter component, and / or a transceiver component. Communication interface 170 enables device 100 to establish connections with other devices (e.g., a server, another device) and / or transmit communications. Communication may be implemented via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 170 may allow device 100 to receive information from and / or provide information to another device. In embodiments, communication interface 170 may provide communication with another device via networks such as local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), private networks, ad hoc networks, intranets, the Internet, fiber-optic networks, cellular networks (e.g., fifth-generation (5G) networks, long-term evolution (LTE) networks, third-generation (3G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMNs), telephone networks (e.g., public switched telephone networks (PSTNs)), and / or combinations of these or other types of networks. In this embodiment, the communication interface 170 may be connected via a device-to-device (D2D) communication link (such as FlashLinQ, WiMedia, etc.). , Low-power (BLE), ZigBee, IEEE 802.11x (Wi-Fi), LTE, 5G, etc., provide communication with another device. In embodiments, the communication interface 170 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a USB interface, an IEEE 1094 (FireWire) interface, etc.

[0035] In an embodiment, the apparatus 100 may include a demosaic component 180 configured to demosaic an image. For example, the demosaic component 180 may be configured to obtain a plurality of mosaic images, concatenate the plurality of mosaic images with an encoded embedding, provide the concatenated plurality of mosaic images to a machine learning model, and obtain a plurality of demosaic images corresponding to the plurality of mosaic images.

[0036] Apparatus 100 may perform one or more of the processes described herein. Apparatus 100 may perform operations based on processor 120 executing computer-readable instructions and / or code that may be stored in non-transitory computer-readable media, such as memory 130 and / or storage components 140. Computer-readable media may refer to non-transitory memory devices. Non-transitory memory devices may include memory space within a single physical storage device and / or memory space distributed across multiple physical storage devices.

[0037] Computer-readable instructions and / or code can be read from another computer-readable medium or from another device into memory 130 and / or storage component 140 via communication interface 170. The computer-readable instructions and / or code stored in memory 130 and / or storage component 140, if executed by processor 120 or when executed by processor 120, can cause device 100 to perform one or more of the processes described herein.

[0038] In the embodiments, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more of the processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0039] Figure 1 The number and arrangement of components shown are provided as an example. In reality, with... Figure 1 Compared to the components shown, there may be additional components, fewer components, different components, or components with different arrangements. Additionally, Figure 1 The two (2) or more components shown can be implemented within a single component, or Figure 1 The single component shown can be implemented as multiple distributed components. In the embodiment, Figure 1 The collection of (one or more) components shown is executable and described as being composed of Figure 1 Another set of components shown performs one or more functions.

[0040] Examples of apparatuses that can be used to provide and / or implement aspects of this disclosure have already been discussed, and several embodiments will now be discussed in further detail. Specifically, and as described above, some aspects of this disclosure generally relate to demosaicing of images.

[0041] Figure 2A An example of a color filter array (CFA) according to one aspect of this disclosure is depicted. Figure 2B An example of demosaic processing according to one aspect of this disclosure is shown. Figure 2C An example depicting a four-Bayer pattern of a CFA according to one aspect of this disclosure. Figure 2D An example of a nine-bayer pattern of the CFA according to one aspect of this disclosure is shown.

[0042] Reference Figure 2AThe CFA can be and / or may include a mosaic of color filters 220 (e.g., red filter 220R, green filter 220G, and blue filter 220B) that may be located in front of the imaging sensor 210 of the electronic device (e.g., device 100). That is, each pixel of the imaging sensor 210 may be placed behind a separate color filter 220, such that incident light reaching each pixel can pass through the color filter 220. In embodiments, the mosaic of the color filters 220 may be configured according to a repeating pattern of alternating colors. The mosaic layout of the CFA may be referred to as a Bayer pattern. For example, as... Figure 2A As shown, the CFA can be configured in a mosaic layout that may be referred to as a 1×1 single Bayer pattern and / or a Bayer pattern. Figure 2A As shown, a single Bayer pattern may include alternating red filters 220R and green filters 220G for odd-numbered rows of the imaging sensor 210, and alternating green filters 220G and blue filters 220B for even-numbered rows of the imaging sensor 210. Thus, a single Bayer pattern may also be referred to as an RGGB pattern.

[0043] In one embodiment, each pixel of the imaging sensor 210 may be located behind a corresponding color filter 220, and therefore each pixel may output a value (or level) corresponding to the original intensity (e.g., light intensity, brightness, photon count, etc.) of the corresponding color filter color (e.g., red, green, or blue). That is, the imaging sensor 210 may output a single color value for each pixel of the imaging sensor 210. However, the device 100 may require all color values ​​(e.g., red, green, and blue) for each pixel of the image output by the imaging sensor 210. Therefore, the device 100 may need to estimate and / or generate color values ​​to obtain a panchromatic image (e.g., a red-green-blue (RGB) image) from the imaging sensor 210. Such processing may be referred to as demosaicing.

[0044] Reference Figure 2B This illustrates the demosaicing process of a single Bayer pattern image 230. (Example) Figure 2B As shown, a complete three-channel (RGB) image 240 can be obtained by demosaicing a single Bayer pattern image 230. Demosaicing can refer to an image processing step that estimates a panchromatic image (e.g., RGB image 240) from a color filter mosaic pattern (e.g., a Bayer pattern) output by an imaging sensor 210.

[0045] In one embodiment, the imaging sensor 210 may include a CFA with a single Bayer pattern. However, recent improvements in both imaging sensor hardware and image processing software may have allowed for the introduction of other CFA mosaic patterns. For example, such as Figure 2C As shown, the imaging sensor 210 can use a 2×2 quad Bayer pattern 250. For example, as Figure 2D As shown, the imaging sensor 210 may use a 3×3 nine-Bayer pattern 260. However, this disclosure is not limited in this respect, and the imaging sensor 210 may use other CFA mosaic patterns without departing from the scope of this disclosure. For example, the imaging sensor 210 may use a CFA with a Q×Q Bayer pattern, where Q is a positive integer greater than three (3). For example, the imaging sensor 210 may use a CFA with a Q×Q Bayer pattern. CFA for X-Trans patterns, etc.

[0046] In embodiments, electronic devices (e.g., device 100, computing device, smartphone, PDA, laptop computer, tablet computer, smart device, wearable device, IoT device, or any other similar functional device) may include those described in reference to [reference needed]. Figure 3 and Figure 4 The discussion covers multiple imaging sensors (e.g., cameras). In an embodiment, the imaging sensor may have a CFA with multiple mosaic patterns (e.g., Bayer patterns).

[0047] Figure 3 An example of a mobile device having multiple cameras according to one aspect of this disclosure is depicted. (See also...) Figure 3 The image shows a mobile device 320. The mobile device 320 may include and / or may be similar in many respects to the one described above. Figure 1 The described device 100 may include additional features not mentioned above. For example, the mobile device 320 may include features as described in reference... Figure 1 The processor 120, memory 130, storage component 140, input component 150, output component 160, and communication interface 170 are described. Therefore, for the sake of brevity, the above reference to the mobile device 320 can be omitted. Figure 1 The description is a repetition of the previous one.

[0048] like Figure 3 As shown, the mobile device 320 may include a plurality of imaging sensors 322 (e.g., a first imaging sensor 322A, a second imaging sensor 322B, and a third imaging sensor 322C). However, this disclosure is not limited in this respect. For example, the mobile device 320 may include more than three (3) imaging sensors (e.g., four (4) or more imaging sensors), or may include fewer than three (3) imaging sensors (e.g., one (1) or two (2) imaging sensors).

[0049] Each imaging sensor 322 of the mobile device 320 may include a CFA having a corresponding mosaic pattern 324 (e.g., a first mosaic pattern 324A, a second mosaic pattern 324B, and a third mosaic pattern 324C). For example, the first imaging sensor 322A may have a first mosaic pattern 324A, the second imaging sensor 322B may have a second mosaic pattern 324B, and the third imaging sensor 322C may have a third mosaic pattern 324C. In an embodiment, the first mosaic pattern 324A may be a 1×1 single Bayer pattern, the second mosaic pattern 324B may be a 2×2 four Bayer pattern, and the third mosaic pattern 324C may be a 3×3 nine Bayer pattern. However, this disclosure is not limited in this respect. For example, the first mosaic pattern 324A may be a 2×2 four Bayer pattern, the second mosaic pattern 324B may be a 3×3 nine Bayer pattern, and the third mosaic pattern 324C may be a 1×1 single Bayer pattern. For example, one or more of the imaging sensors 322 may have the same mosaic pattern 324. For example, both the first mosaic pattern 324A and the second mosaic pattern 324B may be a 1×1 single Bayer pattern. For example, one or more of the mosaic patterns 324 may be different from the mosaic patterns described above. For example, one or more of the mosaic patterns 324 may be a Q×Q Bayer pattern or another mosaic pattern configuration.

[0050] In an embodiment, the mobile device 320 may be configured to demosaic an image output by multiple imaging sensors 322 using a corresponding demosaic model 330 (e.g., a 1×1 single Bayer demosaic model 330A, a 2×2 quadruple Bayer demosaic model 330B, or a 3×3 nine Bayer demosaic model 330C). For example, as Figure 3 As shown, the mobile device 320 can demosaic the mosaic image output by the first imaging sensor 322A using a 1×1 single Bayer demosaic model 330A, can demosaic the mosaic image output by the second imaging sensor 322B using a 2×2 four Bayer demosaic model 330B, and can demosaic the mosaic image output by the third imaging sensor 322C using a 3×3 nine Bayer demosaic model 330C.

[0051] In embodiments, the demosaic model 330 may be and / or may include a conventional demosaic algorithm for performing demosaicing on a mosaic image and directly outputs a demosaiced complete image (e.g., RGB image 240). In embodiments, the demosaic model 330 may be and / or may include a machine learning model configured to provide a demosaiced complete image (e.g., RGB image 240) directly from mosaic images output by multiple imaging sensors 322. For example, in embodiments, a 1×1 single Bayer demosaic model 330A may directly output a demosaiced complete image from a mosaic image that may have been captured by a first imaging sensor 322A using a first mosaic pattern 324A (e.g., a 1×1 single Bayer pattern). A 2×2 quad Bayer demosaic model 330B may directly output a demosaiced complete image from a mosaic image that may have been captured by a second imaging sensor 322B using a second mosaic pattern 324B (e.g., a 2×2 quad Bayer pattern). The 3×3 nine-bay de-mosaic model 330C can directly output a de-mosaiced complete image from a mosaic image that may have been captured by a third imaging sensor 322C using a third mosaic pattern 324C (e.g., a 3×3 nine-bay pattern).

[0052] In an embodiment, the mobile device 320 may be configured to demosaic an image output by multiple imaging sensors 322 by converting a mosaic image into a specific Bayer pattern and demosaicing the converted image data using a demosaic model configured for that Bayer pattern. For example, a 2×2 quad Bayer demosaic model 330B can remosaic an image captured using a 2×2 quad Bayer pattern into a 1×1 single Bayer pattern, and perform the demosaic operation using a 1×1 single Bayer demosaic model 330A. That is, the 2×2 quad Bayer demosaic model 330B can perform the operation as described above. Figure 11B The four-wash operation is used to obtain a converted (e.g., re-mosaiced) mosaic image in the form of a 1×1 single Bayer pattern to which a 1×1 single Bayer demosaic operation can be performed.

[0053] For example, the 3×3 nine-Bayer demosaic model 330C can remosaic a mosaic image captured using a 3×3 nine-Bayer pattern into a 1×1 single-Bayer pattern, and perform the demosaic operation using the 1×1 single-Bayer demosaic model 330A. In other words, the 3×3 nine-Bayer demosaic model 330C can perform the operation as described in the reference... Figure 11A The nine-wash operation is used to obtain a converted (e.g., re-mosaiced) mosaic image in the form of a 1×1 single Bayer pattern to which a 1×1 single Bayer demosaic operation can be performed. However, this disclosure is not limited in this respect, and the mosaic image can be re-mosaiced (e.g., converted, washed) into other mosaic patterns and then de-mosaiced.

[0054] This approach may be preferred because it can be performed using a single demosaic model (e.g., a 1×1 single-Bayer demosaic model 330A), potentially reducing the resource overhead required to perform demosaicing using multiple demosaic models 330 (e.g., a 1×1 single-Bayer demosaic model 330A, a 2×2 quadruple-Bayer demosaic model 330B, and a 3×3 nine-Bayer demosaic model 330C). However, remosaicing may introduce artifacts (e.g., noise, distortion, etc.) into the resulting demosaiced image.

[0055] Additionally, the above reference Figure 3 The methods discussed may not address the additional resource overhead (e.g., memory footprint, processing load, etc.) that might be required for using separate demosaic models 330 for each imaging sensor 322 of device 100. For example, when a user of device 100 dynamically changes the zoom factor of the image to be captured, device 100 (e.g., image signal processor) may need to switch from a first imaging sensor 322 to a second imaging sensor 322. This switching may require changing the demosaic model 330 needed to process the mosaic image output by the second imaging sensor 322, which could impact the performance of device 100 and / or user experience due to the potential introduction of switching latency. The effects of switching latency can be reduced and / or eliminated by preloading the demosaic model 330 onto device 100, which could significantly increase the memory requirements of device 100 (e.g., space footprint, speed). Aspects of this disclosure provide a unified demosaic model that can be used to demosaic mosaic images that may have been captured by imaging sensors using multiple mosaic layout patterns, as referenced. Figure 4 Described.

[0056] Figure 4 An example of a mobile device having multiple cameras according to one aspect of this disclosure is shown. (Refer to...) Figure 4 The image shows a mobile device 420. The mobile device 420 may include the components shown above. Figures 1 to 3 The described device 100 and moving device 320, and / or moving device 420 may be similar in many respects to those referenced above. Figures 1 to 3 The described device 100 and mobile device 320 may include additional features not mentioned above. For example, mobile device 420 may include features as described in reference... Figure 1 The processor 120, memory 130, storage component 140, input component 150, output component 160, communication interface 170, and de-mosaic component 180 are described. For example, mobile device 420 may include, as referenced... Figure 3The description includes multiple imaging sensors 322 with corresponding mosaic patterns 324. Therefore, for simplicity, the upper reference of the moving device 420 can be omitted. Figures 1 to 3 The description is a repetition of the previous one.

[0057] In the embodiments, as referred to Figures 1 to 3 At least a portion of the described demosaic operation can be performed by mobile device 420 (which may include demosaic component 180). In embodiments, another computing device (e.g., UE, server, laptop computer, smartphone, camera, wearable device, smart device, TV, printer, IoT device, etc.) that may include demosaic component 180 can perform at least a portion of the demosaic operation. That is, mobile device 420 can perform the described operation. Figure 4 The demosaic operation described is part of the demosaic operation, and the remainder of the demosaic operation may be performed by one or more other computing devices.

[0058] like Figure 4 As shown, the mobile device 420 can be configured to demosaic mosaic images output by multiple imaging sensors 322 using a unified demosaic model 430. For example, the mobile device 420 can use the unified demosaic model 430 to demosaic images captured using a 1×1 single Bayer pattern 324A and output by a first imaging sensor 322A, to demosaic images captured using a 2×2 quad Bayer pattern 324B and output by a second imaging sensor 322B, and to demosaic images captured using a 3×3 nine Bayer pattern 324C and output by a third imaging sensor 322C. (Refer to...) Figures 12 to 14 Further description of the unified demosaicing model 430.

[0059] In other words, the mobile device 420 can be configured to use a unified demosaic model 430, wherein the unified demosaic model 430 is capable of directly demosaicing mosaic images that may have been captured using any mosaic pattern 324 used by multiple imaging sensors 322. Thus, when compared with a reference... Figure 3 Compared to the described method, the unified demosaic model 430 may have potentially reduced resource overhead (e.g., memory footprint, processing load, etc.). Additionally, the unified demosaic model 430 avoids the need for remosaicing that may introduce artifacts (e.g., noise, distortion, etc.) into the resulting demosaiced image and / or the need to switch between multiple demosaic models 330 that may introduce performance latency.

[0060] See below for reference. Figures 5 to 17As described in further detail, the unified demosaic model 430 can be configured to reduce and / or potentially eliminate various noise levels from the mosaic image. Additionally, the unified demosaic model 430 can be configured to correct the resulting demosaic image for the presence of dead pixels in multiple imaging sensors 322.

[0061] Figure 5 A flowchart is depicted for selecting a demosaic model according to one aspect of this disclosure.

[0062] Reference Figure 5 This diagram illustrates a flowchart of an example method 500 for selecting a demosaic model 330 by means of implementing one or more aspects of this disclosure (e.g., means 100 or mobile device 320). In embodiments, at least a portion of method 500 may be performed by means of means including a demosaic component 180. In embodiments, another computing device including the demosaic component 180 (e.g., UE, server, laptop computer, smartphone, camera, wearable device, smart device, TV, printer, IoT device, etc.) may perform at least a portion of method 500. For example, in embodiments, means and another computing device may be combined to perform method 500. That is, means may perform a portion of method 500, and the remainder of method 500 may be performed by one or more other computing devices.

[0063] exist Figure 5 In block 510, method 500 may include acquiring (e.g., receiving, obtaining, accessing, etc.) a mosaic image (e.g., raw image data) that may have been acquired by a plurality of imaging sensors 322 having one or more mosaic sensor patterns 324. In embodiments, the mosaic image may include a plurality of noisy mosaic images. That is, the mosaic image may include mosaic images with one or more noise levels. In embodiments, the mosaic image may include a noise-free mosaic image. That is, the mosaic image may include a mosaic image that may have been previously denoised and / or a mosaic image that may have been captured under conditions where the noise level is practically zero.

[0064] In box 520, method 500 may include determining whether a mosaic image has been captured using a 1×1 single Bayer pattern 324A. When it is determined that a mosaic image has been captured using a 1×1 single Bayer pattern 324A (yes in box 520), method 500 may proceed to box 530, and train a 1×1 single Bayer demosaic model 330A using the mosaic image and / or demosaic the mosaic image using the 1×1 single Bayer demosaic model 330A. That is, method 500 may perform demosaicing on the mosaic image using the 1×1 single Bayer demosaic model 330A. In an embodiment, when it is determined that a mosaic image has not been captured using a 1×1 single Bayer pattern 324A (no in box 520), method 500 may proceed to box 540.

[0065] In box 540, method 500 may include determining whether a mosaic image has been captured using a 2×2 quad Bayer pattern 324B. When it is determined that a mosaic image has been captured using a 2×2 quad Bayer pattern 324B (Yes in box 540), method 500 may proceed to box 550, and train a 2×2 quad Bayer demosaic model 330B using the mosaic image and / or demosaic the mosaic image using the 2×2 quad Bayer demosaic model 330B. That is, method 500 may perform demosaicing on the mosaic image using the 2×2 quad Bayer demosaic model 330B. In an embodiment, when it is determined that a mosaic image has not been captured using a 2×2 quad Bayer pattern 324B (No in box 520), method 500 may proceed to box 560.

[0066] In box 560, method 500 may include training and / or testing a 3×3 nine-Bayer demosaicing model 330C. That is, method 500 may train the 3×3 nine-Bayer demosaicing model 330C on a mosaic image and / or perform demosaicing on a mosaic image using the 3×3 nine-Bayer demosaicing model 330C.

[0067] Figure 6 A flowchart illustrating an example training phase of a unified demosaicing model according to one aspect of this disclosure is shown.

[0068] Reference Figure 6This illustrates a training method 600 for a unified demosaic model 430 performed by means of implementing one or more aspects of this disclosure (e.g., means 100 or mobile device 420). In embodiments, at least a portion of the training method 600 may be performed by means of means including a demosaic component 180. In embodiments, another computing device including the demosaic component 180 (e.g., a UE, server, laptop computer, smartphone, camera, wearable device, smart device, TV, printer, IoT device, etc.) may perform at least a portion of the training method 600. For example, in embodiments, means and another computing device may be combined to perform the training method 600. That is, means may perform a portion of the training method 600, and the remainder of the training method 600 may be performed by one or more other computing devices.

[0069] like Figure 6 As shown, in block 610, training method 600 may include obtaining (e.g., receiving, acquiring, accessing, etc.) a batch of training image data (e.g., raw image data) that may have been generated from one or more mosaic sensor patterns 324, wherein one or more mosaic sensor patterns 324 may be targeted at the final unified demosaicing model 430. For example, the batch of training image data may have been generated from a 1×1 single Bayer pattern 324A, a 2×2 quad Bayer pattern 324B, a 3×3 nine Bayer pattern 324C, and / or a Q×Q Bayer pattern. However, this disclosure is not limited in this respect, and other training image data that may have been generated from other mosaic patterns may be used.

[0070] In one embodiment, the training image data may include multiple noisy mosaic images. That is, the training image data may include mosaic images with one or more noise levels. In another embodiment, the training image data may include images from, as referenced... Figure 10A and Figure 10B The original images of the training dataset described.

[0071] In box 620, training method 600 may include concatenating pattern information to training image data. For example, each of a plurality of mosaic images may be concatenated with an encoded embedding of a corresponding target mosaic pattern 324. In embodiments, the encoded embedding may indicate a corresponding mosaic pattern 324. For example, the encoded embedding may include values ​​that correspond to a predetermined mosaic pattern (e.g., a 1×1 single Bayer pattern 324A, a 2×2 quad Bayer pattern 324B, or a 3×3 nine Bayer pattern 324C).

[0072] In embodiments, the encoding embedding may include positional information of one or more colors of the corresponding mosaic pattern 324. For example, the encoding embedding may indicate the location of each color filter in the color filters of the corresponding mosaic pattern 324. That is, the encoding embedding may include one or more uniquely thermally encoded patterns that may correspond to the mosaic layout pattern of the corresponding imaging sensor 322. In embodiments, each of the one or more uniquely thermally encoded patterns may correspond to the location and / or position of a color in the mosaic pattern 324. For example, for any spatial location within the mosaic pattern, one uniquely thermally embedded pattern in the uniquely thermally embedded pattern may be active (e.g., hot, high, -, "1", etc.) for one of the color filters.

[0073] However, this disclosure is not limited in this respect, and the encoded embedding may include additional information that can help identify the mosaic layout pattern of the training image data. It is worth noting that the encoded embedding may include information that can describe the mosaic pattern of the imaging sensor.

[0074] In block 630, training method 600 may include providing training image data concatenated with encoded embeddings to unified demosaic model 430. In embodiments, unified demosaic model 430 may be provided with multiple channels including training image data and encoded embeddings. For example, the multiple channels may include a channel containing training image data and a separate channel for each color in the mosaic pattern 324. For example, when the mosaic pattern 324 corresponds to RGB imaging sensor 322, unified demosaic model 430 may be provided with four (4) channels, wherein a first channel includes training image data, a second channel includes uniquely heated pattern embedding data for red (R) color, a third channel includes uniquely heated pattern embedding data for green (G) color, and a fourth channel includes uniquely heated pattern embedding data for blue (B) color. For example, the multiple channels may include a separate channel for each color in the mosaic pattern 324, wherein the location and / or position of the color of the mosaic pattern 324 in the channel is set to the corresponding color value of that location in the training image data. For example, when the mosaic pattern 324 corresponds to the RGB imaging sensor 322, the unified demosaic model 430 can be provided with three (3) channels, wherein the first channel includes the red (R) value of the training image data corresponding to the mosaic pattern, the second channel includes the green (G) value of the training image data corresponding to the mosaic pattern, and the third channel includes the blue (B) value of the training image data corresponding to the mosaic pattern.

[0075] In box 640, training method 600 may include determining whether the training phase of the unified demosaic model 430 is complete. If the training phase of the unified demosaic model 430 has been determined to be complete, or when the training phase of the unified demosaic model 430 has been determined to be complete, training method 600 may terminate. In an embodiment, if the training phase of the unified demosaic model 430 has been determined to be incomplete, or when the training phase of the unified demosaic model 430 has been determined to be incomplete, training method 600 may return to box 610 and process another batch of image training data. The subsequent batch of image training data may be the same as and / or different from the previous batch of image training data. In an embodiment, one or more weights and / or hyperparameters of the unified demosaic model 430 may be adjusted before returning to box 610.

[0076] In this embodiment, the training phase of the unified demosaic model 430 can be determined as completed based on the number of iterations already performed. For example, the training phase can be determined as completed if the number of iterations already performed exceeds a predetermined counting threshold, or when the number of iterations already performed exceeds a predetermined counting threshold. In this embodiment, the predetermined counting threshold can be dynamically adjusted based on the image training data. That is, the predetermined counting threshold can vary based on one or more characteristics of the image training data (e.g., complexity, size, etc.).

[0077] In an embodiment, the training phase of the unified demosaic model 430 can be determined to be completed based on the analysis of its output. For example, the demosaic image output by the unified demosaic model 430 can be compared with the real demosaic image corresponding to the image training data.

[0078] In an embodiment, one or more metrics that can be interpreted according to human perception can be used to evaluate the quality of the demosaic image provided by the unified demosaic model 430. In an embodiment, the training phase of the unified demosaic model 430 can be determined to be complete based on the quality of the demosaic image exceeding a predetermined quality threshold. For example, in an embodiment, the CIE 1976 (CIE 76) mean chromatic aberration... It can be used to evaluate a uniform demosaic model 430. For example, an average color difference of less than or equal to two (2) (e.g., The two (2) colors can be considered indistinguishable to the average observer. However, this disclosure is not limited in this respect, and other metrics can be used to evaluate the quality of the demosaic image provided by the unified demosaicing model 430.

[0079] In an embodiment, the training phase of the unified demosaic model 430 can be determined to be completed based on the calculation of the loss of the unified demosaic model 430. In an embodiment, the training phase of the unified demosaic model 430 can be determined to be completed if the loss value is less than or equal to a predetermined loss threshold, or if the loss value reaches a minimum value, or if the loss value decreases by a relatively small amount over a predetermined number of iterations. However, this disclosure is not limited in this respect, and other conditions can be used to determine the completion of the training phase of the unified demosaic model 430.

[0080] In an embodiment, the loss of the unified demosaicing model 430 may be computed using one or more known functions, including those for determining the loss of a machine learning model. For example, the loss may correspond to, but is not limited to, at least one of the following: mean absolute error (MAE) loss (L1 loss), adversarial loss, Color loss, mean squared error (MSE) loss (L2 loss), cross-entropy loss, or a combination thereof.

[0081] Training method 600 can be used to validate the training of the unified demosaic model 430. That is, the unified demosaic model 430 can be validated by performing training method 600. In an embodiment, the validation of the unified demosaic model 430 may differ from the training method 600 in the weights and / or hyperparameters of the unified demosaic model 430. In an embodiment, the training image data used to validate the unified demosaic model 430 may differ from the training image data used to train the unified demosaic model 430.

[0082] In one embodiment, the validated unified demosaicing model 430 may be tested using a different batch of training image data than the training image data used to train and validate the unified demosaicing model 430.

[0083] Figure 7 A flowchart depicting an example reasoning phase of a unified demosaic model 430 according to one aspect of this disclosure.

[0084] Reference Figure 7This illustrates an inference method 700 for a unified demosaic model 430 executed by means of implementing one or more aspects of this disclosure (e.g., means 100 or mobile device 420). In embodiments, at least a portion of the inference method 700 may be executed by means of means including a demosaic component 180. In embodiments, another computing device including the demosaic component 180 (e.g., a UE, server, laptop computer, smartphone, camera, wearable device, smart device, TV, printer, IoT device, etc.) may execute at least a portion of the inference method 700. For example, in embodiments, means and another computing device may be combined to execute the inference method 700. That is, means may execute a portion of the inference method 700, and the remainder of the inference method 700 may be executed by one or more other computing devices.

[0085] like Figure 7 As shown, in block 710, method 700 may include obtaining (e.g., receiving, acquiring, accessing, etc.) a mosaic image (e.g., raw image data) that may have been acquired by multiple imaging sensors 322 having one or more mosaic sensor patterns 324. For example, the mosaic image may have been generated from a 1×1 single Bayer pattern 324A, a 2×2 quad Bayer pattern 324B, a 3×3 nine Bayer pattern 324C, and / or a Q×Q Bayer pattern. However, this disclosure is not limited in this respect, and the mosaic image may be generated from other mosaic patterns. In embodiments, the mosaic image may include multiple noisy mosaic images. That is, the mosaic image may include mosaic images having one or more noise levels. In embodiments, the mosaic image may include a noise-free mosaic image. That is, the mosaic image may include a mosaic image that may have been previously denoised and / or a mosaic image that may have been captured under conditions where the noise level is virtually zero.

[0086] In box 720, the inference method 700 may include concatenating pattern information to a mosaic image. The operations performed in box 720 may include those referenced above. Figure 6 The operation described in box 620 of the training method 600 shown above is similar in many respects to the one referenced above. Figure 6 The operations described in box 620 of the training method 600 shown may include additional operations not mentioned above. Therefore, for the sake of brevity, the above reference to box 720 may be omitted. Figure 6 The description is a repetition of the previous one.

[0087] In box 730, inference method 700 may include providing mosaic image data concatenated with encoded embeddings to unified demosaicing model 430. The operations performed in box 730 may include those referenced above. Figure 6The operation described in box 630 of the training method 600 shown above is similar in many respects to the one referenced above. Figure 6 The operations described in box 630 of the training method 600 shown may include additional operations not mentioned above. Therefore, for the sake of brevity, the above reference to box 730 may be omitted. Figure 6 The description is repetitive. For example, the unified demosaic model 430 in box 730 may have already been referenced above. Figure 6 The description is of training, validation, and / or testing.

[0088] In block 740, the inference method 700 may include obtaining a plurality of demosaic images corresponding to the mosaic image from the unified demosaic model 430. In embodiments, for example, the plurality of demosaic images may be provided to a device (e.g., device 100) for further processing and / or display to a user.

[0089] Advantageously, training method 600 and inference method 700 provide a unified demosaic model 430 capable of demosaicing and / or denoising mosaic images that may have been captured using multiple sensor mosaic layouts, without needing to switch between different models that can be configured for specific mosaic layouts. By jointly training the unified demosaic model 430 using training image data for one or more target mosaic layouts, the resulting unified demosaic model 430 can perform demosaicing and / or denoising on mosaic images captured using one or more target mosaic layouts. Furthermore, by having a single model rather than corresponding models for each of the target mosaic layouts, the unified demosaic model 430 can result in reduced resource overhead (e.g., memory footprint, processing load, etc.) compared to related demosaic models.

[0090] Additionally, the encoded embedding indicating the location and / or position of each color in the mosaic pattern 324 can be modified to reflect pixels that may be discarded from the mosaic pattern 324. For example, each of the plurality of imaging sensors 322 may have one or more color pixels that may fail and / or shut down (e.g., "dead"). Color pixels may die due to manufacturing errors, physical damage (e.g., dust, water, physical impact, etc.), disconnected electrical connections, etc. However, this disclosure is not limited in this respect, and pixels may cease to function due to other factors. It is worth noting that the number of dead pixels may constitute a large portion of the total number of pixels in the imaging sensor 322 (e.g., about 1% of the total number of pixels).

[0091] The associated imaging sensors and / or devices may attempt to correct dead pixels by interpolating the pixels of the mosaic image before performing demosaicing. For example, interpolation may include replacing the dead pixel value with a weighted average of surrounding pixels having the same color channel. However, such interpolation techniques and / or calculations may be imperfect and may introduce artifacts into the final image.

[0092] One aspect of this disclosure provides a mosaic masking enhancement that can be used to correct dead pixels in connection with the above embodiments. Mosaic masking allows a unified demosaic model 430 to provide demosaiced and denoised images from imaging data that may include dead pixels, as shown in reference... Figure 8 and Figure 9 The description is as follows. For example, during the training phase, mosaic masking can remove different parts of the pixels in the encoded embedding (e.g., set to zero (0) values), and during the inference phase, the encoded embedding can be modified to reflect which pixels might be killed from the mosaic pattern 324 (e.g., dropped).

[0093] Figure 8 A flowchart illustrating an example training phase of a unified demosaic model with mosaic masking according to one aspect of this disclosure is shown.

[0094] Reference Figure 8 This illustrates a training method 800 for a unified demosaic model 430 performed by means of implementing one or more aspects of this disclosure (e.g., means 100 or mobile device 420). In embodiments, at least a portion of the training method 800 may be performed by means of means including a demosaic component 180. In embodiments, another computing device including the demosaic component 180 (e.g., a UE, server, laptop computer, smartphone, camera, wearable device, smart device, TV, printer, IoT device, etc.) may perform at least a portion of the training method 800. For example, in embodiments, means and another computing device may be combined to perform the training method 800. That is, means may perform a portion of the training method 800, and the remainder of the training method 800 may be performed by one or more other computing devices.

[0095] Training methods 800 may include the above references. Figure 6 The training method described is 600, and / or may be similar in many ways to the one mentioned above. Figure 6 The training method 600 described may include additional operations not mentioned above. For example, training method 800 may include the operations described above with reference to training method 600 in boxes 610, 620, 630, and 640. Therefore, for the sake of brevity, the above references to training method 800 may be omitted. Figure 6 The description is a repetition of the previous one.

[0096] In box 822, training method 800 may include randomly discarding k% of pixels from each of a plurality of mosaic patterns 324, where k is a positive value ranging from 0 to 100. That is, a plurality of pixels in the plurality of mosaic patterns 324 may be masked (e.g., set to a zero (0) value) to simulate dead pixels in the imaging sensor 322. In embodiments, the number of pixels discarded may be selected to fall within the range of 0% to 5% of the total number of pixels in the mosaic pattern 324 (e.g., 0%). k% 5%). For example, the number of dropped pixels can be determined to be 0% (e.g., k=0), such that if the imaging sensor 322 has no dropped (e.g., dead) pixels, or when the imaging sensor 322 has no dropped (e.g., dead) pixels, the unified demosaic model 430 can perform demosaicing on the mosaic image. In an embodiment, the number of dropped pixels can be determined to be greater than 0% and less than or equal to 5% (e.g., 0%). <k% 5%). However, this disclosure is not limited in this respect, and the number of discarded pixels can be determined in another way, and / or from different ranges of values ​​(e.g., 0%). k% Select the number of pixels to discard (1%).

[0097] In an embodiment, the number of pixels to be discarded can be determined for each iteration of the training phase of the unified demosaic model 430. For example, the number of pixels to be discarded can be determined randomly for each iteration of the training phase, such that a different number of pixels can be discarded for each iteration of the training phase of the unified demosaic model 430. However, this disclosure is not limited in this respect.

[0098] In box 824, training method 800 may include updating pattern information already concatenated to the training image data in box 620 based on the number of discarded pixels determined in box 822. In an embodiment, each mosaic pattern 324 in the mosaic pattern 324 may have k% of the pixels discarded (e.g., set to a zero (0) value). For example, discarded pixels may be randomly selected for each mosaic pattern 324. In an embodiment, discarded pixels may be selected for each iteration of the training phase of the uniform demosaic model 430. That is, subsequent training iterations may have a different number (e.g., a different k%) of discarded pixels than previous iterations and / or may have a different set of discarded pixels than previous iterations. However, this disclosure is not limited in this respect, and discarded pixels may be selected in other ways without departing from the scope of this disclosure.

[0099] For example, when k equals 1% (e.g., k=1), the 1×1 single Bayer pattern 324A, the 2×2 quad Bayer pattern 324B, and the 3×3 nine Bayer pattern 324C can each be updated to discard 1% of their respective pixels, and the corresponding pattern information already concatenated to the training image data can also be updated. However, this disclosure is not limited in this respect. In embodiments, each mosaic pattern 324 may have a different number of discarded pixels.

[0100] In box 630, training method 800 can provide training imaging data with updated pattern information to unified demosaic model 430. In this way, unified demosaic model 430 can be configured to correct dead pixels in the imaging sensor during the inference phase of unified demosaic model 430, as shown in reference... Figure 9 Described.

[0101] although Figure 8 Boxes 610 to 640 are shown to execute training method 800 in a specific order, but it should be understood that this disclosure is not limited in this respect. For example, boxes 822 and / or 824 may be executed before box 620 without departing from the scope of this disclosure.

[0102] Figure 9 A flowchart depicting an example inference stage of a unified demosaic model 430 with mosaic masking according to one aspect of this disclosure.

[0103] Reference Figure 9 This illustrates an inference method 900 for a unified demosaic model 430 executed by means of devices (e.g., device 100 or mobile device 420) implementing one or more aspects of this disclosure. In embodiments, at least a portion of the inference method 900 may be executed by a device that may include a demosaic component 180. In embodiments, another computing device (e.g., UE, server, laptop computer, smartphone, camera, wearable device, smart device, TV, printer, IoT device, etc.) including the demosaic component 180 may execute at least a portion of the inference method 900. For example, in embodiments, the device and another computing device may be combined to execute the inference method 900. That is, the device may execute a portion of the inference method 900, and the remainder of the inference method 900 may be executed by one or more other computing devices.

[0104] Reasoning method 900 may include the above references Figure 7 The reasoning method described in 700, and / or may be similar in many ways to the one mentioned above. Figure 7The reasoning method 700 is described and may include additional operations not mentioned above. For example, reasoning method 900 may include the operations described above with reference to reasoning method 700 in boxes 710, 720, 730, and 740. Therefore, for the sake of brevity, the above references to reasoning method 900 may be omitted. Figure 7 The description is a repetition of the previous one.

[0105] In block 922, the inference method 900 may include updating pattern information already cascaded to the mosaic image in block 720 based on a dead pixel mask of the corresponding imaging sensor 322. The dead pixel mask may indicate one or more pixels that may be faulty and / or dead in the corresponding imaging sensor 322 and should therefore be discarded from the de-mosaic operation. In embodiments, the dead pixel mask may be calibrated by the manufacturer of the imaging sensor. In embodiments, the dead pixel mask may be updated based on error information provided by the imaging sensor 322. For example, the dead pixel mask may be updated periodically and / or non-periodically based on configuration events and / or specified events. However, this disclosure is not limited in this respect, and the dead pixel mask may be determined and / or updated in other ways without departing from the scope of this disclosure.

[0106] In box 730, inference method 900 can provide a mosaic image concatenated with updated pattern information to unified demosaic model 430. In this way, unified demosaic model 430 can correct dead pixels in the imaging sensor during the inference phase of unified demosaic model 430.

[0107] That is, in block 740, the inference method 700 may include obtaining multiple demosaic images corresponding to the mosaic images from the unified demosaic model 430, which may have been corrected to account for one or more dead pixels in the respective imaging sensor 322. In embodiments, for example, the multiple demosaic images may be provided to a device (e.g., device 100) for further processing and / or display to a user.

[0108] although Figure 9 Boxes 710 to 740 are shown to execute reasoning method 900 in a specific order, but it should be understood that this disclosure is not limited thereto. For example, box 922 may be executed before box 720 without departing from the scope of this disclosure.

[0109] Advantageously, training method 800 and inference method 900 provide a unified demosaic model 430 capable of demosaicing and / or denoising mosaic images that may have been captured using multiple sensor mosaic layouts, without needing to switch between different models that can be configured for specific mosaic layouts. Additionally, the unified demosaic model 430 can correct errors in mosaic images that may be caused by dead pixels. By jointly training the unified demosaic model 430 with training image data for one or more target mosaic layouts that may have been cascaded with pattern information that has been updated to reflect dead pixels in the imaging sensor, the resulting unified demosaic model 430 can perform demosaicing and / or denoising, as well as dead pixel correction, on mosaic images captured using one or more target mosaic layouts. Furthermore, by having a single model rather than a corresponding model for each target mosaic layout, the unified demosaic model 430 can result in reduced resource overhead (e.g., memory footprint, processing load, etc.) compared to related demosaic models.

[0110] Figure 10A and Figure 10B An example of a training dataset for training a demosaicing model according to one aspect of this disclosure is shown. (Refer to...) Figure 10A and Figure 10B The training dataset 1000 is shown.

[0111] Synthetic datasets can be used to train and / or evaluate relevant learning-based demosaic models, which may limit their application in real-world scenarios. These synthetic datasets can be generated by synthesizing Bayer inputs using previously demosaiced sRGB images. As mentioned above, the imaging sensor's ISP can perform denoising and / or demosaicing operations on the raw sensor images. Therefore, creating a training dataset that includes the raw images may be superior to a dataset of synthetic sRGB images. Additionally, relevant training datasets may lack images with high-frequency details, which could pose a challenge to demosaic algorithms.

[0112] Therefore, the training dataset 1000 may include a dataset of original images that can be constructed to contain high-frequency details. As used herein, the training dataset 1000 may be referred to as the Hard Demosaic Dataset (HDD). Figure 10A As shown, the training dataset 1000 may include multiple scenarios 1020 (e.g., first scenario A 1020A, second scenario B 1020B to the Sth scenario S1020S, where S is a positive integer greater than one (1)).

[0113] Each scene 1020 in the training dataset 1000 may include multiple original images (e.g., 600 original images) that can be constructed to contain multiple high-frequency regions by arranging a relatively large number of highly textured and / or small objects within the regions. For example, objects included in the original images may have a size smaller than a predetermined threshold. As used herein, a high-frequency region may refer to a region in an image that may contain fine details, such as, but not limited to, hair, texture, pores, lines, skin features, etc. In embodiments, high-frequency regions may contain a relatively large set of different contrasts. In embodiments, each scene 1020 may include a set of images of similar objects (such as, but not limited to, cotton balls, plants, stationery, cosmetics, fused beads, pom-poms, paper scraps, rope, clothing, ribbons, toy animals, organized toy blocks, unorganized toy blocks, sports equipment, paint samples, pony beads, etc.) of different sizes and / or colors.

[0114] Each scene 1020 may include multiple raw images 1060 (e.g., first raw image 1060A, second raw image 1060B to Nth raw image 1060N) that can be captured from multiple viewpoints (e.g., first viewpoint A, second viewpoint B to Nth viewpoint N, where N is a positive integer greater than one (1)). Figure 10B As shown in the illustration. In a non-limiting embodiment, each scene 1020 may include 7 to 59 original images (e.g., For example, multiple viewpoints can capture the corresponding scene from various positions, orientations, and / or zoom levels. In an embodiment, multiple original images 1060 may have the same pixel size (e.g., 8640 × 5760 pixels). However, this disclosure is not limited in this respect, and multiple original images 1060 may have a different pixel size and / or multiple original images 1060 may have multiple different pixel sizes.

[0115] In an embodiment, each view may be captured at multiple focal lengths (e.g., a first focal length level A, a second focal length level B to a Pth focal length level P, where P is a positive integer greater than one (1)). For example, a raw image 1060A captured from a first viewpoint A may include a first raw image 1060AA taken at the first focal length level A, a second raw image 1060AB taken at the second focal length level B, and a Pth raw image 1060AP taken at the Pth focal length level P. For example, a raw image 1060B captured from a second viewpoint B may include a first raw image 1060BA taken at the first focal length level A, a second raw image 1060BB taken at the second focal length level B, and a Pth raw image 1060BP taken at the Pth focal length level P. For example, the raw image 1060N captured from the Nth viewpoint N may include a first raw image 1060NA taken at a first focal length level A, a second raw image 1060NB taken at a second focal length level B, and a Pth raw image 1060NP taken at a Pth focal length level P. In a non-limiting embodiment, each view may include 1 to 3 focal length levels (e.g., It is worth noting that each view may include images at multiple focal length levels, such that all image regions are focally aligned in at least one image of the view.

[0116] In an embodiment, portions (e.g., blocks) of multiple original images 1060 in the training dataset 1000 can be used to train, validate, and / or test the unified demosaicing model 430. For example, each block may include a portion of the original image 1060 captured from a specific viewpoint at a specific focal length level. Each block may have a size smaller than the original image 1060. For example, a block may have a size of 48 × 48 pixels. However, this disclosure is not limited in this respect, and blocks may have a smaller size (e.g., 32 × 32 pixels) or a larger size (e.g., 64 × 64 pixels). In an embodiment, the shape of the block may be polygonal or another shape.

[0117] In embodiments, blocks may be selected based on criteria used to select the leading “hardest” block for each viewpoint. That is, the selection criteria may be configured to identify candidate blocks based on the level of complexity (e.g., high-frequency details) contained in the blocks for each viewpoint. For example, the selection method may include dividing the original image 1060 for each viewpoint (e.g., first original image 1060A, second original image 1060B to Nth original image 1060N) into multiple blocks and mosaicking the blocks based on a 1×1 single Bayer pattern. The selection method may include applying bilinear interpolation to the mosaicked blocks as a form of simple demosaicing. However, this disclosure is not limited in this respect, and other forms of demosaicing may be performed on the mosaicked blocks. The selection method may include ranking the demosaiced blocks by one or more complexity criteria (e.g., reconstructed peak signal-to-noise ratio (PSNR)). The selection method may include identifying the leading portion (e.g., the first 25%) of the ranked demosaiced blocks for each viewpoint as training blocks for selection based on the 1×1 single Bayer pattern. A repeatable selection method is used to select training blocks for 2×2 four-Bayer patterns and 3×3 nine-Bayer patterns by de-mosaicing the blocks based on 2×2 four-Bayer patterns and 3×3 nine-Bayer patterns, respectively.

[0118] The training dataset 1000 may include training pairs of clean and noisy raw images and / or patches. In an embodiment, a noisy raw image can be generated from a corresponding clean raw image by adding noise from a calibrated noise model and sampling the clean raw image using a sensor pattern.

[0119] In an embodiment, a first portion of the training dataset 1000 can be used to train the unified demosaic model 430, a second portion of the training dataset 1000 can be used to validate the unified demosaic model 430, and a third portion of the training dataset 1000 can be used to test the unified demosaic model 430. The first to third portions of the training dataset 1000 may be different from each other. That is, an original image 1060 included in one portion may not be included in another portion of the training dataset 1000. However, this disclosure is not limited in this respect, and the original image 1060 may be included in two or more portions of the training dataset 1000. In an embodiment, the size of the portions (e.g., the number of original images 1060) may be the same as or different from each other. The original image 1060 to be included in each portion of the training dataset 1000 may be determined based on various criteria and / or design constraints. For example, in the case where the original image 1060 to be included in each part is determined based on the scenes included in the training dataset 1000 and the training dataset 1000 includes seventeen (17) scenes, the first part of the training dataset 1000 that can be used to train the unified demosaic model 430 may include scenes 1 to 10, the second part of the training dataset 1000 that can be used to validate the unified demosaic model 430 may include scenes 11 and 12, and the third part of the training dataset 1000 that can be used to test the unified demosaic model 430 may include scenes 13 to 17.

[0120] Advantageously, training dataset 1000 may contain challenging scenes with hard blocks that may have already been annotated for training, validation, and / or testing of the demosaicing model. Furthermore, by including raw images of multiple scenes with high-frequency details captured at multiple viewpoints and along multiple focal length levels, training dataset 1000 may outperform relevant datasets of synthetic images because it may pose a challenge to the demosaicing algorithm and better represent real-world scenes.

[0121] Figures 11A to 11D Depiction can be used to achieve reference Figures 12 to 14 Examples of elements and / or blocks of an example embodiment of the unified demosaic model 430 described herein. Specifically, according to one aspect of this disclosure, Figure 11A An example depicting a nine-shaded block uniformly de-mosaiced model. Figure 11B An example of a four-wash block in a unified demosaic model is shown. Figure 11C An example of a color extraction head for a uniform demosaic model is depicted, and Figure 11D This shows an example of a modified color extraction head for a unified demosaic model.

[0122] Reference Figure 11A An example depicting a nine-shaded block uniformly de-mosaiced model. For example... Figure 11A As shown, the nine-wash block 1110 can wash a nine (Nona) mosaic image 1112 based on a 3×3 nine-Bayer pattern into a single (Single) mosaic image 1114 corresponding to a 1×1 single-Bayer pattern. Not all washing operations are performed in... Figure 11A The nine-wash block 1110 can be visualized by repeatedly performing the nine-wash operation on each 6×6 pixel block of the nine-mosaic image 1112 in a sliding window manner.

[0123] Reference Figure 11B This shows an example of a four-shaded block in a unified demosaic model. For example... Figure 11B As shown, the quad wash frame 1120 can wash a quad mosaic image 1122 based on a 2×2 quad Bayer pattern into a single mosaic image 1124 corresponding to a 1×1 single Bayer pattern. Not all wash operations are performed in... Figure 11B The image is visualized in the image. For example, the quad shuffle block 1120 can repeatedly perform the quad shuffle operation on each 4×4 pixel block of the quad mosaic image 1122 in a sliding window manner.

[0124] Reference Figure 11C This describes an example of a color extraction head 1130 that depicts a unified demosaic model. (Example:) Figure 11C As shown, the color extraction head 1130 may include three (3) convolutional layers 1150. In an embodiment, the first convolutional layer 1150 may have… Mosaic images downsampled to have spatial dimensions and feature channel counts The second convolutional layer 1150 upsamples the mosaic image to a reduced spatial dimension. The third convolutional layer 1150 can downsample the mosaic image to have feature channel counting. The spatial dimensions and feature channel counts. However, this disclosure is not limited in this respect, and the color extraction head 1130 may perform other convolution operations on the mosaic image without departing from the scope of this disclosure.

[0125] Reference Figure 11D This shows an example of a modified color extraction head 1140 for a unified demosaic model. (Example follows) Figure 11D As shown, the color-modified extraction head 1140 may include three (3) convolutional layers 1150. In an embodiment, the first convolutional layer 1150 may have… Mosaic images downsampled to have spatial dimensions and feature channel counts The mosaic image is represented by feature channel counts. The second convolutional layer 1150 upsamples the mosaic image to a feature channel count. The third convolutional layer 1150 can downsample the mosaic image to have feature channel counting. The feature channel count. However, this disclosure is not limited in this respect, and the modified color extraction head 1140 may perform other convolution operations on the mosaic image without departing from the scope of this disclosure.

[0126] Various aspects of this disclosure provide a unified demosaic model (e.g., unified demosaic model 430) that can be implemented to demosaic images captured by one or more imaging sensors 322 having multiple mosaic patterns 324 (such as, but not limited to, 1×1 single Bayer pattern 324A, 2×2 quad Bayer pattern 324B, 3×3 nine Bayer pattern 324C, etc.). Figures 12 to 14 An example embodiment of a unified demosaic model 430 according to one aspect of this disclosure is shown.

[0127] Figure 12 An example of a re-mosaic unified model (SRUM) based on one aspect of this disclosure is depicted. (See also...) Figure 12 This is an example of a block diagram depicting the SRUM 1200. The SRUM 1200 may include the components referenced above. Figures 4 to 9 The described unified demosaic model 430, and / or may be similar in many ways to the above reference. Figures 4 to 9 The unified demosaic model 430 is described and may include additional features not mentioned above. Therefore, for the sake of brevity, the above reference to SRUM 1200 can be omitted. Figures 4 to 9 The description is a repetition of the previous one.

[0128] In an embodiment, at least a portion of SRUM 1200 may be performed by a mobile device 420 that may include a demosaic component 180. In an embodiment, another computing device (e.g., device 100, UE, server, laptop computer, smartphone, camera, wearable device, smart device, TV, printer, IoT device, etc.) that may include the demosaic component 180 may perform at least a portion of the operations performed by SRUM 1200. That is, mobile device 420 may perform operations as described above. Figure 12 A portion of the SRUM 1200 described, and the remainder of the SRUM 1200 may be executed by one or more other computing devices.

[0129] like Figure 12 As shown, the SRUM 1200 may have a multi-head architecture, wherein each head can be configured to accept different mosaic pattern types, and a shared backbone can perform demosaicing on the mosaic image transformed by the input heads. For example, the SRUM 1200 may include a four-mix washing head 1210, a nine-mix washing head 1220, and a shared backbone 1230 that can accept a 1×1 single Bayer mosaic pattern image 1232.

[0130] The four-way washing head 1210 can be configured to accept a 2×2 quad Bayer mosaic pattern image 1212, re-mosaic (e.g., wash) the 2×2 quad Bayer mosaic pattern image 1212 using the four-way washing block 1120, and convert it into a four-way washing single Bayer mosaic pattern image 1214 using the color extraction head 1130 and the convolutional layer 1150, and then convert the four-way washing single Bayer mosaic pattern image 1214 into a color image with... The spatial dimensions and feature channel counts are calculated, and the converted four-mix wash single Bayer mosaic pattern image is provided to the shared backbone 1230 for demosaicing. In an embodiment, the four-mix wash head 1210 can be configured to provide an image that has been converted to have... The spatial dimensions and feature channel counts of the four-washed single Bayer mosaic pattern image 1214 can preserve some positional information of the four-Bayer mosaic pattern in the resulting image.

[0131] The nine-wash head 1220 can be configured to accept a 3×3 nine-Bayer mosaic pattern image 1222, re-mosaic (e.g., wash) the 3×3 nine-Bayer mosaic pattern image 1222 using the nine-wash block 1110, and convert it into a nine-wash single-Bayer mosaic pattern image 1224 using the color extraction head 1130 and the convolutional layer 1150, and then convert the nine-wash single-Bayer mosaic pattern image 1224 into a color image with... The spatial dimensions and feature channel counts are calculated, and the converted nine-mix wash Bayer mosaic pattern image is provided to the shared backbone 1230 for demosaicing. In an embodiment, the nine-mix wash head 1220 can be configured to provide an image that has been converted to have... The spatial dimensions and feature channel counts of the nine-washed single Bayer mosaic pattern image 1224 can preserve some positional information of the nine-Bayer mosaic pattern in the resulting image.

[0132] The shared backbone 1230 can be configured to accept a 1×1 single Bayer mosaic pattern image 1232 and perform joint demosaicing and denoising on the 1×1 single Bayer mosaic pattern image 1232, wherein the 1×1 single Bayer mosaic pattern image 1232 may be directly provided as input to SRUM 1200, or may be provided by a four-mix shampoo head 1210 or a nine-mix shampoo head 1220. In embodiments, the shared backbone 1230 may include a convolutional neural network (CNN), such as, but not limited to, Figure 12 The Joint Denoising and De-mosaicing (JDNDM) model is shown. The shared backbone 1230 may include a color extraction section, a feature extraction section, and a reconstruction section. The color extraction section may include a color extraction head 1130 with relatively large filters (e.g., 256 channels) and convolutional layers to amplify the main resolution (e.g., ...). The feature extraction part may include a residual channel attention network (RCAN) 1160 and long skip connections (LSC). In the reconstruction part, two (2) convolutional layers 1150 can be used to convert the extracted features into a demosaic and denoised (clean) image 1240.

[0133] In an embodiment, SRUM 1200 may be jointly trained using an image dataset (e.g., training dataset 1000) containing images of mosaic images for one or more target mosaic layouts (e.g., 1×1 single Bayer pattern, 2×2 quad Bayer pattern, 3×3 nine Bayer pattern, etc.), as described above. Figure 6 , Figure 8 , Figure 10A and Figure 10B Described.

[0134] As shown above (refer to the reference) Figure 12 As described, SRUM 1200 can be and / or may include an example of a unified demosaic model 430 based on a multi-head method that can force (e.g., remosaic) a mosaic image into a single-channel representation.

[0135] Figure 13 An example of a latent space unification model (LSUM) according to one aspect of this disclosure is shown. (See also...) Figure 13 The image shows an example of a block diagram of the LSUM 1300. The LSUM 1300 may include the components described above. Figures 4 to 9 The described unified demosaic model 430, and / or may be similar in many ways to the above reference. Figures 4 to 9 The unified demosaic model 430 is described and may include additional features not mentioned above. Therefore, for the sake of brevity, the above reference to LSUM 1300 can be omitted. Figures 4 to 9 The description is a repetition of the previous one.

[0136] In an embodiment, at least a portion of LSUM 1300 may be performed by a mobile device 420, which may include a demosaic component 180. In an embodiment, another computing device (e.g., device 100, UE, server, laptop computer, smartphone, camera, wearable device, smart device, TV, printer, IoT device, etc.) that may include the demosaic component 180 may perform at least a portion of the operations performed by LSUM 1300. That is, mobile device 420 may perform operations as described above. Figure 13 A portion of the LSUM 1300 described, and the remainder of the LSUM 1300 may be executed by one or more other computing devices.

[0137] like Figure 13As shown, the LSUM 1300 may have a multi-head architecture, wherein each head can be configured to accept different mosaic pattern types, and a shared backbone can demosaic the mosaic image transformed by the input heads. For example, the LSUM 1300 may include a four-mix washing head 1310, a nine-mix washing head 1320, and a shared backbone 1330 that can accept a 1×1 single Bayer mosaic pattern image 1332.

[0138] The LSUM 1300 utilizes a unified latent space that can circumvent and / or prevent the implicit bottlenecks of re-mosaic methods. In other words, the LSUM 1300 can transform each mosaic image into one with... The shared potential space of spatial dimensions, rather than converting mosaic images into images with... A single-channel mosaic image with spatial dimensions. Therefore, the LSUM 1300 can include the above reference. Figure 12 The SRUM 1200 described herein, and / or may be similar in many respects to the references above. Figure 12 The SRUM 1200 is described and may include additional features not mentioned above. For example, the difference between the four-mix wash head 1310 and the nine-mix wash head 1320 of the LSUM 1300 and the four-mix wash head 1210 and the nine-mix wash head 1220 of the SRUM 1200 may be that the forced convolutional layer 1150 for the four-mix wash single Bayer mosaic pattern image 1214 and the nine-mix wash single Bayer mosaic pattern image 1224 may not exist in the four-mix wash head 1310 and the nine-mix wash head 1320. In an embodiment, a shared latent space may be formed before RCAN 1160.

[0139] Shared backbone 1330 may include references Figure 12 The described shared backbone 1230, and / or may be similar to the reference in many respects. Figure 12 The shared backbone 1230 is described and may include additional features not mentioned above. Therefore, for the sake of brevity, the above reference to the shared backbone 1330 can be omitted. Figure 12 The description is a repetition of the previous one.

[0140] In an embodiment, the LRUM 1300 may be jointly trained using an image dataset (e.g., training dataset 1000) containing images of mosaic images for one or more target mosaic layouts (e.g., 1×1 single Bayer pattern, 2×2 quad Bayer pattern, 3×3 nine Bayer pattern, etc.), as described above. Figure 6 , Figure 8 , Figure 10A and Figure 10B Described.

[0141] As shown above (refer to the reference) Figure 13As described, LSUM 1300 can be and / or may include examples of a unified demosaic model 430 that can be based on a multi-head approach that can use a unified latent space, and thus circumvent bottlenecks that may be inherent in the remosaic approach.

[0142] Figure 14 An example of a modified joint denoising and demosaic model (M-JDNDM) according to one aspect of this disclosure is depicted. (See also...) Figure 14 This shows an example block diagram of architecture 1400 including M-JDNDM 1440. M-JDNDM 1440 may include the components described above. Figures 4 to 9 The described unified demosaic model 430, and / or may be similar in many ways to the above reference. Figures 4 to 9 The unified demosaic model 430 is described and may include additional features not mentioned above. Therefore, for the sake of brevity, the above reference to M-JDNDM 1440 can be omitted. Figures 4 to 9 The description is a repetition of the previous one.

[0143] In an embodiment, at least a portion of architecture 1400 may be performed by a mobile device 420 that may include demosaic component 180. In an embodiment, another computing device (e.g., device 100, UE, server, laptop computer, smartphone, camera, wearable device, smart device, TV, printer, IoT device, etc.) that may include demosaic component 180 may perform at least a portion of the operations performed by architecture 1400. That is, mobile device 420 may perform operations as described above. Figure 14 The architecture 1400 described is a part of the architecture 1400, and the remainder of the architecture 1400 can be executed by one or more other computing devices.

[0144] like Figure 14 As shown, the M-JDNDM 1440 may have a single-head architecture where pattern information can be used to encode channel information. For example, the M-JDNDM 1440 can provide a mosaic image that can be cascaded with pattern information corresponding to a pattern of a mosaic image via a channel stack 1170. That is, the channel stack 1170 may provide a first mosaic image 1410 that can be captured using a 1×1 single Bayer mosaic pattern 324A, a second mosaic image 1420 that can be captured using a 2×2 quad Bayer mosaic pattern 324B, or a third mosaic image 1430 that can be captured using a 3×3 nine Bayer mosaic pattern 324C.

[0145] In one embodiment, the channel stack 1170 may provide a noisy single-mosaic image 1412 that can be cascaded with pattern information 1414 (e.g., red pattern information 1414R, green pattern information 1414G, and blue pattern information 1414B) corresponding to a 1×1 single Bayer mosaic pattern 324A. In another embodiment, the channel stack 1170 may provide a noisy quad-mosaic image 1422 that can be cascaded with pattern information 1424 (e.g., red pattern information 1424R, green pattern information 1424G, and blue pattern information 1424B) corresponding to a 2×2 quad-Bayer mosaic pattern 324B. In an embodiment, the channel stack 1170 may be provided with a noisy nine-mosaic image 1432 that can be cascaded with pattern information 1434 (e.g., red pattern information 1434R, green pattern information 1434G, and blue pattern information 1434B) corresponding to a 3×3 nine-bayer mosaic pattern 324C.

[0146] As shown above (refer to the reference) Figure 6 The pattern information described may include positional information for one or more colors of the corresponding mosaic pattern 324. For example, the pattern information may indicate the location of each color filter in the color filters of the corresponding mosaic pattern 324. That is, the pattern information may include one or more thermally coded patterns that may correspond to the sensor mosaic layout pattern of the corresponding imaging sensor 322. In embodiments, each of the one or more thermally coded patterns may correspond to the location and / or position of a color in the mosaic pattern 324. For example, for any spatial location within the mosaic pattern, one thermally coded pattern embedding in the thermally coded pattern embedding may be active (e.g., hot, high, -, "1", etc.) for one of the color filters.

[0147] M-JDNDM 1440 may include references Figure 12 The described shared backbone 1230, and / or may be similar to the reference in many respects. Figure 12 The shared backbone 1230 is described and may include additional features not mentioned above. For example, the M-JDNDM 1440 may include a modified color extraction head 1140 that may not use encapsulated convolution, and thus avoids removing the natural encoding of mosaic pattern information in the mosaic image. Therefore, for the sake of brevity, the above reference to M-JDNDM 1440 may be omitted. Figure 12 The description is a repetition of the previous one.

[0148] In an embodiment, the M-JDNDM 1440 may be jointly trained using an image dataset (e.g., training dataset 1000) containing images of mosaic images for one or more target mosaic layouts (e.g., 1×1 single Bayer pattern, 2×2 quad Bayer pattern, 3×3 nine Bayer pattern, etc.), as described above. Figure 6 , Figure 8 , Figure 10A and Figure 10B Described.

[0149] As shown above (refer to the reference) Figure 14 As described, M-JDNDM 1440 may be and / or may include an embodiment of a unified demosaic model 430 that can be based on a single-head method that can use pattern embedding to perform demosaicing on multiple mosaic patterns.

[0150] Figure 15A An example of a mobile device with mosaic masking 1500 according to one aspect of this disclosure is shown. Figure 15B An example of pattern information with mosaic masking is shown according to one aspect of this disclosure.

[0151] Reference Figure 15A An example of a mobile device 420 is shown. The mobile device 420 may include the components described above. Figures 1 to 4 The described device 100 and moving device 320, and / or may be similar in many respects to those referenced above. Figures 1 to 4 The described device 100 and moving device 320 may include additional features not mentioned above. Therefore, for the sake of brevity, the above reference to the moving device 420 may be omitted. Figures 1 to 4 The description is a repetition of the previous one.

[0152] like Figure 15A As shown, mosaic patterns 1524 (e.g., first mosaic pattern 1524A, second mosaic pattern 1524B, and third mosaic pattern 1524C) may have been updated to indicate dead pixels in the corresponding imaging sensors 322 (e.g., first imaging sensor 322A, second imaging sensor 322B, and third imaging sensor 322C). Dead pixels may be... Figure 15A The black squares at the pixel locations of the mosaic pattern 1524 indicate the location. For example, the first mosaic pattern 1524A may have one (1) dead pixel, the second mosaic pattern 1524B may have two (2) dead pixels, and the third mosaic pattern 1524C may have two (2) dead pixels. However, this disclosure is not limited in this respect, and the imaging sensor 322 may have different numbers of dead pixels (including zero (0)) and / or dead pixels may be located at different positions in the imaging sensor 322.

[0153] Reference Figure 15B This shows pattern information 1534 with mosaic masking. (Example) Figure 15BAs shown, pattern information 1534 masks (e.g., sets to a zero (0) value) the pixels corresponding to dead pixels in mosaic pattern 1524. For example, first pattern information 1534A may include pattern information corresponding to first mosaic pattern 1524A, such as, for example, first red pattern information 1534AR, first green pattern information 1534AG, and first blue pattern information 1534AB. In such an example, first green pattern information 1534AG may indicate dead pixels in first mosaic pattern 1524A.

[0154] For example, the second pattern information 1534B may include pattern information corresponding to the second mosaic pattern 1524B, such as, for example, second red pattern information 1534BR, second green pattern information 1534BG, and second blue pattern information 1534BB. For example, the second green pattern information 1534BG and the second blue pattern information 1534BB may indicate dead pixels in the second mosaic pattern 1524B.

[0155] For example, the third pattern information 1534C may include pattern information corresponding to the third mosaic pattern 1524C, such as, for example, third red pattern information 1534CR, third green pattern information 1534CG, and third blue pattern information 1534CB. For example, the third red pattern information 1534CR may indicate dead pixels in the third mosaic pattern 1524C.

[0156] In one embodiment, the updated pattern information 1534 can be cascaded to the mosaic image generated by the imaging sensor 322 and provided to the unified demosaic model 430. In this way, the unified demosaic model 430 can generate a demosaiced and denoised (clean) image 1450 that has been corrected for dead pixels in the imaging sensor 322, as referred to above. Figure 9 and Figure 14 Described.

[0157] Advantageously, refer to the above Figures 1 to 15B The methods, apparatus, systems, and non-transitory computer-readable media described herein for demosaicing and denoising images provide a unified demosaicing model capable of demosaicing and / or denoising mosaic images that may have been captured using multiple sensor mosaic layouts, without the need to switch between different models that can be configured for specific mosaic layouts. Furthermore, the aspects proposed herein provide correction for errors in mosaic images that may be caused by dead pixels in the imaging sensor. The aspects described herein are also applicable to other image processing techniques and image processing standards employing these techniques.

[0158] Figure 16A block diagram of an example device for demosaicing an image is shown according to one aspect of this disclosure. Device 1600 may be a computing device (e.g., Figure 1 The device 1600 may be a computing device (or a device 100). In an embodiment, the device 1600 may include a receiving component 1602 configured to receive communication (e.g., wired, wireless) from another device (e.g., device 1608), a demosaic component 180 configured to demosaic an image, and a transmitting component 1606 configured to transmit communication (e.g., wired, wireless) to another device (e.g., device 1608). The components of the device 1600 may communicate with each other (e.g., via one or more buses or electrical connections). Figure 16 As shown, device 1600 can communicate with another device 1608 (such as, but not limited to, a server, laptop computer, smartphone, UE, camera, wearable device, smart device, IoT device, etc.) using receiving component 1602 and / or transmitting component 1606.

[0159] In an embodiment, device 1600 may be configured to perform the actions described herein. Figures 1 to 15B One or more operations described herein. In embodiments, device 1600 may be configured to perform one or more processes described herein (such as...). Figure 17 Method 1700). In an embodiment, device 1600 may include reference to Figure 1 One or more components of the described device 100.

[0160] Receiving component 1602 may receive communications (such as control information, data communications, or combinations thereof) from device 1608 (e.g., server, laptop computer, smartphone, UE, camera, wearable device, smart device, IoT device, etc.). Receiving component 1602 may provide the received communications to one or more other components of device 1600 (such as de-mosaic component 180). In embodiments, receiving component 1602 may perform signal processing on the received communications and may provide the processed signals to one or more other components. In embodiments, receiving component 1602 may include references... Figure 1 The described device 100 includes one or more antennas, a receiver processor, a controller / processor, a memory, or a combination thereof.

[0161] Transmitting component 1606 can transmit communications (such as control information, data communications, or a combination thereof) to device 1608 (e.g., server, laptop computer, smartphone, UE, camera, wearable device, smart device, IoT device, etc.). In an embodiment, demosaic component 180 can generate communications and can send the generated communications to transmitting component 1606 for transmission to device 1608. In an embodiment, transmitting component 1606 can perform signal processing on the generated communications and can send the processed signals to device 1608. In an embodiment, transmitting component 1606 may include references... Figure 1 The described device 100 includes one or more antennas, a transmitting processor, a controller / processor, a memory, or a combination thereof. In embodiments, the transmitting component 1606 may be co-located with the receiving component 1602 in a transceiver and / or transceiver assembly.

[0162] The demosaic component 180 can be configured to demosaic and / or denoise mosaic images that may have been captured using multiple sensor mosaic layouts without requiring switching between different models. In embodiments, the demosaic component 180 may include a set of components such as an acquisition component 1610 configured to acquire multiple mosaic images, a concatenation component 1620 configured to concatenate encoding embeddings to multiple mosaic images, a providing component 1630 configured to provide the concatenated multiple mosaic images to a machine learning model, and an acquisition component 1640 configured to acquire multiple demosaic images corresponding to the multiple mosaic images.

[0163] In an embodiment, the set of components may be separate from and distinct from the demosaic component 180. In an embodiment, one or more components in the set of components may include those referenced above. Figure 1 The controller / processor (e.g., processor 120), memory (e.g., memory 130), or a combination thereof of the described device 100, or may be implemented as referenced above. Figure 1 The described apparatus 100 is located within a controller / processor (e.g., processor 120), a memory (e.g., memory 130), or a combination thereof. In embodiments, one or more of the collection of components may be implemented at least partially as software stored in memory (such as memory 130). For example, a component (or a portion of a component) may be implemented as computer-executable instructions or code stored in a computer-readable medium (e.g., a non-transitory computer-readable medium) and executable by a controller or processor to perform the function or operation of the component.

[0164] Figure 16 The number and arrangement of components shown are provided as an example. In practice, with Figure 16Compared to the components shown, there may be additional components, fewer components, different components, or components with different arrangements. Additionally, Figure 16 The two or more components shown can be implemented within a single component, or Figure 16 The single component shown can be implemented as multiple distributed components. In the embodiment, Figure 16 The collection of (one or more) components shown is executable and described as being composed of Figures 1 to 15B Another set of components shown performs one or more functions.

[0165] Reference Figure 17 In operation, device 1600 may perform method 1700 for demosaicing an image. Method 1700 may be performed by device 100 (which may include processor 120, memory 130, and storage component 140, and device 100 may be the entire device 100 and / or include one or more components of device 100 (such as input component 150, output component 160, communication interface 170, and / or demosaic component 180)) and / or device 1600. Method 1700 may be performed by device 100, device 1600, and / or demosaic component 180 communicating with device 1608 (e.g., server, laptop computer, smartphone, UE, camera, wearable device, smart device, IoT device, etc.).

[0166] exist Figure 17 In block 1710, method 1700 may include acquiring multiple mosaic images from multiple sensors of the device, wherein each of the multiple sensors has a corresponding mosaic pattern from multiple mosaic patterns. For example, in one aspect, device 100, demosaic component 180, and / or acquisition component 1610 may be configured to or may include means for acquiring multiple mosaic images from multiple sensors 322 of device 1600, wherein each of the multiple sensors 322 has a corresponding mosaic pattern from multiple mosaic patterns 324.

[0167] In an embodiment, obtaining block 1710 may include obtaining multiple mosaic images that may have been generated from at least one of a 1×1 single Bayer pattern 324A, a 2×2 quad Bayer pattern 324B, a 3×3 nine Bayer pattern 324C, or a Q×Q Bayer pattern, as referenced above. Figure 7 Described.

[0168] In an embodiment, the obtaining operation in block 1710 may include obtaining a plurality of mosaic images, which may include one or more mosaic images with noise.

[0169] In an embodiment, at least one of the plurality of sensors 322 may include one or more dead pixels in the CFA of at least one sensor 322.

[0170] exist Figure 17 In block 1720, method 1700 may include cascading each of a plurality of mosaic images with an encoded embedding of a corresponding mosaic pattern from a plurality of sensors that capture each of the plurality of mosaic images. For example, in one aspect, device 100, demosaic component 180, and / or cascading component 1620 may be configured to, or may include, means for cascading each of a plurality of mosaic images with an encoded embedding of a corresponding mosaic pattern 324 from a plurality of sensors 322 that capture each of the plurality of mosaic images.

[0171] For example, each coded embedding in the coded embedding can indicate the corresponding mosaic pattern of the color filter array of the corresponding sensor in multiple sensors.

[0172] In an embodiment, each encoded embedding in the encoded embedding may include position information of one or more colors of the corresponding mosaic pattern 324.

[0173] In an embodiment, each coding embedding in the coding embedding may include one or more uniquely thermally coded patterns corresponding to the sensor patterns of the respective sensors in the plurality of sensors 322.

[0174] In an embodiment, each of one or more uniquely thermally encoded patterns may correspond to the color of the sensor pattern of the corresponding sensor 322.

[0175] In an embodiment, the coded embedding may indicate one or more dead pixels of the corresponding sensor 322.

[0176] exist Figure 17 In box 1730, method 1700 may include providing a concatenated plurality of mosaic images to a machine learning model. For example, in one aspect, means 100, demosaic component 180, and / or providing component 1630 may be configured to or may include means for providing a concatenated plurality of mosaic images to machine learning model 430.

[0177] For example, the provisioning operation in box 1730 may include providing multiple channels, including multiple mosaic images and coded embeddings, to the unified demosaicing model 430.

[0178] exist Figure 17In box 1740, method 1700 may include obtaining multiple demosaic images corresponding to multiple mosaic images from a machine learning model. For example, in one aspect, device 100, demosaic component 180 and / or acquisition component 1640 may be configured to or may include means for obtaining multiple demosaic images 1450 corresponding to multiple mosaic images from machine learning model 430.

[0179] For example, the acquisition operation in box 1740 may include acquiring one or more de-mosaic images 1450, which have been de-noised, corresponding to one or more mosaic images, from machine learning model 430.

[0180] In aspects that can be combined with any other aspect, method 1700 may include training a machine learning model 430 based on a first portion of image dataset 1000 and encoded embeddings of multiple mosaic patterns 324, validating the machine learning model 430 using a second portion of image dataset 1000 and encoded embeddings of multiple mosaic patterns 324, and testing the machine learning model 430 using a third portion of image dataset 1000 and encoded embeddings of multiple mosaic patterns 324. The second portion of image dataset 1000 may differ from the first portion of image dataset 1000. The third portion of image dataset 1000 may differ from the first and second portions of image dataset 1000.

[0181] Image dataset 1000 may include multiple scenes 1020 captured according to multiple views 1060A to 1060N. Each of the multiple views may have been captured according to multiple focal lengths. Each scene 1020 may include at least one high-frequency region, wherein the at least one high-frequency region includes at least one of multiple textures or multiple objects that may have a size less than a predetermined threshold.

[0182] In one aspect that can be combined with any other aspect, method 1700 may include: determining a random percentage value between zero and a predetermined masking limit; calculating a random number of pixels to be masked in the encoded embedding based on the random percentage value for each training iteration of the machine learning model; and randomly selecting a random number of pixels in the encoded embedding to be masked.

[0183] The following aspects are merely illustrative and may be combined with other embodiments or teachings described herein without limitation.

[0184] One aspect is a method for demosaicing images by a device. The method may include: acquiring multiple mosaic images from multiple sensors of the device; concatenating each of the multiple mosaic images with an encoded embedding of a corresponding mosaic pattern from the sensors that captured each of the multiple mosaic images; providing the concatenated multiple mosaic images to a machine learning model; and obtaining multiple demosaic images corresponding to the multiple mosaic images from the machine learning model. Each of the multiple sensors has a corresponding mosaic pattern from the multiple mosaic patterns.

[0185] In an embodiment, the multiple mosaic patterns in the method may include at least one of a single Bayer pattern, a four-Bayer pattern, a nine-Bayer pattern, or a Q×Q Bayer pattern. In the method, Q may be a positive integer greater than three (3).

[0186] In an embodiment, in the method, each encoded embedding in the encoded embedding may indicate a corresponding mosaic pattern of the color filter array of the corresponding sensor among a plurality of sensors.

[0187] In an embodiment, each encoded embedding in the method may include positional information of one or more colors of the corresponding mosaic pattern.

[0188] In an embodiment, in the method, each coding embedding in the coding embedding may include one or more uniquely thermally coded patterns corresponding to the sensor patterns of the respective sensors among a plurality of sensors.

[0189] In an embodiment, in the method, each of one or more uniquely thermally encoded patterns may correspond to the color of the sensor pattern of the corresponding sensor.

[0190] In an embodiment, in the method, at least one of the multiple sensors may include one or more dead pixels in the color filter array of the at least one sensor. In the method, the coded embedding corresponding to the at least one sensor may indicate one or more dead pixels. In the method, acquiring multiple demosaic images may include: acquiring one or more demosaic images corresponding to the at least one sensor and corrected for one or more dead pixels from a machine learning model.

[0191] In an embodiment, the multiple mosaic images in the method may include one or more mosaic images with noise. In the method, obtaining multiple de-mosaic images includes: obtaining one or more noise-removed de-mosaic images corresponding to one or more mosaic images from a machine learning model.

[0192] In an embodiment, the method may include: training a machine learning model based on a first portion of an image dataset and encoded embeddings of multiple mosaic patterns. The method may include: validating the machine learning model using a second portion of the image dataset and encoded embeddings of multiple mosaic patterns. The method may include: testing the machine learning model using a third portion of the image dataset and encoded embeddings of multiple mosaic patterns. The image dataset may include multiple scenes captured according to multiple views. Each of the multiple views may be captured according to multiple focal lengths. Each scene may include at least one high-frequency region, said at least one high-frequency region comprising at least one of multiple textures or multiple objects having a size less than a predetermined threshold. The second portion may differ from the first portion, and the third portion may differ from the first and second portions.

[0193] In an embodiment, training the machine learning model in the method may include: for each training iteration of the machine learning model, determining a random number of pixels to be masked in the encoded embedding. In the method, training the machine learning model may also include: randomly selecting a random number of pixels in the encoded embedding to be masked.

[0194] In one embodiment, determining the random number of pixels in the method may include: determining a random percentage value between zero and a predetermined mask limit. In another embodiment, determining the random number of pixels may include: calculating a random number of masked pixels based on the random percentage value.

[0195] One aspect is a device for demosaicing images. The device may include: multiple sensors; a memory, storing instructions; and one or more processors communicatively coupled to the multiple sensors and the memory. Each of the multiple sensors has a corresponding mosaic pattern from multiple mosaic patterns. The one or more processors are configured to execute instructions to perform one or more of the aforementioned methods.

[0196] In an embodiment, the plurality of mosaic patterns in the device may include at least one of a single Bayer pattern, a four-Bayer pattern, a nine-Bayer pattern, or a Q×Q Bayer pattern, and Q may be a positive integer greater than three (3).

[0197] In an embodiment, in the device, each coded embedding in the coded embedding may indicate a corresponding mosaic pattern of the color filter array of the corresponding sensor among a plurality of sensors.

[0198] In an embodiment, in the device, each encoded embedding in the encoded embedding may include position information of one or more colors of the corresponding mosaic pattern.

[0199] In an embodiment, in the device, each coding embedding may include one or more uniquely thermally coded patterns corresponding to the sensor patterns of a corresponding sensor among a plurality of sensors. In the device, each of the one or more uniquely thermally coded patterns may correspond to the color of the sensor pattern of the corresponding sensor.

[0200] In an embodiment, in the device, at least one of a plurality of sensors may include one or more dead pixels in the color filter array of the at least one sensor. In the device, an encoded embedding corresponding to the at least one sensor may indicate one or more dead pixels. In the device, one or more processors may also be configured to execute instructions to obtain from a machine learning model one or more demosaic images corresponding to the at least one sensor and corrected for one or more dead pixels.

[0201] In one embodiment, the plurality of mosaic images in the device may include one or more mosaic images with noise. In the device, one or more processors may also be configured to execute instructions to obtain, from a machine learning model, one or more de-mosaic images with noise removed, corresponding to the one or more mosaic images.

[0202] In an embodiment, one or more processors in the device may also be configured to execute instructions to train a machine learning model based on a first portion of the image dataset and encoded embeddings of multiple mosaic patterns. In the device, one or more processors may also be configured to execute instructions to validate the machine learning model using a second portion of the image dataset and encoded embeddings of multiple mosaic patterns, wherein the second portion differs from the first portion. In the device, one or more processors may also be configured to execute instructions to test the machine learning model using a third portion of the image dataset and encoded embeddings of multiple mosaic patterns, wherein the third portion differs from the first and second portions. The image dataset may include multiple scenes captured according to multiple views. Each of the multiple views may be captured according to multiple focal lengths. Each scene may include at least one high-frequency region, said at least one high-frequency region may include at least one of multiple textures or multiple objects having a size less than a predetermined threshold.

[0203] In an embodiment, in order to train a machine learning model, one or more processors may also be configured to execute instructions to determine a random percentage value between zero and a predetermined mask limit. In order to train a machine learning model, one or more processors may also be configured to execute instructions to calculate, for each training iteration of the machine learning model, a random number of pixels to be masked in the coded embedding, based on the random percentage value. In order to train a machine learning model, one or more processors may also be configured to execute instructions to randomly select a random number of pixels in the coded embedding to be masked.

[0204] According to one aspect of this disclosure, an apparatus for demosaicing an image includes: means for acquiring a plurality of mosaic images from a plurality of sensors of the apparatus; means for concatenating each of the plurality of mosaic images with encoded embeddings of a corresponding mosaic pattern from a sensor among the plurality of sensors that captured each of the plurality of mosaic images; means for providing the concatenated plurality of mosaic images to a machine learning model; and means for acquiring a plurality of demosaic images corresponding to the plurality of mosaic images from the machine learning model. Each of the plurality of sensors has a corresponding mosaic pattern from the plurality of mosaic patterns.

[0205] The foregoing disclosure provides examples and descriptions, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure, or may be derived from practice of the implementations.

[0206] For example, the terms "component," "module," "system," etc., are intended to include computer-related entities such as, but not limited to, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As an example, both an application running on a computing device and the computing device itself can be components. One or more components may reside within a process and / or an execution thread, and components may reside on a single computer and / or be distributed across two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate in a local and / or remote manner, such as based on signals having one or more data packets (such as data interacting between one component and another in a local system or a distributed system), and / or across a network such as the Internet.

[0207] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail integration. A computer-readable medium may include a computer-readable non-transitory storage medium having computer-readable program instructions thereon for causing a processor to perform operations. A non-transitory computer-readable medium excludes transient signals.

[0208] A computer-readable storage medium can be a tangible means for retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EEPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), DVDs, memory sticks, floppy disks, mechanical encoding devices (such as punch cards or raised structures in recesses on which instructions are recorded), and any suitable combination of the foregoing. For example, a computer-readable storage medium may not be construed as a transient signal itself (such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0209] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0210] Computer-readable program code / instructions used to perform operations can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, wherein the programming languages ​​include object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or can be connected to an external computer (e.g., via the Internet through an Internet service provider). In embodiments, electronic circuitry (including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs)) can execute computer-readable program instructions to personalize the electronic circuitry for performing aspects or operations by utilizing the status information of the computer-readable program instructions.

[0211] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to generate a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in the boxes of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that instructs a computer, programmable data processing apparatus, and / or other means to function in a particular manner, such that the computer-readable storage medium storing the instructions comprises an article of manufacture containing instructions that implement aspects of the functions / actions specified in the boxes of a flowchart and / or block diagram.

[0212] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus or other device to generate a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus or other device perform the functions / actions specified in the frames of a flowchart and / or block diagram.

[0213] According to exemplary embodiments, at least one of the components, elements, modules, or units (collectively referred to as "components" in this paragraph) indicated by the boxes in the accompanying drawings can be implemented as various numbers of hardware, software, and / or firmware structures performing the functions described above. According to exemplary embodiments, at least one of these components can use a direct circuit structure (such as a memory, processor, logic circuit, lookup table, etc.) that can perform the functions under the control of one or more microprocessors or other control devices. Additionally, at least one of these components can be embodied in a portion of a module, program, or code that may contain one or more executable instructions for performing a specified logical function, and can be executed by one or more microprocessors or other control devices. Furthermore, at least one of these components may include a processor (such as a central processing unit (CPU) capable of performing the corresponding function), a microprocessor, etc., or may be implemented by a processor (such as a central processing unit (CPU) capable of performing the corresponding function), a microprocessor, etc. Two or more of these components can be combined into a single component performing all the operations or functions of the combined two or more components. Additionally, at least a portion of the functionality of at least one of these components can be performed by another component. The functional aspects of the above exemplary embodiments can be implemented as algorithms that execute on one or more processors. In addition, the components represented by the boxes or processing steps may employ any number of related technologies for electronic configuration, signal processing and / or control, data processing, etc.

[0214] In this disclosure, the article “a” is intended to include one or more items and is used interchangeably with “one or more.” The term “a” or similar language is used where only one item is intended. For example, the term “processor” can refer to a single processor or multiple processors. When a processor is described as performing an operation and a processor is referred to as performing another operation, multiple operations can be performed by a single processor, or by any one or a combination of multiple processors.

[0215] A processor may include various processing circuits and / or multiple processors. For example, as used herein (including the claims), the term "processor" may include various processing circuits including at least one processor, wherein one or more of the at least one processor may be configured individually and / or collectively in a distributed manner to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform a number of functions, these terms cover, for example, cases where one processor performs some of the functions while another processor performs other functions of the functions, and cases where a single processor can perform all the described functions. Additionally, at least one processor may include, for example, a combination of processors performing various described / disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions.

[0216] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent an instruction module, instruction segment, or instruction portion comprising one or more executable instructions for implementing a specified logical function. The method, computer system, and computer-readable medium may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those depicted in the figures. In some alternative implementations, the functions recorded in the blocks may not occur in the order shown in the figures. For example, two blocks shown consecutively may actually be executed simultaneously or substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block illustrated in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0217] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not a limitation of the implementation method. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code—it is understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0218] Unless explicitly stated otherwise, the elements, actions, or instructions described in this disclosure should not be construed as critical or essential. Additionally, for example, the article “a” may be intended to include one or more items and may be used interchangeably with “one or more.” Additionally, for example, the term “set” may be intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more.” The term “a” or similar language may be used where it may be intended for only one item. Additionally, for example, the terms “having,” “including,” “comprising,” etc., may be intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase “based on” may be intended to mean “at least partially based on.” Furthermore, expressions such as “at least one of [A] and [B]” or “at least one of [A] or [B]” may be understood to include only A, only B, or both A and B.

[0219] References to “an embodiment,” “embodiment,” or similar language throughout this specification indicate that a particular feature, structure, or characteristic described in connection with the indicated embodiment may be included in at least one embodiment of the solution. Therefore, the phrases “in one embodiment,” “in an embodiment,” and similar language throughout this specification may, but not necessarily all, refer to the same embodiment. For example, terms such as “first” and “second” or “first” and “second” may be used to simply distinguish corresponding components from one another without otherwise limiting the components (e.g., importance or order). It is understood that, where the terms “operably” or “communically” are used, or where the terms “operably” or “communically” are not used, if an element (e.g., a first element) is referred to as “coupled to another element (e.g., a second element),” “coupled to another element (e.g., a second element),” “connected to another element (e.g., a second element),” or “connected to another element (e.g., a second element),” it means that the element may be directly (e.g., wiredly) coupled to the other element, wirelessly coupled to the other element, or coupled to the other element via a third element.

[0220] Understandably, when a component or layer can be described as being "above," "on," "below," "under," "connected to," or "coupled to" another component or layer, the component or layer can be directly above, above, below, or coupled to the other component or layer, or there may be intermediate components or layers. Conversely, when a component can be described as being "directly above," "directly on," "directly below," "directly under," "directly connected to," or "directly coupled to," there are no intermediate components or layers.

[0221] Descriptions of various aspects and embodiments have been presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Even combinations of features recited in the claims and / or disclosed in the specification are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or not disclosed in the specification. Although each dependent claim listed below may be directly subordinated to only one claim, the disclosure of possible implementations includes combinations of each dependent claim with each other claim in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein may be chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0222] It is understood that the specific order or hierarchy of boxes in the disclosed process / flowchart is schematic of exemplary methods. Based on design preferences, it is understood that the specific order or hierarchy of boxes in the process / flowchart may be rearranged. Additionally, some boxes may be combined and / or omitted. The appended claims present the elements of the various boxes in a sample order and are not intended to limit one to the specific order or hierarchy presented.

[0223] Furthermore, the features, advantages, and characteristics described in this disclosure may be combined in any suitable manner in one or more embodiments. Based on the description herein, those skilled in the art will recognize that this disclosure may be practiced without one or more specific features or advantages of a particular embodiment. In other instances, additional features and advantages that may not be present in all embodiments of this disclosure may be recognized in specific embodiments.

Claims

1. A method for demosaicing an image by a device, the method comprising: Multiple mosaic images are obtained from multiple sensors of the device, wherein each of the multiple sensors has a corresponding mosaic pattern from multiple mosaic patterns; Each of the plurality of mosaic images is concatenated with the coded embedding of the corresponding mosaic pattern of the sensor that captured each image; The concatenated mosaic images are then provided to a machine learning model; and Multiple demosaic images corresponding to the multiple mosaic images are obtained from the machine learning model.

2. The method according to claim 1, wherein, The plurality of mosaic patterns include at least one of a single Bayer pattern, a four-Bayer pattern, a nine-Bayer pattern, or a Q×Q Bayer pattern, and Where Q is a positive integer greater than 3.

3. The method according to any one of claims 1 and 2, wherein, Each coding embedding in the plurality of sensors includes one or more uniquely thermally encoded patterns corresponding to the sensor patterns of the corresponding sensors among the plurality of sensors, and Each of the one or more uniquely thermally encoded patterns corresponds to the color of the sensor pattern of the corresponding sensor.

4. The method according to any one of claims 1 to 3, wherein, At least one of the plurality of sensors includes one or more dead pixels in the color filter array of the at least one sensor. Wherein, the coded embedding corresponding to the at least one sensor indicates the one or more dead pixels, and The acquisition of the plurality of demosaic images includes: acquiring one or more demosaic images corresponding to the at least one sensor and correcting the one or more dead pixels from the machine learning model.

5. The method according to any one of claims 1 to 4, wherein, The plurality of mosaic images includes one or more mosaic images with noise, and The process of obtaining the plurality of demosaic images includes: obtaining one or more demosaic images corresponding to the one or more mosaic images, with the noise removed, from the machine learning model.

6. The method according to any one of claims 1 to 5, further comprising: The machine learning model is trained based on the first part of the image dataset and the encoded embeddings of the plurality of mosaic patterns, wherein the image dataset includes a plurality of scenes captured according to a plurality of views, each of the plurality of views being captured according to a plurality of focal lengths, each scene including at least one high-frequency region, the at least one high-frequency region including at least one of a plurality of textures or a plurality of objects having a size less than a predetermined threshold. The machine learning model is validated using the second portion of the image dataset and the encoded embeddings of the plurality of mosaic patterns, wherein the second portion differs from the first portion; and The machine learning model is tested using a third portion of the image dataset and the encoded embeddings of the plurality of mosaic patterns, wherein the third portion is different from the first and second portions.

7. The method according to any one of claims 1 to 6, wherein, The training of the machine learning model includes: Determine a random percentage value between zero and a predetermined mask limit; For each training iteration of the machine learning model, based on the random percentage value, calculate the random number of pixels masked in the encoded embedding; and Randomly select the random number of pixels to be masked from the encoded embedding.

8. An apparatus for depixelating an image, comprising: Multiple sensors, wherein each of the multiple sensors has a corresponding mosaic pattern from multiple mosaic patterns; Memory, storage instructions; and One or more processors, communicatively coupled to the plurality of sensors and the memory, wherein the one or more processors are configured to execute the instructions to perform the following operations: Multiple mosaic images are obtained from the multiple sensors; Each of the plurality of mosaic images is concatenated with the coded embedding of the corresponding mosaic pattern of the sensor that captured each image; The concatenated mosaic images are then provided to a machine learning model; and Multiple demosaic images corresponding to the multiple mosaic images are obtained from the machine learning model.

9. The device according to claim 8, wherein, The plurality of mosaic patterns include at least one of a single Bayer pattern, a four-Bayer pattern, a nine-Bayer pattern, or a Q×Q Bayer pattern, and Where Q is a positive integer greater than 3.

10. The device according to any one of claims 8 and 9, wherein, Each coding embedding in the plurality of sensors includes one or more uniquely thermally encoded patterns corresponding to the sensor patterns of the corresponding sensors among the plurality of sensors, and Each of the one or more uniquely thermally encoded patterns corresponds to the color of the sensor pattern of the corresponding sensor.

11. The device according to any one of claims 8 to 10, wherein, At least one of the plurality of sensors includes one or more dead pixels in the color filter array of the at least one sensor. Wherein, the coded embedding corresponding to the at least one sensor indicates the one or more dead pixels, and The one or more processors are further configured to execute the instructions to obtain from the machine learning model one or more demosaic images corresponding to the at least one sensor, which have been corrected for the one or more dead pixels.

12. The device according to any one of claims 8 to 11, wherein, The plurality of mosaic images includes one or more mosaic images with noise, and The one or more processors are further configured to execute instructions to obtain from the machine learning model one or more de-mosaic images corresponding to the one or more mosaic images, with the noise removed.

13. The device according to any one of claims 8 to 12, wherein, The one or more processors are also configured to execute the instructions to perform the following operations: The machine learning model is trained based on the first part of the image dataset and the encoded embeddings of the plurality of mosaic patterns, wherein the image dataset includes a plurality of scenes captured according to a plurality of views, each of the plurality of views being captured according to a plurality of focal lengths, each scene including at least one high-frequency region, the at least one high-frequency region including at least one of a plurality of textures or a plurality of objects having a size less than a predetermined threshold. The machine learning model is validated using the second portion of the image dataset and the encoded embeddings of the plurality of mosaic patterns, wherein the second portion differs from the first portion; and The machine learning model is tested using a third portion of the image dataset and the encoded embeddings of the plurality of mosaic patterns, wherein the third portion is different from the first and second portions.

14. The device according to any one of claims 8 to 13, wherein, In order to train the machine learning model, the one or more processors are configured to execute the instructions to perform the following operations: Determine a random percentage value between zero and a predetermined mask limit; For each training iteration of the machine learning model, a random number of pixels are masked in the encoded embedding based on the random percentage value. as well as Randomly select the random number of pixels to be masked from the encoded embedding.

15. A computer-readable storage medium for storing instructions, wherein, When executed by at least one processor, the instructions cause the at least one processor to perform the method according to any one of claims 1 to 7.

Images