Fusion techniques for combining most significant bits and least significant bits of image data in image processing or other applications

Abstract

A method includes obtaining raw image data, where the raw image data includes data values, each data value having a most significant bit and a least significant bit. The method further includes providing the raw image data to the trained machine learning model and generating processed image data using the trained machine learning model. The method further includes presenting an image based on the processed image data. The trained machine learning model is trained to modulate a feature map associated with the most significant bits of the data values of the original image data based on the least significant bits of the data values of the original image data in order to generate a fusion of the most significant bits and the least significant bits of the data values of the original image data.

Description

Technical Field

[0001] The present disclosure relates generally to imaging systems and more particularly to fusion techniques for combining most significant bits and least significant bits of image data in image processing or other applications. Background Art

[0002] Digital cameras in mobile electronic devices, such as smart phones and tablet computers, have continued to increase in performance over time. Among other things, the resolution of images captured using digital cameras in mobile electronic devices continues to increase rapidly. It is expected that digital cameras in mobile electronic devices will soon be able to generate images with hundreds of megapixels or more of image data. Not only do these images need to be stored in memory, but they also typically need to be converted from the original image data format to a different image data format (such as a red-green-blue or "RGB" image color space) for display to the user. The conversion process typically involves the use of additional memory space in order to process and convert the original image data. As a result, capturing, processing, and storing images of increasing resolution can be time consuming and require significant processing and memory resources. Summary of the invention

[0003] Technical Solution

[0004] The present disclosure relates to fusion techniques for combining most significant bits and least significant bits of image data in image processing or other applications.

[0005] In a first embodiment, a method includes obtaining raw image data, wherein the raw image data includes data values, each data value having a most significant bit and a least significant bit. The method also includes providing the raw image data to a trained machine learning model and generating processed image data using the trained machine learning model. The method also includes presenting an image based on the processed image data. The trained machine learning model is trained to modulate a feature map associated with a most significant bit of the data value of the raw image data based on the least significant bit of the data value of the raw image data so as to generate a fusion of the most significant bit and the least significant bit of the data value of the raw image data.

[0006] In a second embodiment, an electronic device includes at least one imaging sensor and at least one processing device. The at least one processing device is configured to obtain raw image data using the at least one imaging sensor, wherein the raw image data includes data values, each data value having a most significant bit and a least significant bit. The at least one processing device is also configured to provide the raw image data to a trained machine learning model and generate processed image data using the trained machine learning model. The at least one processing device is also configured to initiate presentation of an image based on the processed image data. The trained machine learning model is trained to modulate a feature map associated with a most significant bit of a data value of the raw image data based on a least significant bit of the data value of the raw image data, so as to generate a fusion of the most significant bit and the least significant bit of the data value of the raw image data.

[0007] In a third embodiment, a machine-readable medium containing instructions, when executed, causes at least one processor of a device to perform operations corresponding to the method described above. Other technical features may be easily understood by those skilled in the art according to the following drawings, description and claims.

[0008] Before proceeding to the following "Detailed Description", it may be beneficial to set forth the definitions of certain words and phrases used throughout this patent document. The terms "send", "receive" and "communicate" and their derivatives cover direct and indirect communications. The terms "include" and "comprises" and their derivatives mean inclusion without limitation. The term "or" is inclusive, meaning and / or. The phrase "associated with..." and its derivatives mean including, included in, interconnected with, including, contained in, connected to or connected with, coupled to, or coupled with, communicable with, collaborate with, interlaced, juxtaposed, close to, bound to, or bound with, having, having the property of, having a relationship with, etc.

[0009] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data, or a part thereof adapted for implementation with a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. "Non-transitory" computer-readable media excludes wired links, wireless links, optical links, or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media in which data can be permanently stored and media in which data can be stored and overwritten later (such as rewritable optical disks or erasable memory devices).

[0010] As used herein, terms and phrases such as "having", "may have", "include" or "may include" a feature (such as a number, function, operation or component (such as a part)) indicate the presence of the feature and do not exclude the presence of other features. Moreover, as used herein, the phrases "A or B", "at least one of A and / or B" or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B" and "at least one of A or B" may indicate (1) including at least one A, (2) including at least one B or (3) including at least one A and at least one B. In addition, as used herein, the terms "first" and "second" may modify various components regardless of their importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate user devices that are different from each other, regardless of the order or importance of the devices. Without departing from the scope of the present disclosure, a first component may be represented as a second component and vice versa.

[0011] It will be understood that when an element (such as a first element) is referred to as being (operably or communicatively) "coupled / connected to" or "coupled / connected to" another element (such as a second element), it may be coupled or connected to or coupled or connected to the other element directly or via a third element. In contrast, it will be understood that when an element (such as a first element) is referred to as being "directly coupled / connected to" or "directly coupled / connected to" another element (such as a second element), no other element (such as a third element) is interposed between the element and the other element.

[0012] As used herein, the phrase "configured (or set) to ..." may be used interchangeably with the phrases "suitable for ...", "capable of ...", "designed to ...", "adapted to ...", "made to ...", or "capable of ...", depending on the circumstances. The phrase "configured (or set) to ..." does not inherently mean "specifically designed in hardware to ...". Rather, the phrase "configured to ..." may mean that a device may perform an operation together with another device or component. For example, the phrase "a processor configured (or set) to perform A, B, and C" may mean a general-purpose processor (such as a CPU (central processing unit) or an application processor) that can perform operations by executing one or more software programs stored in a memory device, or a special-purpose processor (such as an embedded processor) for performing operations.

[0013] The terms and phrases used herein are provided only to describe some embodiments of the present disclosure, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that the singular forms "one", "an" and "the" include plural references unless the context clearly stipulates otherwise. All terms and phrases used herein (including technical and scientific terms and phrases) have the same meaning as the meanings commonly understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It will also be understood that terms and phrases such as those defined in common dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined as such herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.

[0014] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of the following: a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head-mounted device (HMD), electronic clothing, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include at least one of the following: a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave, a washing machine, a dryer, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (such as SAMSUNG HOMESYNC, APPLETV, or GOOGLE TV), a smart speaker or a speaker with an integrated digital assistant (such as SAMSUNG GALAXY HOME, APPLEHOMEPOD, or AMAZON ECHO), a game console (such as XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a portable camera, or an electronic photo frame.Additional examples of electronic devices include at least one of the following: various medical devices (such as a variety of portable medical measuring devices (e.g., blood sugar measuring devices, heart rate measuring devices, or body temperature measuring devices), magnetic resource angiography (MRA) devices, magnetic resource imaging (MRI) devices, computed tomography (CT) devices, imaging devices, or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDR), flight data recorders (FDR), automotive infotainment devices, marine electronic devices (such as marine navigation devices or gyrocompasses), avionics equipment, security equipment, vehicle head units, industrial or household robots, automatic teller machines (ATMs), point of sales (POS) devices, or Internet of Things (IoT). Things (IoT) devices (such as light bulbs, various sensors, electricity or gas meters, sprinklers, fire alarms, thermostats, street lights, toasters, fitness equipment, hot water tanks, heaters, or boilers). Other examples of electronic devices include at least a portion of a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). Note that according to various embodiments of the present disclosure, the electronic device may be one of the devices listed above or a combination thereof. According to some embodiments of the present disclosure, the electronic device may be a flexible electronic device. The electronic devices disclosed herein are not limited to the devices listed above, and may include new electronic devices depending on the development of technology.

[0015] In the following description, an electronic device is described with reference to the accompanying drawings according to various embodiments of the present disclosure. As used herein, the term "user" may refer to a person using an electronic device or another device (such as an artificial intelligence electronic device).

[0016] Definitions for other specific words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most, instances, such definitions apply to prior, as well as future uses of such defined words and phrases. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0018] Figure 1A An example network configuration including electronic devices according to the present disclosure is shown;

[0019] Figure 1B A block diagram of an electronic device according to the present disclosure is shown.

[0020] Figure 2 An example process supporting a fusion technique for combining the most significant bits and the least significant bits of image data according to the present disclosure is shown;

[0021] Figure 3 An example architecture supporting a fusion technique for combining the most significant bits and the least significant bits of image data according to the present disclosure is shown;

[0022] Figure 4 and Figure 5 It shows that according to the present disclosure Figure 3 Example spatially-adaptive denormalization (SPADE) functionality in the architecture of

[0023] Figure 6 The use of the present disclosure is shown Figure 3 An example of an architecture that separates raw image data into color channels for processing;

[0024] Fig. 7A and Figure 7B shows example results that may be obtained using a fusion technique for combining the most significant bits and the least significant bits of image data in accordance with the present disclosure;

[0025] Figure 8 An example method for combining the most significant bits and the least significant bits of image data according to the present disclosure is shown; and

[0026] Fig. 9 A flow chart for combining the most significant bits and the least significant bits of image data according to the present disclosure is shown. DETAILED DESCRIPTION

[0027] Figures 1 to 2 discussed below Figure 8 And various embodiments of the present disclosure are described with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to these embodiments, and all variations and / or equivalents or alternatives thereof also belong to the scope of the present disclosure. Throughout the specification and the accompanying drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0028] As mentioned above, digital cameras in mobile electronic devices (such as smart phones and tablet computers) have been increasing in performance over time. Among them, the resolution of images captured using digital cameras in mobile electronic devices is increasing rapidly. It is expected that digital cameras in mobile electronic devices will soon be able to generate images with hundreds of megapixels or more image data. Not only do these images need to be stored in memory, but they also usually need to be converted from the original image data format to a different image data format (such as red-green-blue or "RGB" image color space) for display to the user. The conversion process usually involves the use of additional memory space in order to process and convert the original image data. As a result, capturing, processing and storing images with increasing resolution can be time-consuming and require a large amount of processing resources and memory resources.

[0029] In one approach, the raw image data values ​​can be unpacked by expanding the number of bits of each data value in the image from 10 bits to 16 bits. The image is then segmented into smaller blocks for parallel processing, and the processed blocks are subsequently combined to produce a processed image. However, unpacking the raw data values ​​in the image may require a lot of processing time and memory resources. Moreover, processing the individual blocks using structures such as the UNET model may not produce a high-quality RGB image. In addition, when processing high-resolution image data, machine learning-based methods may suffer from slow processing times and large storage requirements.

[0030] The present disclosure provides a fusion technique for combining the most significant bit and the least significant bit of image data in image processing or other applications. As described in more detail below, raw image data can be obtained, wherein the raw image data includes data values. Each data value includes a most significant bit and a least significant bit, such as when each data value includes eight most significant bits and two least significant bits. The raw image data is provided to a trained machine learning model, and the processed image data is generated using the trained machine learning model. The trained machine learning model is trained to modulate at least one feature map associated with the most significant bit of the data value based on the least significant bit of the data value. This produces a fusion of the most significant bit and the least significant bit of the data value, thereby allowing the raw image data to be converted into different formats (such as when converted into RGB data). An image based on the processed image data can be presented on a display such as a user's mobile device, or an image based on the processed image data can be used in any suitable manner.

[0031] In this way, high-resolution raw image data containing a very large number of data values ​​can be quickly converted into another format for display or other purposes. The trained machine learning model used here can represent a lightweight machine learning model, which allows the trained machine learning model to be used in a faster time and use less processing resources and memory resources. For example, the trained machine learning model can be used without unpacking the data value to include additional bits, which can save a lot of processing resources and memory resources. The trained machine learning model can also be used to generate high-quality RGB images or other high-quality images based on the raw image data. Among them, this is because the most significant bit of the data value in the raw image data may contain many or most of the visual features in the image and can be used to determine one or more feature maps of the image. Since the most significant bit of the data value in the raw image data may lack fine texture details or other fine details in the image, the fusion of the most significant bit and the least significant bit of the data value in the raw image data can help restore these fine image details to the converted image data. In addition, the fusion process can be implemented on a device (such as on an end user's device) without transmitting the image data to an external destination for processing.

[0032] Note that while the following discussion may often describe the fusion techniques of the present disclosure in terms of converting raw image data into RGB image data, the fusion techniques herein may also be used for any other suitable conversion process or other process. For example, the fusion techniques herein may be used to convert raw image data to any suitable color space or other image data domain. Therefore, the following references to RGB image data are for illustration only and may be easily changed depending on the implementation. Moreover, while the fusion techniques of the present disclosure may often be described as being performed using a machine learning model on an end user's device, the fusion techniques of the present disclosure may also be performed using any (multiple) other suitable devices.

[0033] Figure 1A An example network configuration 100 including electronic devices according to the present disclosure is shown. Figure 1A The embodiment of the network configuration 100 shown is for illustration only. Other embodiments of the network configuration 100 may be used without departing from the scope of the present disclosure.

[0034] According to an embodiment of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In some embodiments, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 includes circuits for connecting the components 120-180 to each other and for transmitting communications (such as control messages and / or data) between the components.

[0035] The processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSP), application specific integrated circuits (ASIC), or field programmable gate arrays (FPGA). In some embodiments, the processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processor unit (GPU). The processor 120 is capable of performing control on at least one other component of the electronic device 101 and / or performing operations or data processing related to communication or other functions. As described below, the processor 120 can be used to perform a fusion technique for combining the most significant bit and the least significant bit of image data (such as converting raw image data into RGB image data or other image data for display or other purposes).

[0036] The memory 130 may include volatile and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or programs 140. The programs 140 include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0037] The kernel 141 can control or manage system resources (such as bus 110, processor 120 or memory 130) for executing operations or functions implemented in other programs (such as middleware 143, API 145 or application 147). The kernel 141 provides an interface that allows the middleware 143, API 145 or application 147 to access the various components of the electronic device 101 to control or manage system resources. The application 147 may include one or more applications for executing a fusion technology for combining the most significant bit and the least significant bit of the image data (such as converting the original image data into RGB image data or other image data for display or other purposes). These functions can be performed by a single application or multiple applications (each application implements one or more of these functions). For example, the middleware 143 can act as a relay to allow the API 145 or application 147 to transmit data with the kernel 141. Multiple applications 147 can be provided. The middleware 143 is capable of controlling work requests received from the applications 147, such as by assigning a priority of using system resources (such as the bus 110, the processor 120, or the memory 130) of the electronic device 101 to at least one of the plurality of applications 147. The API 145 is an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (such as a command) for archive control, window control, image processing, or text control.

[0038] The I / O interface 150 serves as an interface that can transmit, for example, a command or data input from a user or other external device to (multiple) other components of the electronic device 101. The I / O interface 150 can also output commands or data received from (multiple) other components of the electronic device 101 to the user or other external devices.

[0039] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum-dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen, and may receive, for example, touch, gesture, proximity, or hovering input using an electronic pen or a part of the user's body.

[0040] For example, the communication interface 170 can establish communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for sending and receiving signals (such as images).

[0041] The electronic device 101 also includes one or more sensors 180, wherein the one or more sensors 180 can measure physical quantities or detect the activation state of the electronic device 101, and convert the measured or detected information into electrical signals. For example, the one or more sensors 180 may include one or more cameras or other imaging sensors that can be used to capture images of a scene. (Multiple) sensors 180 may also include one or more buttons for touch input, one or more microphones, gesture sensors, gyroscopes or gyro sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (such as RGB sensors), biophysical sensors, temperature sensors, humidity sensors, lighting sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalogram (EEG) sensors, electrocardiogram (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors, or fingerprint sensors. The sensor(s) 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. In addition, the sensor(s) 180 may include a control circuit for controlling at least one of the sensors included here. Any of these sensor(s) 180 may be located within the electronic device 101.

[0042] The first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) in which an electronic device may be installed. When the electronic device 101 is installed in the electronic device 102 (such as an HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without involving a separate network. The electronic device 101 may also be an augmented reality wearable device including one or more cameras, such as glasses.

[0043] The wireless communication can use, for example, at least one of long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (5G), millimeter wave or 60 GHz wireless communication, wireless USB (universal serial bus), code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM) as a cellular communication protocol. The wired connection can include, for example, at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). The network 162 includes at least one communication network, such as a computer network (eg, a local area network (LAN) or a wide area network (WAN)), the Internet, or a telephone network.

[0044] The first external electronic device 102 and the second external electronic device 104 and the server 106 may all be devices of the same or different types as the electronic device 101. According to certain embodiments of the present disclosure, the server 106 includes a group of one or more servers. Moreover, according to certain embodiments of the present disclosure, all or some operations performed on the electronic device 101 may be performed on another or more other electronic devices (such as the electronic devices 102 and 104 or the server 106). In addition, according to certain embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform some functions or services, the electronic device 101 may request another device (such as the electronic devices 102 and 104 or the server 106) to perform at least some functions associated therewith, instead of performing the functions or services by itself or additionally. Other electronic devices (such as the electronic devices 102 and 104 or the server 106) are capable of performing the requested functions or additional functions and transmitting the results of the execution to the electronic device 101. The electronic device 101 may provide the requested functions or services by processing the received results as is or additionally. To this end, for example, cloud computing, distributed computing, or client-server computing technologies may be used. Although Figure 1A The electronic device 101 is shown to include a communication interface 170 to communicate with an external electronic device 104 or a server 106 via a network 162 , but according to some embodiments of the present disclosure, the electronic device 101 may also operate independently without a separate communication function.

[0045] The server 106 may include components that are the same or similar to the electronic device 101 (or a suitable subset thereof). The server 106 may support driving the electronic device 101 by performing at least one operation (or function) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101. As described below, the server 106 may be used to perform a fusion technique for combining the most significant bit and the least significant bit of the image data (such as to convert the raw image data into RGB image data or other image data for display or other purposes).

[0046] although Figure 1A An example of a network configuration 100 including an electronic device 101 is shown, but Figure 1A Various changes may be made. For example, network configuration 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems come in a variety of configurations, and Figure 1A The scope of the present disclosure is not limited to any particular configuration. Figure 1A One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may also be used in any other suitable system.

[0047] Figure 1B A block diagram of an electronic device according to the present disclosure is shown.

[0048] like Figure 1B As shown, the electronic device 101 may include at least one processor 120 and a memory 130. In some embodiments, the electronic device 101 excludes at least one of these components, or may add at least one other component. It should be noted that Figure 1B This is merely one example of a particular implementation and is intended to illustrate the types of components that may be included as part of the electronic device 101. According to an embodiment of the present disclosure, the electronic device 101 is included in a network configuration.

[0049] The processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). In some embodiments, the processor 1010 includes one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processor unit (GPU). The processor 120 is capable of performing control on at least one of the other components of the electronic device 101 and / or performing operations or data processing related to communication or other functions. As described below, the processor 120 can be used to perform a fusion technique for combining the most significant bit and the least significant bit of image data (such as converting raw image data into RGB image data or other image data for display or other purposes).

[0050] The memory 130 may include volatile and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or programs. The program includes, for example, a kernel, middleware, an application programming interface (API) and / or an application program (or "application"). At least a portion of the kernel, middleware or API may be represented as an operating system (OS).

[0051] Figure 2 An example process 200 is shown that supports a fusion technique for combining the most significant bits and the least significant bits of image data according to the present disclosure. For ease of explanation, Figure 2 The process 200 shown is described as consisting of Figure 1A The network configuration 100 is performed by the electronic device 101, however, Figure 2 The illustrated process 200 may be used with any other suitable device(s) and may be used in any other suitable system(s).

[0052] like Figure 2 As shown, process 200 is generally used to receive and process raw image data 202. Raw image data 202 can be obtained from any (multiple) suitable sources, such as when raw image data 202 is generated by at least one camera or other imaging sensor 180 of electronic device 101 during image capture operation. Raw image data generally refers to image data that has hardly undergone any processing after being captured. The availability of raw image data is useful in many cases because the raw image data can be subsequently processed to achieve the desired effect in the output image. For example, in many cases, raw image data 202 can have a wide dynamic range or a wide color gamut, which is narrowed during the image processing operation to produce a still or video image suitable for display or other purposes. Raw image data 202 can represent image data in one or more images, and each image can have any suitable resolution (such as image data up to hundreds of megapixels or more).

[0053] Each data value in the raw image data 202 includes a specified total number of bits, and these bits include a specified number of most significant bits and a specified number of least significant bits. The total number of bits in each data value of the raw image data 202 may vary depending on the implementation, and may vary based on the settings for image capture. For example, in some embodiments, each data value of the raw image data 202 may include ten bits. Moreover, the number of most significant bits and the number of least significant bits in each data value of the raw image data 202 may vary depending on the implementation, and may vary based on the settings for image capture. For example, in some embodiments, each data value of the raw image data 202 may include eight most significant bits and two least significant bits. However, note that these bit values ​​are merely examples and may vary as needed or desired.

[0054] The raw image data 202 is provided to a most significant bit (MSB) processing function 204 and a least significant bit (LSB) processing function 206. Each processing function 204 and 206 can extract desired bits from the data values ​​of the raw image data 202 for subsequent fusion. Each processing function 204 and 206 can also optionally perform one or more desired pre-processing operations on the bits of the raw image data 202. In some embodiments, the raw image data 202 can have a RAW10 data format. This format is used to represent a set of four image data values ​​within five data bytes (such as in the following manner).

[0055] Table 1.

[0056]

[0057] As can be seen here, the format uses each of the first four bytes to store eight of the ten bits of four image data values. The fifth byte is used to store two of the ten bits of four image data values. The format can be repeated using additional sets of five data bytes to store additional sets of four image data values.

[0058] In this particular example, MSB processing function 204 may process raw image data 202 to extract the most significant bit of each data value, and LSB processing function 206 may process raw image data 202 to extract the least significant bit of each data value. These extractions may be readily accomplished if eight most significant bits and two least significant bits are used for each data value in raw image data 202, since the data format already separates the most significant bits (in the first four data bytes shown above) from the least significant bits (in the last data byte shown above). However, other data formats for raw image data 202 may be used, in which case processing functions 204 and 206 may be used to extract the most significant bits and the least significant bits of the data values ​​in raw image data 202 in other ways.

[0059] The fusion function 208 is used to combine the most significant bits and the least significant bits of the raw image data 202 to produce converted image data 210, such as RGB image data. The fusion function 208 here can be implemented using at least one trained machine learning model (such as at least one machine learning model that includes various convolutions, deconvolutions or upsampling and other layers for processing image data). As described in more detail below, the machine learning model can be trained to generate one or more feature maps associated with the most significant bits of the data values ​​in the raw image data 202 for each raw image being processed. The machine learning model can also be trained to modulate one or more feature maps based on the least significant bits of the data values ​​in the raw image data 202 for each raw image being processed. This produces a fusion of the most significant bits and the least significant bits of the data values ​​in the raw image data 202 for each raw image being processed, thereby generating converted image data 210 for each raw image.

[0060] like Figure 2As can be seen in the process 200, the process 200 provides a method in which the most significant bits of the data values ​​in the raw image data 202 and the least significant bits of the data values ​​in the raw image data 202 can be processed separately. This allows the most significant bits and the least significant bits of the data values ​​in the raw image data 202 to be processed in an appropriate manner, rather than trying to force all bits of the raw image data 202 to be processed in the same manner. In addition, as described above, the most significant bits of the data values ​​in the raw image data 202 may contain many or most visual features in the raw image being processed and can be used to determine one or more feature maps of the raw image. However, the most significant bits of the data values ​​in the raw image data 202 may lack fine texture details or other fine details in the raw image. To help compensate for this, the fusion function 208 can modulate the (multiple) feature maps associated with the most significant bits based on the least significant bits of the data values ​​in the raw image data 202. This helps to restore fine image details to the converted image data 210 for each raw image being processed.

[0061] Details of an example implementation of a machine learning model for performing the fusion function 208 are provided below. Note that these details are merely examples and do not limit the scope of the present disclosure to the specific machine learning model implementation described below. In general, any suitable one or more machine learning model architectures may be used to implement the fusion function 208.

[0062] It should be noted that Figure 2 shown or concerning Figure 2 The described functionality may be implemented in any suitable manner in the electronic device 101, the server 106, or other device(s). For example, in some embodiments, Figure 2 shown or concerning Figure 2 At least some of the described functionality may be implemented or supported using one or more software applications or other software instructions executed by a processor 120 of the electronic device 101, server 106, or other device(s). Figure 2 shown or concerning Figure 2 At least some of the functions described may be implemented or supported using dedicated hardware components. Figure 2 shown or concerning Figure 2 The functions described may be performed using any suitable hardware, or any suitable combination of hardware and software / firmware instructions.

[0063] although Figure 2 One example of a process 200 that supports a fusion technique for combining the most significant bits and the least significant bits of image data is shown, but may be used for Figure 2For example, the converted image data 210 may be post-processed in any suitable manner prior to display or other use of the converted image data 210.

[0064] Figure 3 An example architecture 300 is shown that supports a fusion technique for combining the most significant bits and the least significant bits of image data according to the present disclosure. For example, the architecture 300 herein may represent an example implementation of the fusion function 208 in the process 200 described above. For ease of explanation, Figure 3 The illustrated architecture 300 is described as Figure 1A The network configuration 100 is implemented on or supported by the electronic device 101. However, Figure 3 The illustrated architecture 300 may be used with any other suitable device(s) and may be used in any other suitable system(s).

[0065] like Figure 3 As shown, architecture 300 receives most significant bit (MSB) image data 302 and least significant bit (LSB) image data 304. MSB image data 302 may be provided by MSB processing function 204, and LSB image data 304 may be provided by LSB processing function 206. In some cases, MSB image data 302 may include the eight most significant bits of the data values ​​in raw image data 202, and LSB image data 304 may include the two least significant bits of the data values ​​in raw image data 202. However, the number of most significant bits and least significant bits may vary depending on the implementation.

[0066] As shown in this example, the architecture 300 is implemented using a convolutional neural network (CNN) as the selected machine learning model structure. Here, the convolutional neural network is implemented using an encoder formed by a set of convolutional layers 306a-306d and a decoder formed by a set of deconvolutional layers or upsampling layers 308a-308c. Each convolutional layer 306a-306d applies a convolution function to its input to generate its output. The convolutional layers 306a-306d generally represent a convolutional neuron layer that applies a convolution function, wherein the convolution function simulates the response of each neuron to a visual stimulus. Each neuron typically applies a certain function to its input value (often by weighting different input values ​​differently) to generate an output value. Each convolutional layer 306a-306d can be associated with an activation function, which can apply a specific function or operation to the output value from the neuron to produce the final output of the convolutional layer. In this example, the first convolutional layer 306a receives and processes the MSB image data 302 to produce an initial feature map 310 that identifies features of the MSB image data 302. Each of the remaining convolutional layers 306b-306d receives and processes a feature map 310 from a previous convolutional layer 306a-306c or a modified version of the feature map 310. The output of the last convolutional layer 306d represents a final encoded feature map that encodes features of the MSB image data 302.

[0067] The final encoded feature map or a modified version thereof is processed using a set of deconvolutional or upsampling layers 308a-308c. Each deconvolutional or upsampling layer 308a-308c applies a deconvolutional or upsampling function to its input to generate its output. In this example, the first deconvolutional or upsampling layer 308a receives and processes the final encoded feature map or a modified version thereof that is provided by the convolutional layer 306d to encode the MSB image data 302. Each of the remaining deconvolutional or upsampling layers 308b-308c receives and processes the output from the previous deconvolutional or upsampling layer 308a-308b or a modified version thereof. The output of the last deconvolutional or upsampling layer 308c represents the converted image data 210 for each original image represented by the original image data 202.

[0068] As described above, if only the MSB image data 302 is used in the conversion process, the converted image data 210 will lose the fine image details contained in the original image data 202. To help compensate for this, the architecture 300 herein includes a plurality of spatially-adaptive denormalization (SPADE) functions 312a-312d. Each SPADE function 312a-312d may receive a feature map 310 and LSB image data 304 as inputs, and the SPADE functions 312a-312d modulate the feature map 310 based on the LSB image data 304. As a result, each SPADE function 312a-312d may be used to denormalize the batch-normalized convolutional features of the associated feature map 310. In effect, the SPADE functions 312a-312d fuse the LSB image data 304 with the features of the MSB image data 302, which helps to recover finer image details in the features of the MSB image data 302. The output of each SPADE function 312a-312d represents a fused feature map, which can be provided to the next convolutional layer or deconvolution layer.

[0069] As can be seen in this example, in some embodiments, SPADE functions 312a-312d can be included in or used with higher layers of the encoder (defined by later convolutional layers in the encoder) and included in or used with lower layers of the decoder (defined by earlier deconvolutional or upsampling layers in the decoder). Note that the number and location of SPADE functions 312a-312d can vary depending on the implementation. Figure 4 and Figure 5 One example implementation of the SPADE functions 312a-312d is described. Note also that the various SPADE functions 312a-312d may use the same LSB image data 304 to help ensure that finer image details are recovered after different convolution operations and deconvolution operations or upsampling operations.

[0070] In this example, one or more skip connections 316 may optionally be used to provide outputs from one or more convolutional layers 306a-306c to one or more corresponding deconvolutional or upsampling layers 308a-308c. Each skip connection 316 permits the feature map 310 generated by the convolutional layer 306a-306c to be provided directly to the corresponding deconvolutional or upsampling layer 308a-308c without passing through any intermediate convolutional or deconvolutional / upsampling layers. This may help facilitate improved information flow in the architecture 300 and allow the deconvolutional or upsampling layers 308a-308c to combine lower resolution feature maps with higher resolution feature maps when performing deconvolutional or upsampling operations.

[0071] It should be noted that Figure 3 shown or concerning Figure 3 The described functionality may be implemented in any suitable manner in electronic device 101, server 106, or other device. For example, in some embodiments, Figure 3 shown or concerning Figure 3 At least some of the functions described may be implemented or supported using one or more software applications or other software instructions executed by a processor 120 of the electronic device 101, server 106, or other device(s). Figure 3 shown or concerning Figure 3 At least some of the described functionality may be implemented or supported using dedicated hardware components. Figure 3 shown or concerning Figure 3 The functions described may be performed using any suitable hardware, or any suitable combination of hardware and software / firmware instructions.

[0072] although Figure 3 An example of an architecture 300 that supports a fusion technique for combining the most significant bits and the least significant bits of image data is shown, but may also be used for Figure 3 Make various changes. For example, Figure 3 The various components and functions in the architecture 300 may be combined, further subdivided, duplicated, or rearranged as required. Moreover, one or more additional components and functions may be included if needed or desired. As specific examples, more or fewer convolutional layers and deconvolutional layers or upsampling layers may be used in the architecture 300, and more or fewer SPADE functions may be used in the architecture 300.

[0073] Figure 4 and Figure 5 It shows that according to the present disclosure Figure 3 Example SPADE functions 312a-312d in the architecture 300. Figure 4As shown, the SPADE functions 312a-312d include two convolutional layers 402a-402b, two SPADE layers 404a-404b, and two rectified linear unit (ReLU) activation layers 406a-406b. Each convolutional layer 402a-402b generally operates to apply a convolution to its input data, where the input data represents features associated with the MSB image data 302. For example, in some embodiments, each convolutional layer 402a-402b may apply a 3×3 convolution to its input data. Each SPADE layer 404a-404b uses the LSB image data 304 to modulate the convolved features associated with the MSB image data 302. As a result, each SPADE layer 404a-404b may output a fusion of features associated with the MSB image data 302 and the LSB image data 304. Each of the ReLU activation layers 406a-406b generally operates to ensure that none of its outputs is negative, such as by selecting (for each input data value) the greater of that value or zero.

[0074] like Figure 5 As shown, each SPADE layer 404a-404b includes convolutional layers 502, 504, and 506. Convolutional layer 502 may apply an initial convolution to LSB image data 304, and convolutional layers 504 and 506 may apply additional convolutions to the output of convolutional layer 502. Convolutional layer 504 here generates an output parameter γ, and convolutional layer 506 here generates an output parameter β. As described below, parameter γ is used to scale the eigenvalues ​​associated with MSB image data 302, and parameter β is used to shift the eigenvalues ​​associated with MSB image data 302. Here, parameters γ and β are based on and determined using LSB image data 304.

[0075] In this example, a feature map 310 (produced by one of the convolutional layers or deconvolutional / upsampling layers) representing features of the MSB image data 302 is provided to a convolutional layer 508. The convolutional layer 508 generally applies batch normalization to the feature map 310, which can center and rescale the values ​​of the feature map 310. The batch-normalized version of the feature map 310 is provided to a multiplier 510, which multiplies the feature values ​​in the batch-normalized version of the feature map 310 by a parameter γ. This rescales the feature values ​​in the batch-normalized version of the feature map 310 based on the LSB image data 304. The rescaled batch-normalized version of the feature map 310 produced by the multiplier 510 is provided to an adder 512, which adds the parameter β to the feature values ​​in the rescaled batch-normalized version of the feature map 310. This shifts the feature values ​​in the rescaled batch normalized version of the feature map 310 based on the LSB image data 304 and produces modulated MSB features 514. The modulated MSB features 514 represent the features of the feature map 310 after modulation is performed based on the LSB image data 304.

[0076] Figure 3 The various components of the architecture 300 shown (including Figure 4 and Figure 5Each instance of the SPADE functions 312a-312d shown) can be trained in any suitable manner. For example, in some cases, training data and reference real data can be obtained. The training data can include the original image data 202, and the reference real data can include converted image data 210 representing the original image data 202 after conversion (such as after conversion from the original image domain to the RGB image domain). As a specific example, the reference real data can be generated based on the training data using one or more conventional or customized raw to RGB conversion algorithms, such as using one or more raw to RGB conversion algorithms that generally run slower or require more processing resources or memory resources than algorithms that are generally available on end-user devices. The training data can be provided to the architecture 300, and the architecture 300 can be used to generate the converted image data 210 based on the training data. The converted image data 210 can be compared with the reference real data, and the difference or error between the converted image data 210 and the reference real data can be identified and used to calculate the overall error or loss of the architecture 300. If the calculated loss exceeds the threshold, the parameters of one or more layers in the architecture 300 (such as convolutional layers or deconvolutional layers / upsampling layers, including convolutional layers or deconvolutional layers / upsampling layers in SPADE functions 312a-312d) can be adjusted, and the training data can be processed again, or the updated architecture 300 can be used to process new training data. This produces new transformed image data 210, which can be compared with the ground truth data to calculate the updated loss. Ideally, the calculated loss decreases over time and eventually drops below the threshold, indicating that the architecture 300 has been trained to accurately (at least within the desired accuracy represented by the threshold) generate transformed image data 210 based on the original image data 202. In some embodiments, the loss function used to train the architecture 300 can represent a combination of an L1 loss and a structural similarity (SSIM) loss.

[0077] although Figure 4 and Figure 5 Shows Figure 3 312a-312d in the architecture 300, but may also be used for Figure 4 and Figure 5 Make various changes. For example, Figure 4 and Figure 5The various components and functions in the SPADE functions 312a-312d may be combined, further subdivided, duplicated, or rearranged as required. Moreover, one or more additional components and functions may be included if needed or desired. As a specific example, more or fewer convolutional layers may be used in the SPADE functions 312a-312d. Furthermore, the use of 3×3 convolutional layers 402a-402b and ReLU activation layers 406a-406b are examples of how certain operations in the SPADE functions 312a-312d may be implemented, but these operations may be implemented in any other suitable manner, including using different convolution kernel sizes or activation functions.

[0078] Figure 6 The use of the present disclosure is shown Figure 3 An example of an architecture 300 that separates raw image data 600 into color channels for processing. In some cases, the raw image data 600 may represent raw image data captured using an electronic device 101 (such as by using one or more imaging sensors 180). In this example, the raw image data 600 may generally be divided into four groups of image data 602a-602d, where each group 602a-602d is associated with a single primary color. For example, in some embodiments, the raw image data 600 may be captured using a Bayer color filter array, and the raw image data 600 may include a green group 602a-602b, a blue group 602c, and a red group 602d. The Bayer color filter array may be used with at least one imaging sensor 180 to produce a green pixel value that is approximately twice the red or blue pixel value, which is more consistent with the actual physiology of the human eye. However, note that other types of color filter arrays may optionally be used with the architecture 300. Here, the original image data 600 is referred to as having a size of H×W×1, where H refers to the pixel height of the original image data 600, W refers to the pixel width of the original image data 600, and 1 refers to the depth of the original image data 600.

[0079] In this example, the raw image data 600 may be unpacked by splitting the raw image data 600 into a plurality of color channels 604. Here each color channel 604 is associated with a portion of the data values ​​from the raw image data 600 (i.e., the data values ​​in one group 602a-602d). The color channels 604 are referred to here as having a total size of H / 2×W / 2×4, meaning that each of the four color channels 604 has a quarter of the data values ​​from the raw image data 600. Note that this may be done for both the most significant bits and the least significant bits of the data values ​​in the raw image data 600.

[0080] The data values ​​in each color channel 604 may be provided to the architecture 300 for independent processing, and the resulting data produced by the architecture 300 may be combined to produce a final output image. For example, each color channel 604 containing the most significant bits of the data values ​​in the raw image data 600 may be used to generate one or more feature maps 310, and the one or more feature maps 310 may be modulated using the color channel 604 containing the corresponding least significant bits of the data values ​​in the raw image data 600. One potential advantage of segmenting the raw image data 600 like this or in some other suitable manner is that the segmentation may significantly reduce the runtime memory required by the architecture 300 because the spatial size of the raw image data processed by the architecture 300 has been reduced by a factor of four. As described above, other approaches unpack the raw image data by increasing the number of bits for each data value, which significantly increases the runtime memory requirements.

[0081] although Figure 6 Shows the use of Figure 3 The architecture 300 of FIG. 1 is an example of separating raw image data 600 into color channels 604 for processing, but it is also possible to Figure 6 Various changes may be made. For example, the raw image data 600 may be segmented in any other suitable manner. Moreover, segmentation of the raw image data 600 is optional and need not occur in all embodiments of the architecture 300.

[0082] Fig. 7A and Figure 7B 1 shows example results that can be obtained using a fusion technique for combining the most significant bits and the least significant bits of image data in accordance with the present disclosure. More specifically, Fig. 7A and Figure 7B Shows that you can use Figure 3 3. However, note that architecture 300 may be used in any other suitable manner.

[0083] like Fig. 7A As shown, image 702 represents an image formed using the most significant bits of the data values ​​in the original data image. Figure 7B As shown, image 704 represents an image formed based on the same most significant bits of the data values ​​in the original data image, but the most significant bits of the data values ​​have been modulated using the least significant bits of the data values ​​in the original data image. As can be seen here, by modulating the most significant bits of the data values ​​based on the least significant bits of the data values, many of the finer details that were lost using only the most significant bits of the data values ​​have been recovered. Figure 7B The resulting image 704 shown is readily suitable for display or other use. In addition, compared to previous RAW to RGB conversion techniques, Figure 7BThe resulting image 704 shown may be generated in a faster time and using fewer processing resources or memory resources.

[0084] although Fig. 7A and Figure 7B An example of a result that can be obtained using a fusion technique for combining the most significant bits and the least significant bits of image data is shown, but it is also possible to Fig. 7A and Figure 7B For example, the actual content of the original image data can vary greatly, and Fig. 7A and Figure 7B The scope of the present disclosure is not limited to any particular type of image content.In general, raw to RGB conversion or other conversions may be performed using any suitable image data associated with any suitable scene being imaged.

[0085] Figure 8 An example method 800 for combining the most significant bits and the least significant bits of image data according to the present disclosure is shown. For ease of explanation, Figure 8 The method 800 shown is described as comprising Figure 1A The network configuration 100 is executed by the electronic device 101, wherein the electronic device 101 can implement Figure 3 The architecture 300 shown and executed Figure 2 The process 200 shown. However, Figure 8 The illustrated method 800 may be performed by any other suitable device(s), and executed in any other suitable system(s).

[0086] like Figure 8As shown, in step S802, raw image data is obtained. This may include, for example, the processor 120 of the electronic device 101 using one or more imaging sensors 180 to obtain raw image data 202. The raw image data 202 includes data values, and each data value has a most significant bit and a least significant bit. In some cases, each data value may have eight most significant bits and two least significant bits. In step S804, the most significant bit and the least significant bit of the data value in the raw image data are identified. This may include, for example, the processor 120 of the electronic device 101 executing the MSB processing function 204 and the LSB processing function 206 to extract the most significant bit and the least significant bit of the data value in the raw image data 202. In step S806, the raw image data may optionally be divided into multiple color channels. This may include, for example, the processor 120 of the electronic device 101 unpacking the raw image data 202 to generate multiple color channels 604. This may be done to create multiple color channels 604 for the most significant bits of the data values ​​in the raw image data 202, and multiple color channels 604 for the least significant bits of the data values ​​in the raw image data 202. In the following discussion, it is assumed that segmentation has not yet occurred. However, if segmentation has already occurred, the following steps may be performed for data in different color channels, and the results associated with the different color channels may be combined.

[0087] At step S808, the most significant bits of the data values ​​in the raw image data are provided to the machine learning model to generate one or more MSB feature maps. This may include, for example, the processor 120 of the electronic device 101 providing the most significant bits of the data values ​​in the raw image data 202 to the fusion function 208, which in some cases may use a feature map having a Figure 3 The architecture 300 shown in FIG. 1 may be implemented by a convolutional neural network of the architecture 300. The architecture 300 may use various layers to generate one or more feature maps 310 based on the most significant bits of the data values ​​in the raw image data 202. At step S810, one or more MSB feature maps are modulated based on the least significant bits of the data values ​​in the raw image data. This may include, for example, the processor 120 of the electronic device 101 executing one or more SPADE functions 312a-312d, each of which may generate parameters γ and β based on the least significant bits of the data values ​​in the raw image data 202, and use these parameters to modulate features associated with the most significant bits of the data values ​​in the raw image data 202. Note that any suitable number of feature maps 310 may be generated and modulated here.

[0088] In step S812, the machine learning model generates converted image data representing the processed image. This may include, for example, the processor 120 of the electronic device 101 using the final layer of the architecture 300 to generate the converted image data 210. The converted image data 210 may represent the original image data 202 converted into a different domain (such as the RGB domain). In step 814, the processed image may be stored, output, or used in some way. This may include, for example, the processor 120 of the electronic device 101 causing the converted image data 210 to be displayed on the display 160 of the electronic device 101, saved to a camera roll stored in the memory 130 of the electronic device 101, or attached to a text message, email, or other communication to be sent from the electronic device 101. Of course, the processed image may be used in any other or additional manner.

[0089] although Figure 8 An example of a method 800 for combining the most significant bits and the least significant bits of image data is shown, but the same may also be done for Figure 8 For example, although shown as a series of steps, Figure 8 The various steps in may overlap, occur in parallel, occur in a different order, or occur any number of times.

[0090] Fig. 9 A flow chart for combining the most significant bit and the least significant bit of image data according to the present disclosure is shown. For ease of explanation, Fig. 9 The method 900 is described as comprising Figure 1A The network configuration 100 is executed by the electronic device 101, wherein the electronic device 101 can implement Figure 3 The architecture 300 shown and executed Figure 2 The process 200 shown. However, Fig. 9 The illustrated method 900 may be performed by any other suitable device(s), and executed in any other suitable system(s).

[0091] like Fig. 9As shown, in step S902, the electronic device 101 can obtain raw image data. This can include, for example, the processor 120 of the electronic device 101 using one or more imaging sensors 180 to obtain raw image data 202. The raw image data 202 includes data values, and each data value has a most significant bit and a least significant bit. In some cases, each data value can have eight most significant bits and two least significant bits. Identify the most significant bit and the least significant bit of the data value in the raw image data. This can include, for example, the processor 120 of the electronic device 101 executing the MSB processing function 204 and the LSB processing function 206 to extract the most significant bit and the least significant bit of the data value in the raw image data 202. The raw image data can be optionally divided into multiple color channels. This can include, for example, the processor 120 of the electronic device 101 unpacking the raw image data 202 to generate multiple color channels 604. This can be done so as to create multiple color channels 604 for the most significant bits of the data values ​​in the raw image data 202, and to create multiple color channels 604 for the least significant bits of the data values ​​in the raw image data 202. In the following discussion, it is assumed that segmentation has not yet occurred. However, if segmentation has already occurred, the following steps can be performed on the data in the different color channels, and the results associated with the different color channels can be combined.

[0092] At step S904, the electronic device may provide the raw image data to the trained machine learning model. The most significant bits of the data values ​​in the raw image data are provided to the trained machine learning model. This may include, for example, the processor 120 of the electronic device 101 providing the most significant bits of the data values ​​in the raw image data 202 to the fusion function 208, which in some cases may use a method having a Figure 3 The architecture 300 shown in FIG. 3 may be implemented using a convolutional neural network of the architecture 300. The architecture 300 may use various layers to generate one or more feature maps 310 based on the most significant bits of the data values ​​in the raw image data 202. One or more MSB feature maps are modulated based on the least significant bits of the data values ​​in the raw image data. This may include, for example, the processor 120 of the electronic device 101 executing one or more SPADE functions 312a-312d, each of which may generate parameters γ and β based on the least significant bits of the data values ​​in the raw image data 202, and use these parameters to modulate features associated with the most significant bits of the data values ​​in the raw image data 202. Note that any suitable number of feature maps 310 may be generated and modulated here.

[0093] At step S906, the electronic device may use the trained machine learning model to generate processed image data. The machine learning model generates processed image data. This may include, for example, the processor 120 of the electronic device 101 using the final layer of the architecture 300 to generate converted image data 210. The converted image data 210 may represent the original image data 202 converted into a different domain (such as an RGB domain).

[0094] In step S908, the electronic device may present an image based on the processed image data. The image is presented based on the processed image data. The processed image may be stored, output, or used in some manner. This may include, for example, the processor 120 of the electronic device 101 causing the converted image data 210 to be displayed on the display 160 of the electronic device 101, saved to a camera album stored in the memory 130 of the electronic device 101, or attached to a text message, email, or other communication to be sent from the electronic device 101. Of course, the processed image may be used in any other or additional manner.

[0095] According to an embodiment of the present disclosure, a method includes obtaining raw image data, wherein the raw image data includes data values, each data value having a most significant bit and a least significant bit. The method also includes providing the raw image data to a trained machine learning model and generating processed image data using the trained machine learning model. The method also includes presenting an image based on the processed image data. The trained machine learning model is trained to modulate a feature map associated with a most significant bit of the data value of the raw image data based on the least significant bit of the data value of the raw image data, so as to generate a fusion of the most significant bit and the least significant bit of the data value of the raw image data.

[0096] According to an embodiment of the present disclosure, the trained machine learning model includes one or more convolutional layers, wherein the one or more convolutional layers are configured to receive the least significant bits of the data values ​​of the original image data and generate one or more parameters for modulating the feature map associated with the most significant bits of the data values ​​of the original image data.

[0097] According to an embodiment of the present disclosure, the trained machine learning model is configured to perform spatially adaptive denormalization so as to modulate a feature map associated with the most significant bits of the data values ​​of the original image data based on the least significant bits of the data values ​​of the original image data.

[0098] According to an embodiment of the present disclosure, the trained machine learning model includes a convolutional layer and a deconvolutional layer or an upsampling layer, and the convolutional layer and the deconvolutional layer or the upsampling layer are configured to generate multiple feature maps associated with the most significant bits of the data values ​​of the original image data, and the trained machine learning model also includes at least one set of additional convolutional layers, and the at least one set of additional convolutional layers are configured to receive the least significant bits of the data values ​​of the original image data and determine one or more parameters for modulating the feature maps.

[0099] According to an embodiment of the present disclosure, the method also includes segmenting the original image data into multiple color channels to form multiple color channels for the most significant bits and multiple color channels for the least significant bits, and the trained machine learning model is trained to modulate the feature map associated with the multiple color channels for the most significant bits based on the multiple color channels for the least significant bits.

[0100] According to an embodiment of the present disclosure, the processed image data includes red, green, and blue (RGB) image data.

[0101] According to an embodiment of the present disclosure, each data value of the original image data includes eight most significant bits and two least significant bits.

[0102] According to an embodiment of the present disclosure, an electronic device includes at least one imaging sensor and at least one processing device. The at least one processing device is configured to obtain raw image data using the at least one imaging sensor, wherein the raw image data includes data values, each of which has a most significant bit and a least significant bit. The at least one processing device is also configured to provide the raw image data to a trained machine learning model and generate processed image data using the trained machine learning model. The at least one processing device is also configured to initiate presentation of an image based on the processed image data. The trained machine learning model is trained to modulate a feature map associated with the most significant bit of the data value of the raw image data based on the least significant bit of the data value of the raw image data, so as to generate a fusion of the most significant bit and the least significant bit of the data value of the raw image data.

[0103] According to an embodiment of the present disclosure, a machine-readable medium contains instructions, which, when executed, causes at least one processor of a device to perform operations corresponding to the above method.

[0104] Although the present disclosure has been described with reference to various exemplary embodiments, various changes and modifications may occur to those skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method comprising: Obtaining original image data, the original image data comprising data values, each data value having a most significant bit and a least significant bit; providing the raw image data to a trained machine learning model; generating processed image data using the trained machine learning model; as well as presenting an image based on the processed image data; wherein the trained machine learning model is trained to modulate a feature map associated with the most significant bit of the data value of the original image data based on the least significant bit of the data value of the original image data so as to generate a fusion of the most significant bit and the least significant bit of the data value of the original image data.

2. The method according to claim 1, wherein: The trained machine learning model comprises one or more convolutional layers configured to receive the least significant bits of the data values ​​of the original image data and generate one or more parameters for modulating the feature map associated with the most significant bits of the data values ​​of the original image data.

3. The method according to any one of claims 1 to 2, wherein: The trained machine learning model is configured to perform spatially adaptive denormalization so as to modulate the feature map associated with the most significant bits of the data values ​​of the raw image data based on the least significant bits of the data values ​​of the raw image data.

4. The method according to any one of claims 1 to 3, wherein: The trained machine learning model comprises convolutional layers and deconvolutional layers or upsampling layers configured to generate a plurality of feature maps associated with the most significant bits of the data values ​​of the original image data; and The trained machine learning model also includes at least one set of additional convolutional layers, which are configured to receive the least significant bits of the data values ​​of the original image data and determine one or more parameters for modulating the feature map.

5. The method according to any one of claims 1 to 4, further comprising: Splitting the original image data into a plurality of color channels to form a plurality of color channels for the most significant bits and a plurality of color channels for the least significant bits; Wherein, the trained machine learning model is trained to modulate the feature map associated with the multiple color channels for the most significant bits based on the multiple color channels for the least significant bits.

6. The method according to any one of claims 1 to 5, wherein: The processed image data includes red, green, and blue (RGB) image data.

7. The method according to any one of claims 1 to 6, wherein: Each data value of the original image data includes eight most significant bits and two least significant bits.

8. An electronic device (101), comprising: a memory for storing instructions (130); as well as At least one processor (120) is configured to execute the instructions to: obtaining raw image data using the at least one imaging sensor, the raw image data comprising data values, each data value having a most significant bit and a least significant bit; providing the raw image data to a trained machine learning model; generating processed image data using the trained machine learning model; as well as initiating rendering of an image based on the processed image data; wherein the trained machine learning model is trained to modulate a feature map associated with the most significant bit of the data value of the original image data based on the least significant bit of the data value of the original image data so as to generate a fusion of the most significant bit and the least significant bit of the data value of the original image data.

9. The electronic device (101) according to claim 8, wherein: The trained machine learning model comprises one or more convolutional layers configured to receive the least significant bits of the data values ​​of the original image data and generate one or more parameters for modulating the feature map associated with the most significant bits of the data values ​​of the original image data.

10. The electronic device (101) according to any one of claims 8 to 9, wherein: The trained machine learning model is configured to perform spatially adaptive denormalization so as to modulate the feature map associated with the most significant bits of the data values ​​of the raw image data based on the least significant bits of the data values ​​of the raw image data.

11. The electronic device (101) according to any one of claims 8 to 10, wherein: The trained machine learning model comprises convolutional layers and deconvolutional layers or upsampling layers configured to generate a plurality of feature maps associated with the most significant bits of the data values ​​of the original image data; and The trained machine learning model also includes at least one set of additional convolutional layers, which are configured to receive the least significant bits of the data values ​​of the original image data and determine one or more parameters for modulating the feature map.

12. The electronic device (101) according to any one of claims 8 to 11, wherein: The at least one processing device is further configured to split the raw image data into a plurality of color channels to form a plurality of color channels for the most significant bits and a plurality of color channels for the least significant bits; and The trained machine learning model is trained to modulate a feature map associated with the plurality of color channels for the most significant bits based on the plurality of color channels for the least significant bits.

13. The electronic device (101) according to any one of claims 8 to 12, wherein: The processed image data includes red, green, and blue (RGB) image data.

14. The electronic device (101) according to any one of claims 8 to 13, wherein: Each data value of the original image data includes eight most significant bits and two least significant bits.

15. A machine-readable medium containing instructions, which, when executed, cause at least one processor of a device to perform operations corresponding to the method according to any one of claims 1-7.

Images