Image generation method, apparatus and electronic device

Abstract

The application discloses an image generation method and device and electronic equipment, relates to the field of image processing, and is used for solving the problem of how to make the color of the generated high dynamic range image as close as possible to the color of a real scene. The method comprises the following steps: acquiring a spectral image of a first scene through a first camera supporting more light-sensitive wave bands than a second camera, and acquiring a color image of a second scene through the second camera, wherein the color image and the spectral image both contain a same target subject. Furthermore, the color deviation of a reconstructed spectral image of the color image is corrected according to the spectral image, and a plurality of corrected reconstructed spectral images are obtained; and a corrected color image with color closer to the color of the second scene than the color of the color image is generated through fusion according to the plurality of corrected reconstructed spectral images, that is, the corrected color image is a high dynamic range image relative to the color image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to an image generation method and device and electronic equipment. BACKGROUND

[0002] At present, a traditional camera can capture the same scene by using different exposure times, filter out the pixels of the effective area in each image according to a preset range of pixel values, exclude the pixels of overexposure and underexposure, and fuse multiple images of different exposure amounts to obtain a high dynamic range image with a wider dynamic range, more details and closer to the natural scene, that is, the high dynamic range imaging (HDRI) technology is used to make the color image generated by the traditional camera restore the real light of the natural scene as much as possible. However, the traditional camera has three light-sensitive wavebands of red, green and blue, and cannot capture and record the light of other light-sensitive wavebands in the natural world. The color of the color image captured by the traditional camera is different from the color in the real scene. If the multiple images of different exposure amounts fused have color deviation, the color of the high dynamic range image generated also has difference from the color in the real scene. Since the multispectral sensor can obtain light of multiple light-sensitive wavebands, the multispectral sensor can obtain a multispectral image with rich colors, that is, an image containing object information of a spectral image of multiple light-sensitive wavebands. However, the spatial resolution of the multispectral sensor is low, and the imaging speed is slow. In the photography application, it is not suitable to present the multispectral image or the color image generated according to the multispectral image to the user. Therefore, how to make the color of the generated high dynamic range image as close as possible to the color of the real scene is a problem to be solved. SUMMARY

[0003] The present application provides an image generation method and device and electronic equipment, which solve the problem of how to make the color of the generated high dynamic range image as close as possible to the color of the real scene.

[0004] Firstly, this application provides an image generation method, which can be applied to an electronic device or an image generation apparatus that supports the electronic device in implementing the method. Specifically, the method includes: after the electronic device receives a user's photo-taking command, a first camera and a second camera capture images of the same target subject; that is, the first camera acquires a spectral image of a first scene, and the second camera acquires a color image of a second scene, both the obtained color image and the spectral image containing the same target subject. Since the photosensitive band of the first camera is no less than that of the second camera, the spectral image contains more color information than the color image. Furthermore, the electronic device corrects the color shift of the reconstructed spectral image of the color image based on the spectral image, obtaining multiple corrected reconstructed spectral images; and fuses the multiple corrected reconstructed spectral images to generate a corrected color image, the color of which is closer to the color of the second scene than the color of the original color image.

[0005] Thus, by combining the second camera with a first camera that has a wider range of photosensitive wavelengths, the rich spectral information contained in the spectral image acquired by the first camera is fully utilized. This allows for the correction of the reconstructed spectral image of the color image in the spectral domain. The corrected reconstructed spectral image is then fused to generate a corrected color image, resulting in a high dynamic range color image with more refined and accurate colors compared to the color image acquired by the second camera. The image generation method provided in this application can be applied to fields such as mobile phone photography, remote sensing imaging, art preservation and reproduction, and medical imaging, effectively improving image quality and the user's photography experience.

[0006] In either case, the first camera and the second camera are both rear-facing cameras, or both the first camera and the second camera are front-facing cameras.

[0007] In practice, the first camera and the second camera can be implemented by different cameras, presenting two cameras to the outside world. In some embodiments, the first camera and the second camera can be located in different modules and operate independently. In other embodiments, the first camera and the second camera can be located in the same module.

[0008] In another possible design, the first and second cameras can be integrated internally within the electronic device, presenting only a single camera to the outside world. This means the first and second cameras reside on different chips within the same module and lens, with their operation controlled by a camera driver module to acquire color and spectral images. Alternatively, different pixel arrangements can be used, integrating traditional RGB pixels with other spectral pixels across the same chip. A driver module controls the operation of the multispectral sensor pixels and the RGB sensor pixels to acquire color and spectral images. In this way, the same camera can capture images of the same subject over a period of time with different parameter configurations, allowing the spectral image to correct color casts in the reconstructed spectral image, resulting in a more refined and accurate high dynamic range color image.

[0009] In one possible implementation, the color shift of the reconstructed spectral image is corrected based on the spectral image to obtain multiple corrected reconstructed spectral images. This includes correcting the pixel values ​​of matching pixels in the reconstructed spectral images based on the pixel values ​​of the spectral image, resulting in multiple corrected reconstructed spectral images. Matching pixels refer to pixels with the same image features within the overlapping region of the image that overlaps with the spectral image. Therefore, correcting the reconstructed spectral image at the pixel level effectively improves the precision and accuracy of the correction.

[0010] Understandably, a spectral image includes spectral images of N photosensitive bands, a color image includes M frames of color images, each with a different exposure level, and a reconstructed spectral image includes M x N reconstructed spectral images, each containing spectral images of N photosensitive bands from each of the M frames of color images, where N and M are both integers greater than or equal to 1.

[0011] Specifically, the process of correcting the pixel values ​​of matching pixels in the reconstructed spectral image based on the pixel values ​​of the spectral image to obtain multiple corrected reconstructed spectral images includes: for the pixel values ​​of the spectral image of each of the N photosensitive bands, correcting the pixel values ​​of matching pixels belonging to the same photosensitive band in the MxN reconstructed spectral images to obtain MxN corrected reconstructed spectral images. The multiple corrected reconstructed spectral images comprise MxN corrected reconstructed spectral images.

[0012] Understandably, the M x N reconstructed spectral images are divided into N sets, each representing one of the N photosensitive bands. Each set contains M reconstructed spectral images with the same photosensitive band. Reconstructed spectral images belonging to different sets have different photosensitive bands. Furthermore, using the pixel values ​​of the spectral images belonging to the same photosensitive band as the reconstructed spectral image set, the pixel values ​​of matching pixels in each reconstructed spectral image within the set are corrected. After correcting the reconstructed spectral images contained in the N sets, M x N corrected reconstructed spectral images are obtained.

[0013] Optionally, for the pixel values ​​of the spectral image of each of the N photosensitive bands, the pixel values ​​of matching pixels belonging to the same photosensitive band in the MxN reconstructed spectral images are corrected. This includes: for the pixel values ​​of the spectral image of each of the N photosensitive bands, the pixel values ​​of matching pixels belonging to the same photosensitive band within the effective region of the MxN reconstructed spectral images are corrected. The effective region is the area in the color image where the pixel values ​​are within a preset range, and the range of the effective region of the reconstructed spectral image is the same as the range of the effective region of the color image. This eliminates meaningless pixel values ​​in the reconstructed spectral image, corrects the pixel values ​​of pixels matching the spectral image within the effective region, and improves the image processing speed.

[0014] In another possible implementation, a corrected color image is generated by fusing multiple corrected reconstructed spectral images, including: for each of the N photosensitive bands, fusing the corrected reconstructed spectral images belonging to the same photosensitive band from the MxN corrected reconstructed spectral images to obtain N fused spectral images; and generating a corrected color image based on the N fused spectral images.

[0015] Understandably, taking each of the N photosensitive bands as a unit, the MxN corrected reconstructed spectral images are divided into N sets of corrected reconstructed spectral images, and the corrected reconstructed spectral images in each set are fused. For each of the N photosensitive bands, a fused spectral image is obtained, ultimately resulting in N fused spectral images.

[0016] Therefore, the reconstructed spectral images of color images with different exposure times are corrected to obtain corrected reconstructed spectral images under different exposure times. Then, the corrected reconstructed spectral images are fused in the spectral domain to obtain a high dynamic range fused spectral image, and a high dynamic range color image with more refined and accurate colors is generated based on the fused spectral image.

[0017] In another possible implementation, a corrected color image is generated by fusing multiple corrected reconstructed spectral images, including: for each of the M exposure durations, generating M intermediate color images based on the M x N corrected reconstructed spectral images; fusing the M intermediate color images to obtain a corrected high dynamic range color image.

[0018] Understandably, using each of the M exposure times as a unit, the M x N corrected reconstructed spectral images are divided into M sets of corrected reconstructed spectral images. Each set of corrected reconstructed spectral images contains N corrected reconstructed spectral images obtained from a color image with a single exposure time. Corrected reconstructed spectral images belonging to different sets are obtained from color images with different exposure times. Furthermore, an intermediate color image is generated based on the corrected reconstructed spectral images contained in each set, ultimately resulting in M ​​intermediate color images. These M intermediate color images are then fused to obtain the corrected high dynamic range color image.

[0019] In another possible implementation, the method further includes: using a reconstruction model to generate reconstructed spectral images of N photosensitive bands for each of the M frame color images, thereby obtaining MxN reconstructed spectral images.

[0020] The electronic device can train the reconstruction model based on the sample color image and the sample spectral image. It can calculate the loss function based on the sample spectral image and the predicted reconstructed spectral image output by the reconstruction model. When the loss function converges and the loss function value is less than or equal to the threshold, the parameters of the reconstruction model output by the reconstruction model in this case are determined as the parameters of the reconstruction model that is finally needed to generate the reconstructed spectral image based on the color image.

[0021] Secondly, an image generation apparatus is provided, the beneficial effects of which are described in the first aspect and will not be repeated here. The image generation apparatus has the function of implementing the behavior in the method example of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the image generation apparatus includes: an acquisition module and an image generation module. The acquisition module is used to acquire a spectral image of a first scene and a color image of a second scene. Both the color image and the spectral image contain the same target subject, and the acquisition module acquires at least as many photosensitive wavelengths as it acquires in the first scene as it acquires in the second scene. The image generation module is used to correct the color shift of the reconstructed spectral image of the color image based on the spectral image, obtaining multiple corrected reconstructed spectral images; and to fuse the multiple corrected reconstructed spectral images to generate a corrected color image.

[0022] In one possible implementation, when the image generation module corrects the color shift of the reconstructed spectral image of the color image based on the spectral image to obtain multiple corrected reconstructed spectral images, it is specifically used to: correct the pixel values ​​of matching pixels in the reconstructed spectral image based on the pixel values ​​of the spectral image to obtain multiple corrected reconstructed spectral images. Matching pixels refer to pixels in the reconstructed spectral image that have the same image features within the overlapping area of ​​the image that overlaps with the spectral image.

[0023] Understandably, a spectral image includes spectral images of N photosensitive bands, a color image includes M frames of color images, each with a different exposure level, and a reconstructed spectral image includes M x N reconstructed spectral images, each containing spectral images of N photosensitive bands from each of the M frames of color images, where N and M are both integers greater than or equal to 1.

[0024] Specifically, when the image generation module corrects the pixel values ​​of matching pixels in the reconstructed spectral image based on the pixel values ​​of the spectral image to obtain multiple corrected reconstructed spectral images, it is specifically used to: for the pixel values ​​of the spectral image of each of the N photosensitive bands, correct the pixel values ​​of matching pixels belonging to the same photosensitive band in the MxN reconstructed spectral images to obtain MxN corrected reconstructed spectral images, and the multiple corrected reconstructed spectral images include the MxN corrected reconstructed spectral images.

[0025] Optionally, when the image generation module corrects the pixel values ​​of matching pixels belonging to the same photosensitive band in the MxN reconstructed spectral images for the pixel values ​​of the spectral images of each of the N photosensitive bands, it specifically performs the following: For the pixel values ​​of the spectral images of each of the N photosensitive bands, it corrects the pixel values ​​of matching pixels belonging to the same photosensitive band in the effective region of the MxN reconstructed spectral images. The effective region is the region in the color image where the pixel values ​​are within a preset range, and the range of the effective region of the reconstructed spectral image is the same as the range of the effective region of the color image.

[0026] In another possible implementation, when the image generation module generates a corrected color image by fusing multiple corrected reconstructed spectral images, it specifically performs the following: for each of the N photosensitive bands, it fuses the corrected reconstructed spectral images belonging to the same photosensitive band from the MxN corrected reconstructed spectral images to obtain N fused spectral images; and generates a corrected color image based on the N fused spectral images.

[0027] In another possible implementation, when the image generation module generates a corrected color image by fusing multiple corrected reconstructed spectral images, it is specifically used to: generate M intermediate color images based on M x N corrected reconstructed spectral images; and fuse the M intermediate color images to obtain the corrected color image.

[0028] In another possible implementation, the image generation module is further used to utilize a reconstruction model to generate reconstructed spectral images of N photosensitive bands for each of the M color images, resulting in M ​​x N reconstructed spectral images. The parameters of the reconstruction model are determined by training the model using sample color images and sample spectral images, and by calculating a loss function based on the sample spectral images and the predicted reconstructed spectral images output by the model. The loss function is determined when it converges and its value is less than or equal to a threshold.

[0029] Thirdly, a computing device is provided, the computing device including at least one processor and a memory for storing a set of computer instructions; when the processor executes the set of computer instructions, it performs the operational steps of the image generation method in the first aspect or any possible implementation of the first aspect.

[0030] Fourthly, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a computing device, causing the computing device to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0031] Fifthly, a computer program product is provided that, when run on a computer, causes a computing device to perform the operational steps of the method as described in the first aspect or any possible implementation thereof.

[0032] A sixth aspect provides a chip system applied to an electronic device; the chip system includes an interface circuit and a processor; the interface circuit and the processor are interconnected via a line; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the chip system performs the operation steps of the method as described in the first aspect or any possible implementation of the first aspect.

[0033] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0034] Figure 1 A schematic diagram of a spectrum provided for an embodiment of this application;

[0035] Figure 2 A simplified architectural diagram of a multispectral sensor provided for an embodiment of this application;

[0036] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0037] Figure 4 A schematic diagram of the camera position provided in an embodiment of this application;

[0038] Figure 5 An example diagram of a camera interface provided in an embodiment of this application;

[0039] Figure 6 A flowchart illustrating an image generation method provided in this application embodiment;

[0040] Figure 7 A schematic diagram of a corrected and reconstructed spectral image provided in an embodiment of this application;

[0041] Figure 8 A flowchart illustrating another image generation method provided in this application embodiment;

[0042] Figure 9 A schematic diagram of an image with different exposure levels provided in an embodiment of this application;

[0043] Figure 10 A flowchart illustrating another image generation method provided in this application embodiment;

[0044] Figure 11 A schematic diagram illustrating an image generation process provided in an embodiment of this application;

[0045] Figure 12 This is a schematic diagram of the structure of a training module and an image generation module provided in an embodiment of this application;

[0046] Figure 13 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0047] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0049] The optical spectrum, or spectrum for short, is a pattern formed by the arrangement of monochromatic light according to wavelength (or frequency) after polychromatic light has been dispersed by a dispersive system (such as a prism or grating). The spectrum includes the visible spectrum and the invisible spectrum. Electromagnetic radiation within the wavelength range of the visible spectrum is called visible light. Figure 1This is a schematic diagram of a spectrum provided for an embodiment of this application. From short wave to long wave, they represent gamma rays, X-rays, ultraviolet light, visible light, infrared light, and radio waves. The visible light that the human eye can generally perceive has a frequency between 380 and 750 terahertz (THz) and a wavelength between 400 and 780 nanometers (nm). The frequency and wavelength of visible light perceived by different people can vary slightly. For example, some people can perceive visible light with a frequency of approximately 340 to 790 THz and a wavelength of approximately 380 to 880 nm. Electromagnetic waves of different frequencies cause different color perceptions in the human eye. Visible light, after being dispersed by a prism or grating, presents a continuous visible spectrum of red, orange, yellow, green, cyan, blue, and violet. Table 1 shows the frequencies and wavelengths of the continuous visible spectrum.

[0050] Table 1

[0051] Color Frequency Wavelength Red 385-482 THz 780-622 nm

[0052] Orange 482-503 THz 622-597 nm Yellow 503-520 THz 597-577 nm Green 520-610 THz 577-492 nm Blue 492-455 nm Blue 620-659 THz 455-435 nm Violet 435-400 nm Figure 2 Figure 3

[0053] Multispectral technology refers to the ability to acquire multiple spectral bands simultaneously or over a period of time, extending beyond visible light into infrared and ultraviolet light. Spectral bands can also be called photosensitive bands or bands. For example, a sensor can acquire spectral images of the first photosensitive band, then the second, and finally the third, by switching different filters over a short period of time, with relatively short time intervals between acquisitions and minimal changes in the target's position. This allows for the acquisition of spectral images of multiple different photosensitive bands within the same short timeframe. Common multispectral sensors contain multiple beam-splitting elements (such as filters or beam splitters) and imaging optical elements. Each beam-splitting element is sensitive to light in one photosensitive band; this can be understood as one beam-splitting element transmitting light in one photosensitive band. The photosensitive bands of different beam-splitting elements may partially overlap or not overlap at all, without limitation. For example, beam splitter 1 has a photosensitive wavelength range of 630–610 nm, and beam splitter 2 has a photosensitive wavelength range of 620–600 nm. The photosensitive wavelength range that overlaps between beam splitter 1 and beam splitter 2 is 620–610 nm. Alternatively, beam splitter 1 may have a photosensitive wavelength range of 630–610 nm, and beam splitter 2 may have a photosensitive wavelength range of 600–580 nm. The photosensitive wavelength ranges of beam splitter 1 and beam splitter 2 do not overlap. Understandably, if two beam splitters are sensitive to light within the same photosensitive wavelength range, it can be considered that both beam splitters transmit light within that photosensitive wavelength range.

[0054] In some embodiments, a multispectral sensor can simultaneously obtain light in multiple photosensitive bands by receiving incident light.Figure 3 This is a simplified schematic diagram of a multispectral sensor architecture provided in an embodiment of this application. The multispectral sensor receives incident light through an aperture. The incident light passes through N beam-splitters. Each beam-splitter transmits light within its own photosensitive wavelength range and reflects light outside its photosensitive wavelength range. Imaging optical elements after each beam-splitter generate a spectral image using the light passing through the beam-splitter. The spectral image is a grayscale image. A grayscale image is an image where each pixel has only one sampled color. Typically, a grayscale image is displayed as grayscale ranging from the darkest black to the brightest white. Therefore, the multispectral sensor can generate N spectral images, i.e., spectral images of N photosensitive wavelengths, where the image information of any two spectral images is determined by light from different photosensitive wavelengths. Furthermore, the multispectral sensor can also generate a multispectral image with rich colors based on the N spectral images. N is a positive integer.

[0055] In other embodiments, the multispectral sensor receives incident light through an aperture. The incident light passes through a beam splitter, which transmits light within its own wavelength range and reflects light outside its range. The multispectral sensor then switches beam splitters and receives incident light through the aperture again. The incident light passes through another beam splitter, which transmits light within its own wavelength range and reflects light outside its range. This process is repeated sequentially, allowing the multispectral sensor to acquire light from multiple different wavelength ranges within a given time period.

[0056] Understandably, the number of spectral images generated by a multispectral sensor is determined by the number of photosensitive bands it supports. The more photosensitive bands a multispectral sensor supports, the more spectral images it generates, and the richer the color information contained in those images. For example, if a multispectral sensor supports 204 photosensitive bands, it can generate 204 spectral images of different photosensitive bands. Multispectral sensors, utilizing multispectral technology, can acquire color information from more photosensitive bands than traditional RGB cameras.

[0057] The electronic devices in this application embodiment can be smartphones, digital cameras, televisions, tablets, projectors, desktops, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, including devices with multispectral sensors and RGB cameras. This application embodiment does not impose any special limitations on the specific form of the electronic device.

[0058] Please refer to Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device includes: a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a power management module 340, an antenna, a wireless communication module 360, an audio module 370, a speaker 370A, a speaker interface 370B, a microphone 370C, a sensor module 380, buttons 390, an indicator 391, a display screen 392, and a camera 393, etc. The sensor module 380 may include sensors such as a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, and an ambient light sensor.

[0059] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0060] Processor 310 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0061] In this embodiment, the processor 310 receives a spectral image and a color image from the camera 393, both containing the same target subject. It corrects the color cast of the reconstructed spectral image from the color image based on the spectral image, obtaining multiple corrected reconstructed spectral images. These multiple corrected reconstructed spectral images are then fused to generate a corrected color image, making the colors of the corrected color image closer to the colors of the real scene than the colors of the original color image. Here, color cast refers to the deviation between the colors of the image and the colors of the real scene.

[0062] A controller can be the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0063] The processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. This memory can store instructions or data that the processor 310 has just used or that are used repeatedly. If the processor 310 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0064] In some embodiments, the processor 310 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and / or a USB interface, etc.

[0065] The power management module 340 is used to connect to a power source. The power management module 340 can also be connected to the processor 310, internal memory 321, display screen 392, camera 393, and wireless communication module 360, etc. The power management module 340 receives power input and supplies power to the processor 310, internal memory 321, display screen 392, camera 393, and wireless communication module 360, etc. In some embodiments, the power management module 340 can also be located within the processor 310.

[0066] Wireless communication functionality in electronic devices can be achieved through antennas and wireless communication modules such as 360. The 360 ​​wireless communication module can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as Wi-Fi), Bluetooth, Global Navigation Satellite System (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0067] The wireless communication module 360 ​​can be one or more devices integrating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via an antenna, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signal to the processor 310. The wireless communication module 360 ​​can also receive signals to be transmitted from the processor 310, frequency-modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna. In some embodiments, the antenna of the electronic device is coupled to the wireless communication module 360, enabling the electronic device to communicate with networks and other devices via wireless communication technology.

[0068] Electronic devices implement display functions through a GPU, a display screen 392, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 392 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 310 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0069] Display screen 392 is used to display images, videos, etc. Display screen 392 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0070] Electronic devices can implement shooting functions through an ISP, camera 393, video codec, GPU, display 392, and application processor. The ISP is used to process the data fed back by the camera 393. In some embodiments, the ISP can be located in the camera 393.

[0071] Camera 393 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include Q cameras 393, where Q is a positive integer greater than 2.

[0072] In this embodiment, the electronic device includes at least a first camera and a second camera. The first camera is capable of capturing light across more photosensitive wavelengths than the second camera. The first camera is used to acquire spectral images. The second camera is used to acquire color images. For example, the first camera is a multispectral sensor or a hyperspectral sensor. The second camera is a camera with three photosensitive wavelengths: red, green, and blue (RGB). This embodiment does not limit the position of the first and second cameras on the electronic device. For example, taking a smartphone as an example, the first and second cameras are located on the same side of the smartphone. Figure 4 The image shown is a schematic diagram illustrating the position of a camera according to an embodiment of this application. Figure 4 As shown in (a) above, the first camera 401 is a rear-facing camera, and the second camera 402 is a rear-facing camera. Figure 4 As shown in (b), the first camera 401 is a front-facing camera, and the second camera 402 is a front-facing camera.

[0073] Within the electronic device, the first and second cameras can be located in different modules and operate independently. For example... Figure 4 As shown in (c), the first camera 401 is located in module 1, and the second camera 402 is located in module 2.

[0074] In other embodiments, the first and second cameras may be located in the same module, and their operating states are controlled by a driver module (e.g., processor 310). The first and second cameras can operate simultaneously to acquire color and spectral images, or only one of the cameras may be activated. Figure 4 As shown in (d) in the diagram, both the first camera 401 and the second camera 402 are located in module 1.

[0075] Optionally, the first and second cameras can be integrated internally within the electronic device, with only one camera being displayed externally. That is, the first and second cameras reside on different chips within the same module and lens, and their operation is controlled by a driver module (e.g., processor 310). The first and second cameras can operate simultaneously to acquire color and spectral images, or only one of them can be activated. The incident light acquired by the externally displayed camera is processed separately by the first and second cameras to obtain spectral and color images. Figure 3 As shown in (e), the electronic device presents a first camera 401 to the outside, and inside the electronic device, a multispectral sensor and an RGB sensor are integrated into module 1.

[0076] In other embodiments, traditional RGB pixels can be designed on the same chip as spectral pixels of other photosensitive bands through different pixel arrangements. The pixels of the multispectral sensor and the RGB sensor are controlled by a driving module (e.g., processor 310). The pixels of the multispectral sensor and the RGB sensor can work simultaneously, or only the spectral pixels can work. When only one type of pixel is working, a spatial interpolation algorithm is used to calculate the values ​​of the inactive pixel positions, thereby obtaining the pixel values ​​of the entire array.

[0077] like Figure 3 As shown in (f), the electronic device presents a first camera 401 to the outside. Inside the electronic device, module 1 integrates pixels of a multispectral sensor and pixels of an RGB sensor. There are three types of RGB pixels: red pixel R, green pixel G, and blue pixel B; there are four types of spectral pixels: spectral pixel U1 of photosensitive band λ1, spectral pixel U2 of photosensitive band λ2, spectral pixel U3 of photosensitive band λ3, and spectral pixel U4 of photosensitive band λ4.

[0078] Alternatively, the electronic device may not include a camera, meaning the aforementioned camera 393 is not integrated into the electronic device (such as a television). The electronic device can connect an external camera 393 via an interface (such as a USB interface 330). This external camera 393 can be secured to the electronic device using an external fastener (such as a camera holder with a clip). For example, the external camera 393 can be secured to the edge of the electronic device's display screen 392, such as the upper edge, using an external fastener.

[0079] Digital signal processors (DSPs) are used to process digital signals, including digital image signals and other digital signals. For example, when an electronic device selects a frequency, a DSP performs a Fourier transform on the frequency energy. Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the device to play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0080] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0081] The external storage interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 310 through the external storage interface 320 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0082] Internal memory 321 can be used to store computer executable program code, which includes instructions. Processor 310 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 321. Internal memory 321 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, etc.). Furthermore, internal memory 321 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0083] The electronic device can implement audio functions through an audio module 370, a speaker 370A, a microphone 370C, a speaker interface 370B, and an application processor. Examples include music playback and recording. In this embodiment, the microphone 370C can be used to receive voice commands from the user to the electronic device. The speaker 370A can be used to respond to the user's voice commands from the electronic device.

[0084] Audio module 370 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 370 can also be used for encoding and decoding audio signals. In some embodiments, audio module 370 may be located in processor 310, or some functional modules of audio module 370 may be located in processor 310. Speaker 370A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Microphone 370C, also called a "microphone" or "microphone," is used to convert sound signals into electrical signals.

[0085] The speaker jack 370B is used to connect wired speakers. The speaker jack 370B can be a USB 330 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0086] Buttons 390 include a power button, volume buttons, etc. Buttons 390 can be mechanical buttons or touch-sensitive buttons. Electronic devices can receive button inputs and generate key signal inputs related to user settings and function control.

[0087] Indicator 391 can be an indicator light, used to indicate whether an electronic device is in a powered-on, standby, or powered-off state. For example, an indicator light that is off indicates that the electronic device is powered off; an indicator light that is green or blue indicates that the electronic device is powered on; and an indicator light that is red indicates that the electronic device is in a standby state.

[0088] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. It may have more than Figure 5 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. For example, the electronic device may also include components such as speakers. Figure 5 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing or application-specific integrated circuits.

[0089] The methods described in the following embodiments can all be implemented in an electronic device having the above-described hardware structure. The following embodiments use a smartphone as an example to illustrate the methods of this application.

[0090] Suppose a user takes a picture of the same subject using a smartphone equipped with a first camera and a second camera. For example, Figure 5 This is an example diagram of a camera interface provided in an embodiment of this application. Figure 5 As shown in (a) of this application, it is a front view of a smartphone provided in an embodiment of the present application. The smartphone's display screen shows a camera application (APP) icon 501, which the user can click. The smartphone's touch sensor receives the touch operation and reports it to the processor, causing the processor to respond to the touch operation and launch the camera application. Furthermore, in this embodiment of the present application, the smartphone can also launch the camera application and display the camera application's user interface on the display screen in other ways. For example, when the smartphone screen is black, displays a lock screen, or displays a user interface after unlocking, it can respond to the user's voice command or shortcut operation to launch the camera application and display the camera application's user interface on the display screen. Figure 5 As shown in (b), in response to a click operation, the smartphone's display shows the user interface of the camera application. The user interface includes mode options such as "Short Video," "Video Recording," "Photo Taking," "Portrait," and "Panorama," as well as a "Preview Image" button, a "Photo Taking" button, and a "Front / Rear Camera Switching" button. The user can first select the "Photo Taking" mode 502, and then click the "Photo Taking" button 503. In response to the click operation, the smartphone automatically activates the first and second cameras. The first camera acquires the spectral image of the target subject, and the second camera acquires the color image of the target subject. The smartphone's processor corrects the color cast of the reconstructed spectral image based on the spectral image, obtaining multiple corrected reconstructed spectral images. These multiple corrected reconstructed spectral images are then fused to generate a corrected color image, making the colors of the corrected color image closer to the colors of the real scene than the colors of the original color image. Thus, by combining the second camera with a first camera that has a wider range of photosensitive wavelengths, and fully utilizing the rich spectral information contained in the spectral image acquired by the first camera, the reconstructed spectral image of the color image is corrected in the spectral domain. The corrected reconstructed spectral image is then fused to generate a corrected color image, resulting in a high dynamic range color image with more refined and accurate colors compared to the color image acquired by the second camera. Furthermore, as... Figure 6 As shown in (c), users can click the "Preview Image" function button 505 to view the captured image containing the target subject.

[0091] The above embodiments illustrate the use of a smartphone automatically activating both the first and second cameras when taking a photo, and performing color correction on the captured image by default. In other embodiments, such as... Figure 7As shown in (d), the camera application's user interface can also display a "color correction" function button 504, allowing the user to decide whether to perform color correction on the captured image. After selecting "Shoot" mode 502, the user can also click the "color correction" function button 504, and then click the "Shoot" function button 503. In response to the click operation, the smartphone activates both the first and second cameras. The first camera acquires a spectral image of the target subject, and the second camera acquires a color image of the target subject, so that the smartphone's processor can use the spectral image to perform color correction on the color image. If the user selects "Shoot" mode 502 but does not click the "color correction" function button 504, and then clicks the "Shoot" function button 503, the smartphone, in response to the click operation, does not activate the first camera but activates the second camera. The second camera acquires a color image of the target subject, and thus, the smartphone's processor does not perform color correction on the color image.

[0092] Next, combined Figure 8 This embodiment provides a detailed description of the image generation method. The apparatus for correcting color cast in the reconstructed spectral image of a color image and for fusing the corrected color image is not limited here. For example, the apparatus may be a processor in a shooting module including a first camera and a second camera, or it may be another processor in an electronic device. The first camera is a multispectral sensor or a hyperspectral sensor. The second camera is a camera with three photosensitive bands: red, green, and blue (RGB). The first camera and the second camera may be located in the same shooting module, or they may be located in different independent shooting modules; for a detailed explanation, please refer to the description in the above embodiment.

[0093] S601. Acquire the spectral image of the first scene through the first camera.

[0094] S602. Acquire a color image of the second scene using the second camera.

[0095] Understandably, when a user takes a photo of a real-world scene using a smartphone, the first camera and the second camera capture images of the scene separately within a time interval. This time interval can be in seconds or milliseconds. In some embodiments, if the time interval is 0 seconds, the first and second cameras can capture images of the scene simultaneously. However, the first and second cameras have different fields of view, resulting in slight differences between the spectral image acquired by the first camera and the color image acquired by the second camera. For example, the field of view (FoV) of the first camera is smaller than that of the second camera, and the range of the first scene within the first camera's field of view is smaller than the range of the second scene within the second camera's field of view. Therefore, the size of the spectral image acquired by the first camera is smaller than the size of the color image acquired by the second camera. It should be noted that the spectral image and the color image contain at least a portion of the same image, meaning both contain the same target subject, to facilitate the correction of color cast in the reconstructed spectral image from the color image using the spectral image.

[0096] S603. Correct the color shift of the reconstructed spectral image from the color image based on the spectral image to obtain multiple corrected reconstructed spectral images.

[0097] Specifically, first, identify the overlapping areas in the spectral image and the reconstructed spectral image, as well as the pixels in both images that exhibit the same image features. These pixels in the reconstructed spectral image can be called matching pixels. Matching pixels refer to pixels with the same image features within the overlapping area of ​​the spectral image and the reconstructed spectral image. It should be understood that both the spectral image and the reconstructed spectral image contain pixels exhibiting the same image features; the positions of these pixels can be the same or different, without limitation. The pixel values ​​of the matching pixels in the spectral image and the reconstructed spectral image may differ. Compared to the pixel values ​​of matching pixels in the reconstructed spectral image, the pixel values ​​of pixels exhibiting the same image features in the spectral image are closer to the color of the target subject in the real scene, while the pixel values ​​of matching pixels in the reconstructed spectral image deviate from the color of the target subject in the real scene.

[0098] Image features mainly include color features, texture features, shape features, and spatial relationship features. Algorithms such as scale-invariant feature transform (SIFT) can be used to identify identical image features between the spectral image and the reconstructed spectral image. SIFT can detect keypoints in an image and is a local feature description algorithm. Identical image features include at least one of the following: color features, texture features, shape features, and spatial relationship features.

[0099] Furthermore, a correction coefficient is set for each matching pixel in the reconstructed spectral image, and the pixel values ​​of the matching pixels in the reconstructed spectral image are corrected based on the pixel values ​​of the spectral image and the correction coefficients to obtain the corrected reconstructed spectral image. Understandably, the pixel values ​​of the corrected reconstructed spectral image differ from those of the original spectral image. The pixel values ​​of pixels exhibiting the same image features in the corrected reconstructed spectral image are closer to the color of the target subject in the real scene. For pixels exhibiting the same image characteristics in both the spectral and reconstructed spectral images, if the pixel value in the reconstructed spectral image is larger than that in the spectral image, the pixel value in the reconstructed spectral image can be decreased; conversely, if the pixel value in the reconstructed spectral image is smaller than that in the spectral image, the pixel value in the reconstructed spectral image can be increased.

[0100] In some embodiments, the pixels in the corrected reconstructed spectral image satisfy the following formula (1).

[0101] r′=f(s,r)=r+v·(sr) (1)

[0102] Where f(s, r) represents the correction function. r′ represents the pixels in the corrected reconstructed spectral image, s represents the pixels in the spectral image, and r represents the pixels in the reconstructed spectral image. v represents the correction coefficient. The correction coefficient can be pre-configured. The value of the correction coefficient ranges from 0 to 1. The more the pixels in the spectral image and the reconstructed spectral image match, the closer the correction coefficient is to 1; the more the pixels in the spectral image and the reconstructed spectral image do not match, the closer the correction coefficient is to 0. For pixels in the reconstructed spectral image that do not match the pixels in the spectral image, the correction coefficient is equal to 0. Mismatched pixels refer to pixels in the reconstructed spectral image that exhibit different image features from those in the spectral image.

[0103] Assume the correction coefficient for pixels outside the effective region is 0, the correction coefficient for pixel r of the reconstructed spectral image that matches pixel s of the spectral image within the effective region is 0.8, and the correction coefficient for the remaining pixels is 0.2. For example, ... Figure 9 As shown, the color correction process is illustrated using a 3x2 pixel image size and a photosensitive wavelength of λ1 as an example. In this paper, "x" represents multiplication. Assume the effective region is 2x2 pixels in the second and third columns, i.e., the effective region includes pixels with coordinates (1,2), (1,3), (2,2), and (2,3). Pixels (1,2), (1,3), and (2,2) are pixels with the same image characteristics that match those in the spectral image, while pixel (2,3) is a mismatched pixel. s(λ1) represents the pixel value in the spectral image. r(λ1) represents the pixel value of the reconstructed spectral image. v represents the value of the correction factor. r′(λ1) represents the pixel value in the corrected reconstructed spectral image.

[0104] S604. Generate a corrected color image by fusing multiple corrected reconstructed spectral images.

[0105] Understandably, the corrected reconstructed spectral image is obtained by correcting the reconstructed spectral images of multiple color images with different exposures. The corrected color image generated by fusing the corrected reconstructed spectral images is a color-corrected high dynamic range image, and the colors of the corrected color image are closer to the colors of the second scene than the colors of the color images.

[0106] In one possible design, the fusion operation is performed in the spectral domain, that is, the corrected reconstructed spectral image is first fused to obtain a fused spectral image, and the corrected color image is generated based on the fused spectral image.

[0107] For details, please refer to Figure 9 This is a flowchart of another image generation method provided in an embodiment of this application.

[0108] S801. Acquire spectral images of N photosensitive bands of the first scene through the first camera.

[0109] S802: Acquire an M-frame color image of the second scene using the second camera.

[0110] The number of spectral images generated by the first camera is determined by the number of photosensitive bands supported by the first camera, which in turn is determined by the number of beam-splitting elements included in the first camera. In this embodiment, the number of photosensitive bands supported by the first camera is greater than the number of photosensitive bands supported by the second camera. If the second camera supports 3 photosensitive bands, N is an integer greater than 3.

[0111] The second camera can acquire color images with different exposure levels based on different exposure durations. For example, the second camera can first capture a color image with a first exposure level at a first exposure duration; then capture a color image with a second exposure level at a second exposure duration. Here, M frames of color images represent images with different exposure levels. M is an integer greater than or equal to 2, meaning that the second camera can acquire at least two frames of color images with different exposure levels. Assuming M = 3, the second camera can acquire 3 frames of color images.

[0112] Because of varying exposure times, the effective pixels within each frame of a color image differ. For example, under short exposures, some pixels are darker and cannot provide effective texture information, resulting in underexposure; under long exposures, some pixels may be brighter and also cannot provide effective texture information, resulting in overexposure. Figure 9The image shown is a schematic diagram of an image with different exposure levels provided in an embodiment of this application. Wherein, Figure 9 Image (a) in the image is the image with an exposure time of t1. Figure 10 Image (b) is the image with an exposure time of t2. Figure 8 Image (c) in the diagram represents the image with an exposure time of t3. Since t1 > t2 > t3, the image with exposure time t1 is the brightest, the image with exposure time t3 is the darkest, and the brightness of the image with exposure time t2 falls between that of the images with exposure time t1 and t3. The image with exposure time t1 contains overexposed pixels. The image with exposure time t2 contains normally exposed pixels. The image with exposure time t3 contains underexposed pixels.

[0113] Other explanations regarding the first scene, the second scene, the spectral image, and the color image can be found in the description in S601 above.

[0114] S803. Using the reconstruction model, generate reconstructed spectral images of N photosensitive bands for each of the M frame color images, resulting in MxN reconstructed spectral images.

[0115] The parameters of the reconstruction model are obtained from sample color images and sample spectral images contained in the training dataset. These sample color images and sample spectral images were captured on the same scene at the same time or within a time interval. That is, the sample color images and sample spectral images can be images captured simultaneously by the first and second cameras receiving shooting commands at the same time, or images captured by the first camera receiving the first shooting command and capturing a spectral image, followed by the second camera receiving the second shooting command and capturing a color image after a certain interval. It should be understood that the sample color images and sample spectral images can be two images with overlapping fields of view; overlapping fields of view can also mean that the sample color images and sample spectral images overlap. Therefore, the reconstruction model is trained using two overlapping sample color images and sample spectral images to improve the accuracy of the training.

[0116] Specifically, the sample color image and sample spectral image are divided into image blocks of a fixed size (e.g., 16x16 pixels). All the resulting image blocks are input into the reconstruction model to obtain several image blocks, which are then combined to form the predicted reconstructed spectral image. If the image blocks input to the reconstruction model are obtained from multiple pairs of sample color images and sample spectral images, the image blocks output by the reconstruction model can form multiple predicted reconstructed spectral images. The predicted reconstructed spectral image is compared with the sample spectral image, and the difference between the two is measured by a loss function. The loss function satisfies the following formula (2).

[0117]

[0118] Formula (2) represents the minimum Euclidean distance between the predicted reconstructed spectral image and the original spectral image. Here, N represents the number of photosensitive bands.

[0119] The loss function is calculated based on the sample spectral image and the predicted reconstructed spectral image. After multiple iterations, the parameters of the reconstruction model are obtained when the loss function converges and its value is less than or equal to a threshold. When the loss function converges and its value is greater than the threshold, the parameters of the reconstruction model are adjusted, and the reconstruction model is trained again using the above method.

[0120] After training and obtaining the parameters of the reconstruction model, M frames of color images can be input into the reconstruction model to generate reconstructed spectral images of N photosensitive bands for each frame of color image, ultimately resulting in MxN reconstructed spectral images.

[0121] S804. For the pixel values ​​of the spectral image of each of the N photosensitive bands, correct the pixel values ​​of matching pixels belonging to the same photosensitive band in the MxN reconstructed spectral images to obtain MxN corrected reconstructed spectral images.

[0122] Specifically, based on N photosensitive bands, the M x N reconstructed spectral images are divided into N sets of reconstructed spectral images. Each set contains M reconstructed spectral images with the same photosensitive band, which are generated from M color images with different exposure levels. The photosensitive bands of the reconstructed spectral images belonging to different sets are different. Furthermore, based on the photosensitive band λ... n Pixel value correction of the spectral image for M photosensitive bands λ n In each of the M x N reconstructed spectral images, the pixel value of the matching pixel in each reconstructed spectral image is corrected, where n = [1, N]. After correcting each of the M x N reconstructed spectral images, M x N corrected reconstructed spectral images are obtained. For specific methods regarding the correction of the pixel values ​​of the matching pixels, please refer to the explanation in S603 above.

[0123] In other embodiments, an effective region of the reconstructed spectral image is selected based on the photosensitive band λ. n Pixel value correction of the spectral image for M photosensitive bands λ n In each reconstructed spectral image, the pixel values ​​of matching pixels in the effective region of each reconstructed spectral image are corrected, resulting in M ​​x N corrected reconstructed spectral images. This process removes meaningless pixel values ​​from the reconstructed spectral images and corrects the pixel values ​​of pixels matching the spectral images in the effective region, thus improving the image processing speed.

[0124] The effective region refers to the area in a color image where the pixel values ​​fall within a preset range. This region can be defined as the effective region. The preset range for pixel values ​​can be greater than 5 and less than 250. Regions with pixel values ​​greater than 250 are defined as overexposed regions. Regions with pixel values ​​less than 5 are defined as underexposed regions. This embodiment does not limit the preset range for pixel values; it can be adjusted according to shooting requirements. The effective region includes all regions except for overexposed and underexposed regions. The region in the reconstructed spectral image that overlaps with the effective region of the color image can be defined as the effective region of the reconstructed spectral image. The range of the effective region of the reconstructed spectral image is the same as the range of the effective region of the color image.

[0125] S805. For each of the N photosensitive bands, fuse the corrected reconstructed spectral images belonging to the same photosensitive band from the MxN corrected reconstructed spectral images to obtain N fused spectral images.

[0126] Specifically, based on N photosensitive bands, the M x N corrected reconstructed spectral images are divided into N sets of corrected reconstructed spectral images. Each set of corrected reconstructed spectral images contains M corrected reconstructed spectral images with the same photosensitive band. Then, for each set of corrected reconstructed spectral images, the M corrected reconstructed spectral images with the same photosensitive band are fused to obtain N fused spectral images.

[0127] In some embodiments, weights can be pre-set for M different exposure times, and a multi-exposure fusion algorithm can be used to fuse M x N corrected reconstructed spectral images belonging to the same photosensitive band to obtain N fused spectral images. The fused spectral images satisfy the following formula (3).

[0128] r * (λ n )=∑ω m (λ n )·r′ m (λ n ),m∈[1,M],n∈[1,N] (3)

[0129] Where, ω m (λ n ) represents the weight of exposure time, ω m (λ n The value of r ranges from 0 to 1. * (λ n () represents the fused spectral image. r′ m (λ n () represents the corrected reconstructed spectral image of the same photosensitive band.

[0130] S806. Generate a corrected color image based on N fused spectral images.

[0131] By mapping the spectral domain to the RGB space, high dynamic range spectral images under N photosensitive bands are converted into a single high dynamic range color image.

[0132] First, calculate the XYZ three-color stimulus values ​​of the object surface. The three-color stimulus values ​​satisfy the following formula (4).

[0133]

[0134] Where K is the adjustment factor. The values ​​are shown in Table 2 below.

[0135] Table 2

[0136]

[0137]

[0138] Then, the RGB three color values ​​are calculated, where L is the CIE1931 standard XYZ to RGB conversion matrix under the D65 standard light source. The corrected color image satisfies the following formula (5).

[0139]

[0140] in,

[0141] In this way, by combining the second camera with the first camera, which has more photosensitive wavelengths than the second camera, the rich spectral information obtained through the first camera is fully utilized to assist in correcting the color of the color image in the spectral domain. The corrected reconstructed spectral image is then fused to obtain a high dynamic range fused spectral image. Based on the high dynamic range fused spectral image, a corrected high dynamic range color image is generated, making the color of the corrected high dynamic range color image more refined and accurate, and closer to the color of the real scene.

[0142] In another possible design, the fusion operation is performed in the color gamut; that is, an intermediate color image is first generated based on the corrected reconstructed spectral image, and then the intermediate color images are fused to generate the corrected color image. For example... Figure 11 As shown above, Figure 11 The difference is that S1001 and S1002 are executed after S804.

[0143] S1001. Generate M intermediate color images based on M x N corrected reconstructed spectral images.

[0144] Specifically, based on M exposure times, the M x N corrected reconstructed spectral images are divided into M sets of corrected reconstructed spectral images. Each set contains N corrected reconstructed spectral images of different photosensitive bands under one exposure time. The exposure time corresponding to each set of corrected reconstructed spectral images is different. Furthermore, for each set of corrected reconstructed spectral images under each exposure time, an intermediate color image is generated based on the N corrected reconstructed spectral images of different photosensitive bands, resulting in M ​​intermediate color images. For the specific method of generating the M intermediate color images, please refer to the explanation of formulas (4) and (5) in S806 above.

[0145] S1002. Fuse the M intermediate color images to obtain the corrected color image.

[0146] In some embodiments, weights can be pre-set for M different exposure durations, and the M intermediate color images can be fused using a multi-exposure fusion algorithm to obtain a corrected color image. The corrected color image satisfies the following formula (6).

[0147]

[0148] Where, ω m (t m ) represents the weight of exposure time, ω m (t m The value range of ) is 0-1. This represents the intermediate color image. * This represents the corrected color image.

[0149] In some embodiments, a standard color chart is used to evaluate the color of the corrected color image. Specifically, the standard color chart is placed in the shooting scene, and the standard color chart is photographed using a first camera and a second camera to obtain a spectral image and a color image. Both the spectral image and the color image contain the pixel values ​​of the standard color chart. Based on the method provided in this embodiment, the color image is corrected using the spectral image to obtain a corrected reconstructed spectral image. Since the CIELAB value of each color block of the standard color chart is known, the corrected reconstructed spectral image contains the CIELAB values ​​of the color blocks of the standard color chart. The color difference ΔE between the CIELAB values ​​of the color blocks in the corrected reconstructed spectral image and the CIELAB values ​​of the color blocks in the standard color chart is calculated according to the CIEDE2000 standard. If ΔE is less than 2, the color correction performance is good.

[0150] In this way, by combining the second camera with the first camera, which has more photosensitive wavelengths than the second camera, the rich spectral information obtained through the first camera is fully utilized to assist in correcting the color of the color image in the spectral domain. An intermediate color image is generated based on the corrected reconstructed spectral image, and the intermediate color image is fused to generate a corrected high dynamic range color image. This makes the color of the corrected high dynamic range color image more refined and accurate, and closer to the color of the real scene.

[0151] The following example illustrates the image generation process provided in the embodiments of this application. Assume that the first camera 1101 supports 5 photosensitive bands, and the second camera 1103 supports 3 exposure durations, i.e., N=5 and M=3.

[0152] like Figure 12 As shown in (a), the first camera acquires spectral images s(λ) of five photosensitive bands, namely spectral image s(λ1), spectral image s(λ2), spectral image s(λ3), spectral image s(λ4) and spectral image s(λ5).

[0153] The second camera acquires three color images p(t) with different exposure times t1, t2, and t3, namely color image p(t1), color image p(t2), and color image p(t3). For example, t1 is 1 / 10 second, t2 is 1 / 30 second, and t3 is 1 / 100 second.

[0154] Using a reconstruction model, reconstructed spectral images r(λ) for five photosensitive bands in each of three color images are generated, resulting in 3x5 reconstructed spectral images. For example, at an exposure time t1, reconstructed spectral images are obtained from the color image p(t1). At an exposure time t2, a reconstructed spectral image is generated based on the color image p(t2). At an exposure time t3, a reconstructed spectral image is generated based on the color image p(t3).

[0155] For the pixel values ​​of the spectral image of each of the five photosensitive bands, the pixel values ​​of matching pixels belonging to the same photosensitive band in each of the 3x5 reconstructed spectral images are corrected to obtain 3x5 corrected reconstructed spectral images. For example, using the spectral image s(λ1) of photosensitive band λ1, the reconstructed spectral image is... Color correction is performed to obtain the corrected reconstructed spectral image. By analogy, other reconstructed spectral images are corrected to obtain the corrected reconstructed spectral images. Corrected reconstructed spectral image Corrected reconstructed spectral image Corrected reconstructed spectral image

[0156] Specifically, for the pixel values ​​of the spectral image of each of the five photosensitive bands, the pixel values ​​of matching pixels belonging to the same photosensitive band in the effective area of ​​the 3x5 reconstructed spectral images are corrected to obtain 3x5 corrected reconstructed spectral images.

[0157] For each of the five photosensitive bands, the corrected reconstructed spectral images belonging to the same photosensitive band as each of the five photosensitive bands are fused together from 3x5 corrected reconstructed spectral images to obtain five fused spectral images. For example, the corrected reconstructed spectral images of all photosensitive bands λ1 for exposure times t1, t2, and t3. The fusion process is performed to obtain the fused spectral image r of the photosensitive band λ1. * (λ1). Similarly, the corrected reconstructed spectral images of photosensitive bands λ2, λ3, λ4, and λ5 are fused to obtain the fused spectral image r. * (λ2), r * (λ3), r * (λ4), r * (λ5).

[0158] A corrected color image p is generated based on five fused spectral images. * The colors of the corrected color image are closer to those of the second scene than those of the original color image.

[0159] like Figure 12 As shown in (b), after obtaining 3x5 corrected reconstructed spectral images, for each of the three exposure times, three intermediate color images are generated based on the 3x5 corrected reconstructed spectral images. For example, based on the corrected reconstructed spectral images of all photosensitive bands λ1 to λ5 at exposure time t1. Generate an intermediate color image p for the exposure time t1. * (t1). Similarly, intermediate color images p are generated based on the corrected reconstructed spectral images of all photosensitive bands λ1 to λ5 for exposure times t2 and t3, respectively. * (t2), p * (t3). The three intermediate color images are fused to obtain the corrected color image p. * .

[0160] Typically, monochrome cameras lack color filters, giving them stronger light sensitivity compared to RGB cameras. This allows them to capture richer brightness information and enhance image detail. While RGB cameras provide rich color information, the spatial arrangement of RGB pixels sacrifices some spatial resolution, resulting in a lower effective resolution compared to monochrome cameras. Against this backdrop, by combining the brightness information of a monochrome image with the color information of a color image, a color image that retains color information but possesses more detail can be obtained. However, monochrome images do not provide color information, thus limiting their color shift correction capabilities compared to the image generation method provided in this application.

[0161] Existing technologies also include techniques for fusing two or more color images. Specifically, there are two implementation methods. One method involves using two RGB cameras to capture the same scene separately, obtaining two color images with different perspectives. This allows for the estimation of certain depth information, enabling effects such as background blurring, foreground enhancement, and motion blur removal. The other method involves using the same RGB camera to capture the same scene multiple times under different exposure conditions, obtaining multiple frames of color images with different dynamic ranges. These multiple frames are then fused using a multi-exposure fusion algorithm to obtain a single high dynamic range color image.

[0162] For the first method, since both cameras are RGB cameras, their ability to perceive the spectrum is limited to the red, green, and blue bands. Although there are two cameras, they cannot provide spectral information for additional light-sensitive bands. Therefore, they cannot obtain color information other than red, green, and blue, and there is no improvement in image color enhancement or color cast correction. In an extreme case, in scenes with large areas of monochrome or multiple light sources, both color images captured by the cameras will have color casts, and even if the two color images are merged, the color cast will still exist.

[0163] The second method suffers from the drawbacks mentioned earlier: multiple color images captured under different exposure conditions may exhibit color casts. Although the fusion process adjusts the weights of each image, it does not involve color cast correction, and color casts may still exist after fusion. The image generation method provided in this application addresses this drawback by introducing a multispectral sensor to perform color cast correction on the reconstructed spectral images of the multiple color images. Then, it performs multi-exposure image fusion in the spectral domain to resolve the color cast problem.

[0164] When executing the image generation method of the software system provided in this embodiment, the image generation module employs an artificial intelligence (AI) model. AI models encompass various types, with neural network models being one such type. A neural network model is a mathematical computational model that mimics the structure and function of biological neural networks (the central nervous system of animals). A neural network model can include multiple neural network layers with different functions, each layer including parameters and calculation formulas. Depending on the calculation formula or function, different layers in a neural network model have different names; for example, the layer performing convolution calculations is called a convolutional layer, which is often used for feature extraction from input signals (e.g., images). A neural network model can also be composed of a combination of multiple existing neural network models. Different neural network models can be used in different scenarios (e.g., classification, recognition). Alternatively, different neural network models may provide different effects when used in the same scenario. Different neural network model structures specifically include one or more of the following: different numbers of network layers, different order of network layers, and different weights, parameters, or calculation formulas in each network layer. Various high-accuracy neural network models exist in the industry for applications such as recognition and classification. Some neural network models can be trained on specific training sets to complete a task independently or in combination with other neural network models (or other functional sub-modules) to complete a task. Other neural network models can also be used directly to complete a task independently or in combination with other neural network models (or other functional sub-modules). In describing this embodiment, the neural network model can be a deep convolutional neural network. A deep convolutional neural network includes an input layer, a convolutional layer, three activation function layers, and an output layer.

[0165] The reconstruction model provided in this embodiment is a deep convolutional neural network used to generate reconstructed spectral images of N photosensitive bands for each of M frames of color images, resulting in M ​​x N reconstructed spectral images. Before generating the reconstructed spectral images from the color images, the reconstruction model is trained by a training module. After the training module has trained the reconstruction model using a dataset, it can be deployed in the calibration submodule within the image generation module, where the calibration submodule generates the reconstructed spectral images based on the color images.

[0166] Figure 12 A schematic diagram of the structure of a training module 1210 and an image generation module 1220 is provided below. Figure 12The structure and function of the training module 1210 and the image generation module 1220 are described below. It should be understood that this embodiment only provides an exemplary division of the structural and functional sub-modules of the training module 1210 and the image generation module 1220, and this application does not limit the specific division in any way.

[0167] Before training the reconstruction model 1211, the initialization submodule 1213 initializes the parameters of each layer in the reconstruction model 1211 (i.e., assigns an initial value to each parameter). Then, the training submodule 1214 reads data from the dataset in the database 1230. The dataset contains sample color images and sample spectral images. The first preprocessing submodule 1212 preprocesses the sample color images, that is, removing meaningless pixel values ​​from the color images and correcting the pixel values ​​of matching pixels in the effective regions, thereby improving the image processing speed of the reconstruction model 1211.

[0168] The preprocessed data is used to train the reconstruction model 1211 until the loss function in the reconstruction model 1211 converges and the loss function value is less than a certain threshold, at which point the training of the reconstruction model 1211 is complete. Alternatively, all the data in the defect localization training set is used for training, at which point the training of the reconstruction model 1211 is complete.

[0169] Optionally, the reconstruction model 1211 may not need to be trained by the training module 1210. For example, the reconstruction model 1211 may use a pre-trained neural network model from a third party that has good accuracy in generating reconstructed spectral images. In this embodiment, a dataset may not be constructed; for example, the dataset may be obtained directly from a third party.

[0170] The reconstruction model 1211, trained by the training module 1210, is used to generate a reconstructed spectral image based on the color image. In this embodiment, as... Figure 13 As shown, the trained reconstruction model 1211 is deployed to the correction submodule 1222 in the image generation module 1220.

[0171] like Figure 3 As shown, the image generation module 1220 includes a second preprocessing submodule 1221, a correction submodule 1222, and a generation submodule 1223.

[0172] The second preprocessing submodule 1221 is used to preprocess the color image from the camera to obtain the effective area of ​​the color image. The specific method for obtaining the effective area can be found in the description of S804 above.

[0173] The correction submodule 1222 is used to generate reconstructed spectral images of N photosensitive bands for each of the M frames of color images based on the reconstruction model 1211, resulting in MxN reconstructed spectral images; and for the pixel values ​​of the spectral image of each of the N photosensitive bands, to correct the pixel values ​​of matching pixels belonging to the same photosensitive band in the effective region of the MxN reconstructed spectral images, resulting in MxN corrected reconstructed spectral images. The specific method for correcting the reconstructed spectral images can be referred to the descriptions in S603, S803, and S804 above.

[0174] The generation submodule 1223 is used to fuse MxN corrected reconstructed spectral images belonging to the same photosensitive band from each of the N photosensitive bands to obtain N fused spectral images, and to generate a corrected color image based on the N fused spectral images. The colors of the corrected color image are closer to the colors of the second scene than the colors of the color image. The specific method for generating the corrected color image can be referred to the descriptions in S604, S805, and S806 above.

[0175] The generation submodule 1223 is also used to generate M intermediate color images based on M x N corrected reconstructed spectral images; and to fuse the M intermediate color images to obtain the corrected color image. The specific method for generating the corrected color image can be found in the descriptions of S604, S1001, and S1002 above.

[0176] Based on the functions of the aforementioned sub-modules, color images can be corrected according to spectral images. Thus, by fully utilizing rich spectral information, color images are corrected in the spectral domain, and high dynamic range spectral images for each photosensitive band are obtained through multi-exposure fusion. Finally, high dynamic range images are obtained through mapping, achieving a more refined and accurate high dynamic range image.

[0177] It is understood that, in order to achieve the functions in the above embodiments, the electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0178] Figure 13 This is a schematic diagram illustrating the structure of a possible image generation apparatus provided for embodiments of this application. These image generation apparatuses can be used to implement the functions of the electronic devices in the above-described method embodiments, and therefore can also achieve the beneficial effects of the above-described method embodiments. In the embodiments of this application, the image generation apparatus may be as follows: Figure 6The electronic device shown can also be a module (such as a chip) applied to an electronic device.

[0179] like Figure 8 As shown, the image generation apparatus 1300 includes an acquisition module 1310, an image generation module 1320, and a display module 1330. The image generation apparatus 1300 is used to implement the above-described... Figure 10 , Figure 6 ,or Figure 8 The method embodiment shown illustrates the function of the electronic device.

[0180] When the image generating device 1300 is used to achieve Figure 10 In the method embodiment shown, the electronic device functions as follows: the acquisition module 1310 is used to execute S601 and S602; the image generation module 1320 is used to execute S603 and S604.

[0181] When the image generating device 1300 is used to achieve Figure 3 In the method embodiment shown, the electronic device functions as follows: the acquisition module 1310 is used to execute S801 and S802; the image generation module 1320 is used to execute S803 to S806.

[0182] When the image generating device 1300 is used to achieve Figure 3 In the method embodiment shown, the electronic device functions as follows: the acquisition module 1310 is used to execute S801 and S802; the image generation module 1320 is used to execute S803 and S804, as well as S1001 and S1002.

[0183] The image generation apparatus 1300 may further include a storage module 1340, which is used to store program instructions related to the image generation method provided in the embodiments of this application, as well as data generated during the execution of the program instructions related to the image generation method.

[0184] The display module 1330 is used to display the output of the display module 1330. Figure 3 The display module 1330 displays a color image captured by camera 393 and a corrected color image generated by image generation module 1320. The function of display module 1330 can be achieved by… Figure 3 The function is implemented by the central display screen 392. The functionality of the acquisition module 1310 can be achieved by... Figure 3 The image generation module 1320 can be implemented using camera 393. Figures 1 to 12 The processor 310 is used for implementation. The function of the storage module 1340 can be achieved by... Figure 14 The image generation module 1320 is implemented using internal memory 321. It may further include a preprocessing submodule, a correction submodule, and a generation submodule.

[0185] For a more detailed description of the acquisition module 1310, image generation module 1320, and display module 1330 mentioned above, please refer to [the relevant documentation]. Figure 6 The relevant descriptions in the illustrated embodiments are directly obtained and will not be repeated here.

[0186] like Figure 8 As shown, the electronic device 1400 includes a processor 1410, an interface circuit 1420, a memory 1430, and a display 1440. The processor 1410 and the interface circuit 1420 are coupled to each other. It is understood that the interface circuit 1420 can be an input / output interface. The memory 1430 is used to store instructions executed by the processor 1410, or to store input data required by the processor 1410 to execute instructions, or to store data generated after the processor 1410 executes instructions. The display 1440 is used to display color images and calibrated color images.

[0187] When electronic device 1400 is used to achieve Figure 10 , ​ ,or ​ In the method shown, the processor 1410 is used to perform the functions of the image generation module 1320, the interface circuit 1420 is used to perform the functions of the acquisition module 1310, and the display 1440 is used to perform the functions of the display module 1330.

[0188] It is understood that the processor in the embodiments of this application may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0189] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Additionally, the ASIC can reside in a network device or a terminal device. Alternatively, the processor and storage medium can exist as discrete components in the network device or terminal device.

[0190] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0191] In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0192] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic. The terms "first," "second," and "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to limit a specific order.

[0193] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0194] In this application, "at least one" means one or more. "More than one" means two or more. "And / or" describes the relationship between the related objects, indicating that there can be three relationships. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, for elements appearing in the singular forms "a", "an", and "the", unless the context explicitly specifies otherwise, it does not mean "one or only one", but rather "one or more one". For example, "a device" means one or more such devices. Moreover, "at least one of..." means one or any combination of the following related objects, such as "at least one of A, B, and C" including A, B, C, AB, AC, BC, or ABC.

Claims

1. An image generation method, characterized in that, The method is applied to an electronic device, the electronic device including a first camera and a second camera, and the method includes: A spectral image of a first scene is acquired using a first camera, the spectral image comprising spectral images of N photosensitive bands; A color image of a second scene is acquired by a second camera. The color image includes M frames of color images, each with a different exposure. Both the color image and the spectral image contain the same target subject. The photosensitive band of the first camera is not less than that of the photosensitive band of the second camera. Using the reconstruction model, reconstructed spectral images of N photosensitive bands of each frame of color image are generated based on the M frames of color images, resulting in MxN reconstructed spectral images. The MxN reconstructed spectral images contain the spectral images of N photosensitive bands of each frame of color image in the M frames of color images, where N and M are both integers greater than or equal to 1. The pixel values ​​of matching pixels in the reconstructed spectral image are corrected based on the pixel values ​​of the spectral image to obtain multiple corrected reconstructed spectral images. The matching pixel refers to a pixel in the reconstructed spectral image that has the same image features in the overlapping area of ​​the image that overlaps with the spectral image. The reconstructed spectral image includes the MxN reconstructed spectral images, and the multiple corrected reconstructed spectral images include the MxN corrected reconstructed spectral images. A corrected color image is generated by fusing the multiple corrected reconstructed spectral images, including: For each of the N photosensitive bands, the corrected and reconstructed spectral images belonging to the same photosensitive band in the MxN corrected and reconstructed spectral images are fused to obtain N fused spectral images; The corrected color image is generated based on the N fused spectral images.

2. The method according to claim 1, characterized in that, The method for generating a corrected color image by fusing the multiple corrected reconstructed spectral images also includes another approach: M intermediate color images are generated from M x N corrected reconstructed spectral images; The corrected color image is obtained by fusing the M intermediate color images.

3. The method according to claim 1 or 2, characterized in that, Correcting the pixel values ​​of matching pixels in the reconstructed spectral image based on the pixel values ​​of the spectral image yields the plurality of corrected reconstructed spectral images, including: For the pixel values ​​of the spectral image of each of the N photosensitive bands, the pixel values ​​of matching pixels belonging to the same photosensitive band in the MxN reconstructed spectral images are corrected to obtain MxN corrected reconstructed spectral images.

4. The method according to claim 3, characterized in that, For the pixel values ​​of the spectral image of each of the N photosensitive bands, correcting the pixel values ​​of matching pixels belonging to the same photosensitive band in the MxN reconstructed spectral images includes: For the pixel values ​​of the spectral image of each of the N photosensitive bands, the pixel values ​​of matching pixels belonging to the same photosensitive band in the effective region of the MxN reconstructed spectral images are corrected. The effective region is the region in the color image where the pixel values ​​are within a preset range, and the range of the effective region of the reconstructed spectral image is the same as the range of the effective region of the color image.

5. The method according to claim 4, characterized in that, The parameters of the reconstruction model are trained based on the sample color image and the sample spectral image. The loss function is calculated based on the sample spectral image and the predicted reconstructed spectral image output by the reconstruction model. The loss function is determined when it converges and the loss function value is less than or equal to a threshold.

6. The method according to any one of claims 1-5, characterized in that, Both the first camera and the second camera are rear-facing cameras, or both the first camera and the second camera are front-facing cameras.

7. The method according to any one of claims 1-6, characterized in that, The corrected color image is a high dynamic range image relative to the color image.

8. An image generation apparatus, characterized in that, include: The acquisition module is used to acquire the spectral image of the first scene, the spectral image including spectral images of N photosensitive bands; The acquisition module is further configured to acquire a color image of the second scene, the color image comprising M frames of color images, each frame of color image having a different exposure level, the color image and the spectral image both containing the same target subject, and the acquisition module acquiring the photosensitive band of the first scene is no less than acquiring the photosensitive band of the second scene; The image generation module is used to generate reconstructed spectral images of N photosensitive bands of each frame of color image based on the M frames of color images using a reconstruction model, thereby obtaining MxN reconstructed spectral images. The MxN reconstructed spectral images contain the spectral images of N photosensitive bands of each frame of color image in the M frames of color images, where N and M are both integers greater than or equal to 1. The image generation module is further configured to correct the pixel values ​​of matching pixels in the reconstructed spectral image based on the pixel values ​​of the spectral image, thereby obtaining multiple corrected reconstructed spectral images. The matching pixel refers to a pixel in the reconstructed spectral image that has the same image features within the overlapping region of the image that overlaps with the spectral image. The reconstructed spectral image includes the M x N reconstructed spectral images, and the multiple corrected reconstructed spectral images include M x N corrected reconstructed spectral images. The image generation module is further configured to generate a corrected color image by fusing the plurality of corrected reconstructed spectral images; When the image generation module generates a corrected color image by fusing the multiple corrected reconstructed spectral images, it is specifically used for: For each of the N photosensitive bands, the corrected and reconstructed spectral images belonging to the same photosensitive band in the MxN corrected and reconstructed spectral images are fused to obtain N fused spectral images; The corrected color image is generated based on the N fused spectral images.

9. The apparatus according to claim 8, characterized in that, When the image generation module generates a corrected color image by fusing the multiple corrected reconstructed spectral images, another specific function is: M intermediate color images are generated from M x N corrected reconstructed spectral images; The corrected color image is obtained by fusing the M intermediate color images.

10. The apparatus according to claim 8 or 9, characterized in that, When the image generation module corrects the pixel values ​​of matching pixels in the reconstructed spectral image based on the pixel values ​​of the spectral image to obtain the plurality of corrected reconstructed spectral images, it is specifically used for: For the pixel values ​​of the spectral image of each of the N photosensitive bands, the pixel values ​​of matching pixels belonging to the same photosensitive band in the MxN reconstructed spectral images are corrected to obtain MxN corrected reconstructed spectral images, and the plurality of corrected reconstructed spectral images include the MxN corrected reconstructed spectral images.

11. The apparatus according to claim 10, characterized in that, When the image generation module corrects the pixel values ​​of matching pixels belonging to the same photosensitive band in the MxN reconstructed spectral images for the pixel values ​​of the spectral image of each of the N photosensitive bands, it is specifically used for: For the pixel values ​​of the spectral image of each of the N photosensitive bands, the pixel values ​​of matching pixels belonging to the same photosensitive band in the effective region of the MxN reconstructed spectral images are corrected. The effective region is the region in the color image where the pixel values ​​are within a preset range, and the range of the effective region of the reconstructed spectral image is the same as the range of the effective region of the color image.

12. The apparatus according to claim 11, characterized in that, The parameters of the reconstruction model are trained based on the sample color image and the sample spectral image. The loss function is calculated based on the sample spectral image and the predicted reconstructed spectral image output by the reconstruction model. The loss function is determined when it converges and the loss function value is less than or equal to a threshold.

13. The apparatus according to any one of claims 8-12, characterized in that, The acquisition module is a rear camera, or the acquisition module is a front camera.

14. The apparatus according to any one of claims 8-13, characterized in that, The corrected color image is a high dynamic range image relative to the color image.

15. An electronic device, characterized in that, include: The system includes at least one processor, a memory, a first camera, and a second camera, wherein the first camera is used to acquire spectral images, the second camera is used to acquire color images, the memory is used to store computer programs and instructions, and the processor is used to invoke the computer programs and instructions to assist the first camera and the second camera in performing the image generation method as described in any one of claims 1-7.

16. A chip system, characterized in that, The chip system is applied to an electronic device; the chip system includes an interface circuit and a processor; the interface circuit and the processor are interconnected by a line; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the chip system executes the image generation method as described in any one of claims 1-7.

17. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the image generation method as described in any one of claims 1-7.

18. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a computer, implement the image generation method as described in any one of claims 1-7.

Images