Image processing method and related device

Abstract

The embodiment of the invention provides an image processing method, and the method comprises the steps: determining first light source part information through a first multispectral image collected by a multispectral sensor and a first model in an end-to-end manner, and carrying out the adjustment of the first image according to the first light source distribution information. And determining first light source distribution information end to end through the multispectral image and the first model. On one hand, the multi-spectral image of multiple channels can be fully utilized, the accuracy of light source decoupling can be improved, and the situation that a scene is mistakenly decolorized due to the influence of metamerism is avoided. On the other hand, the upper limit of end-to-end optimization precision is high, the balance effect of the light source transition area is more natural, the reasoning process is simple, and the deployment cost is low. And on the other hand, continuous light source distribution prediction can be realized according to the first model.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method and related devices. Background Art

[0002] As users' photography needs increase, so do their expectations for the quality of images captured by electronic devices. When users capture images using electronic devices, differences in shooting environments can lead to differences between the images captured by the electronic devices and the actual objects. For camera imaging, white balance is a key factor affecting the quality of the images.

[0003] At present, the local white balance algorithm is the key to solving mixed light source scenes. The most commonly used local white balance method in the industry. One method is to perform white balance processing on the display graphics according to the red, green, blue (RGB) linear image. Another method is: spatial discrete or frequency domain discrete adjustment method. For example, the spatial discrete method is to divide the input image into multiple sub-images and perform white balance separately. For another example, the frequency domain discrete method is to use the color temperature weight calculation model to infer the image to be adjusted, and the optimal weighting coefficient combination corresponding to different preset color temperatures can be obtained, and the weighted sum of each preset white balance effect is taken accordingly. That is, the overall white balance correction effect is achieved through the preset of the spatial domain or frequency domain.

[0004] However, the two methods mentioned above have a low number of RGB image channels, resulting in poor white balance results. The other method uses a weighted recombination of discrete preset effects by predicting weighting coefficients, which has a low upper limit on accuracy. Summary of the Invention

[0005] Embodiments of the present application provide an image processing method and related apparatus. This method uses a first multispectral image acquired by a multispectral sensor and a first model to determine partial information about a first light source. The first light source distribution information is continuous light source distribution information for the first multispectral image in both the spatial and frequency domains. Therefore, adjusting the first image based on the first light source distribution information can improve the accuracy of the light source distribution.

[0006] The first aspect of the present application provides an image processing method, which is performed by an image processing device, or the method is performed by some components in the image processing device (such as a processor, a chip or a chip system, etc.), or the method can also be implemented by a logic module or software that can realize all or part of the functions of the image processing device. In the first aspect and its possible implementation, the method is described as being performed by an image processing device, and the image processing device includes a multispectral sensor and a first sensor. In the method, the image processing device collects a first multispectral image through the multispectral sensor and collects a first image through the first sensor. The image processing device determines the first light source distribution information according to the first model. The image processing device adjusts the first image according to the first light source distribution information to obtain a second image.

[0007] The first model is used to predict the continuous light source distribution information of the image in the spatial domain and the frequency domain. That is, the first light source distribution information is the continuous light source distribution information of the first multispectral image in the spatial domain and the frequency domain.

[0008] Based on the above scheme, the first light source distribution information is determined by combining the first multispectral image acquired by the multispectral sensor with the first model. This allows full utilization of the multi-channel multispectral image to obtain the light source distribution information. Furthermore, because the first light source distribution information is continuous in both the spatial and frequency domains, adjusting the first image based on the first light source distribution information can achieve a high optimization accuracy limit and a more natural light source transition balance effect.

[0009] In one possible implementation, the image processing device may determine second light source distribution information corresponding to the first image based on the first light source distribution information; and then perform local light source compensation on the first image based on the second light source distribution information to obtain the second image. The second light source distribution information is light source distribution information that is continuous in the spatial and frequency domains, resulting from mapping the first light source distribution information to the first image.

[0010] In this possible implementation, the first light source distribution information of the first multispectral image is mapped onto the first image to obtain the second light source distribution information of the first image. Light source prediction does not require pixel-by-pixel alignment between the multispectral device and the imaging device, and light source learning accuracy is not affected by spatial misalignment. This reduces hardware requirements and power consumption.

[0011] In a possible implementation, the image processing device may specifically perform local light source compensation on the first multispectral image according to the first light source distribution information to obtain a third image; and then adjust the first image according to the third image to obtain the second image.

[0012] In this possible implementation, local light source compensation is first performed on the first multispectral image using the first light source distribution information. A third image, obtained by performing local light source compensation on the first multispectral image, is then aligned with the first image to produce a second image. This image alignment allows the adjusted second image to represent the light source distribution of the third image, improving the imaging quality of the second image. This significantly reduces local light source missegmentation caused by metamerism and avoids erroneous color loss.

[0013] In one possible implementation, the image processing device may further capture a second multispectral image using a multispectral sensor, where the first and second multispectral images are captured at different times. Accordingly, the image processing device may determine the first light source distribution information based on the first and second multispectral images.

[0014] In this possible implementation, the first light source distribution information is predicted by multispectral images collected at different times. The image features of the multispectral images at different times can be combined to make the predicted light source distribution information smoother, so as to reduce timing jumps or color instability.

[0015] In a possible implementation manner, the image processing device may specifically input the first multispectral image into the first model to obtain the first light source distribution information.

[0016] In this possible implementation, the first light source distribution information determined by the first multispectral image can be used to subsequently adjust the first image to improve the imaging effect of the adjusted first image.

[0017] In a possible implementation manner, the image processing device may specifically input the first multispectral image and the first image into a first model to obtain the first light source distribution information.

[0018] In this possible implementation, in addition to using the multispectral image, the first model can also determine the first light source distribution information in combination with the first image, which can improve the accuracy of the light source distribution information.

[0019] In one possible implementation, the first model includes a first module, a second module, and a third module; the image processing device can specifically determine the image features corresponding to the first multispectral image based on the first module; determine the spectral features corresponding to the image features based on the second module; and determine the first light source distribution information corresponding to the spectral features based on the third module.

[0020] In this possible implementation, the first light source distribution information is determined by the spectral characteristics corresponding to the image characteristics of the first multispectral image, which can maximize the capabilities of the multispectral device. At the same time, any mixed light source scene can be continuously compensated in the spatial domain and frequency domain to improve the imaging effect of the compensated image.

[0021] In one possible implementation, the first model further includes a fourth module. Accordingly, the image processing device may first fuse the image features of the first multispectral image with the image features of the second multispectral image based on the fourth module to obtain fused image features, and then determine the spectral features based on the second module and the fused image features.

[0022] In this possible implementation, the multispectral images at different times can be combined through a fusion module to improve the smoothness and integrity of the spectral characteristics, thereby improving the richness and accuracy of the subsequent first light source distribution information.

[0023] In a possible implementation manner, the image processing device may specifically determine the first light source distribution information end-to-end based on the first model.

[0024] In this possible implementation method, the first light source distribution information is determined end-to-end based on the first model. On the one hand, it reduces the accuracy loss caused by multiple modules. On the other hand, it does not rely on pre-segmentation and color temperature presets. The inference process is simple and the deployment cost is low.

[0025] In a possible implementation, the first sensor includes a multispectral sensor and / or a non-multispectral sensor.

[0026] In this possible implementation, the first sensor includes a multispectral sensor and / or a non-multispectral sensor, so that it can be applied to various image sensor scenarios to improve the applicability of the image processing method.

[0027] In a possible implementation, the first light source distribution information includes a plurality of light source values, and the plurality of light source values ​​correspond to a plurality of pixel values ​​in the first multispectral image. Alternatively, the plurality of light source values ​​correspond to a plurality of pixel values ​​in the first image.

[0028] In this possible implementation, the effect of subsequently adjusting the first image according to the light source distribution can be improved by distributing the light source values ​​of the image pixel by pixel.

[0029] In one possible implementation, the size of the first multispectral image is h1*w1*c1, and the size of the first light source distribution information is h2*w2*c2, where h1 represents the height of the first multispectral image, w1 represents the width of the first multispectral image, c1 represents the number of channels of the first multispectral image, h2 represents the height of the first light source distribution information, w2 represents the width of the first light source distribution information, c2 represents the number of channels of the first light source distribution information, h1 is greater than 1, w1 is greater than 1, c1 is greater than 4, h2 is greater than 1, w2 is greater than 1, and c2 is greater than 1.

[0030] In this possible implementation, the entire image is input and output, which not only fully utilizes the multi-channel multispectral image, but also improves the accuracy of light source decoupling and is less susceptible to the influence of metamerism and incorrect decolorization of the scene.

[0031] A second aspect of the present application provides an image processing device (or image processing system), the image processing device comprising: a first sensor, a multispectral sensor, and a processing unit;

[0032] The first sensor is used to capture a first image;

[0033] The multispectral sensor is used to collect multispectral images;

[0034] The processing unit is configured to determine first light source distribution information according to the first model end, where the first light source distribution information is continuous light source distribution information of the first multispectral image in the spatial domain and the frequency domain.

[0035] The processing unit is further configured to adjust the first image according to the first light source distribution information to obtain a second image.

[0036] In one possible implementation, the processing unit is specifically configured to determine second light source distribution information corresponding to the first image based on the first light source distribution information. The processing unit is specifically configured to perform local light source compensation on the first image based on the second light source distribution information to obtain the second image. The second light source distribution information is light source distribution information that is continuous in the spatial and frequency domains resulting from mapping the first light source distribution information to the first image.

[0037] In a possible implementation, the processing unit is specifically configured to perform local light source compensation on the first multispectral image according to the first light source distribution information to obtain a third image. The processing unit is specifically configured to adjust the first image according to the third image to obtain a second image.

[0038] In one possible embodiment, the multispectral sensor is configured to capture a second multispectral image, wherein the first multispectral image and the second multispectral image are multispectral images captured at different times. The processing unit is specifically configured to determine the first light source distribution information based on the first multispectral image and the second multispectral image.

[0039] In a possible implementation, the processing unit is specifically configured to input the first multispectral image into the first model to obtain the first light source distribution information.

[0040] In a possible implementation, the processing unit is specifically configured to input the first multispectral image and the first image into a first model to obtain first light source distribution information.

[0041] In one possible embodiment, the first model includes a first module, a second module, and a third module. The processing unit is specifically configured to determine image features corresponding to the first multispectral image based on the first module; the processing unit is specifically configured to determine spectral features corresponding to the image features based on the second module; and the processing unit is specifically configured to determine first light source distribution information corresponding to the spectral features based on the third module.

[0042] In one possible embodiment, the first model also includes a fourth module; a processing unit, specifically used to fuse the image features of the first multispectral image with the image features of the second multispectral image according to the fourth module to obtain fused image features; and a processing unit, specifically used to determine the spectral features based on the second module and the fused image features.

[0043] In a possible implementation, the processing unit is specifically configured to determine the first light source distribution information end-to-end according to the first model.

[0044] In a possible implementation, the first sensor includes a multispectral sensor and / or a non-multispectral sensor.

[0045] In a possible implementation, the first light source distribution information includes a plurality of light source values, where the plurality of light source values ​​correspond to a plurality of pixel values ​​in the first multispectral image.

[0046] In one possible embodiment, the size of the first multispectral image is h1*w1*c1, and the size of the first light source distribution information is h2*w2*c2, where h1 represents the height of the first multispectral image, w1 represents the width of the first multispectral image, c1 represents the number of channels of the first multispectral image, h2 represents the height of the first light source distribution information, w2 represents the width of the first light source distribution information, c2 represents the number of channels of the first light source distribution information, h1 is greater than 1, w1 is greater than 1, c1 is greater than 4, h2 is greater than 1, w2 is greater than 1, and c2 is greater than 1.

[0047] A third aspect of the present application provides an image processing system (or imaging system), comprising: a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor invokes program code in the memory to execute the processing-related functions of any of the image processing methods described in the first aspect. Optionally, the imaging system may include a chip.

[0048] In a fourth aspect, the present application provides an electronic device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing as described in the first aspect or any optional embodiment of the first aspect.

[0049] In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect or any optional embodiment of the first aspect.

[0050] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the first aspect or any optional embodiment of the first aspect.

[0051] In the seventh aspect of the present application, a chip is provided, comprising at least one processor and an interface; at least one processor obtains program instructions or data through the interface; and at least one processor is used to execute program line instructions to implement the method in the first aspect or any implementation of the first aspect.

[0052] Among them, the technical effects brought about by any design method in the second to seventh aspects can refer to the technical effects brought about by the different design methods in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of the structure of an electronic device provided in this application;

[0054] Figure 2 A schematic diagram of a system architecture provided for this application;

[0055] Figure 3 A flowchart of an image processing method provided in this application;

[0056] Figure 4 An example diagram of the first model provided in this application;

[0057] Figure 5A Another example diagram of the first model provided in this application;

[0058] Figure 5B Another example diagram of the first model provided in this application;

[0059] Figure 6 A flowchart of another image processing method provided in this application;

[0060] Figure 7 A flowchart of another image processing method provided in this application;

[0061] Figure 8 A schematic diagram of the architecture of another image processing system provided in this application;

[0062] Figure 9 This is a schematic structural diagram of another electronic device provided in this application. DETAILED DESCRIPTION

[0063] The following will describe the technical solutions in the embodiments of this application in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0064] To facilitate understanding, some terms or concepts involved in the method provided in this application are explained below.

[0065] (1) Color Constancy

[0066] This refers to the perceptual property whereby the perception of an object's surface color remains constant even when the color of the light illuminating it changes. In the field of image processing, separating background elements from illumination elements in an image is crucial, based on the human eye's cognitive characteristics.

[0067] (2) White Balance (WB)

[0068] Its basic concept is to restore white objects to white regardless of light source. It compensates for color casts that occur when shooting under specific light sources by enhancing the corresponding complementary colors. White balance generally describes the accuracy of white produced by mixing the three primary colors of red, green, and blue. It can address a range of issues related to color reproduction and color tone processing, ensuring that camera images accurately reflect the color of the subject.

[0069] (3) Image Signal Processor (ISP)

[0070] The image sensor processes the image signal output by the image sensor. It plays a central and dominant role in the camera system and is a crucial component of the camera. Its main functions include demosaicing, automatic exposure, automatic white balance, lens shading removal, gamma correction, color space conversion, dynamic range correction, and image cropping.

[0071] (4) Multi-camera consistency

[0072] Array cameras are a common way to deploy cameras in electronic devices. The main difference between these cameras is the focal length, and they are mainly divided into the first camera, the second camera, and the third camera. For example, the first camera is a wide-angle camera (main camera), the second camera is an ultra-wide-angle camera, and the third camera is a telephoto camera. Different mobile phones may have different configurations. The above three cameras are just examples. Usually, the number of cameras is set to at least two. These cameras can all independently image and have the need to take pictures and record them. They will all be presented directly to the user for browsing, so users have high requirements for the color consistency of each camera in the array camera.

[0073] (5) Multi-spectral Imaging / Image (MSI)

[0074] MSI refers to a spectral detection technology that can simultaneously capture multiple optical spectrum bands (usually more than three) and expand beyond visible light to infrared and ultraviolet light. Compared to RGB images, MSI can contain information from more channels, such as ultraviolet, near-infrared, or short-wave infrared. For example, it can include information on channels such as luminance (Y), violet (C), magenta (M), green (P), or visible light (V). Therefore, MSI images can contain more color information.

[0075] Furthermore, for MSI images containing a large number of channels, a hyperspectral image (HSI) can also be defined. Compared to conventional MSI, HSI covers a far greater number of spectral bands, capable of distinguishing thousands of individual spectral bands, providing richer and more detailed spectral information. Furthermore, HSI can capture many extremely narrow bands, typically between 10 and 20 nanometers in width. This high resolution enables HSI to more precisely analyze the composition and properties of matter.

[0076] (6) Spatial / Spectral Resolution

[0077] An image is represented by H×W×C, where H×W represents the spatial resolution of the image and C represents the spectral resolution of the image. For example, a common 3-channel color image has a spectral resolution of 3, while for multispectral images, the spectral resolution C is usually greater than 3.

[0078] (7) Raw data: Raw data records the original information of the camera sensor in an unprocessed and uncompressed format. RAW can be conceptualized as "raw image encoding data" or more figuratively as "digital negative."

[0079] (8) Metamerism

[0080] It means that the imaging signal (or color) is the same, but the spectral composition is different.

[0081] In terms of the deployment method of the method provided in this application, the method provided in this application can be divided into multiple deployment methods. For example, the method provided in this application can be deployed in an electronic device (i.e., the image processing device is an electronic device), and the user can directly use the electronic device to perform image processing. Alternatively, it can be deployed in the cloud to provide image processing services to user terminals. The following introduces different deployment methods.

[0082] Deployment method 1: Deploy in electronic equipment.

[0083] The electronic devices provided in the embodiments of the present application may specifically include handheld devices, vehicle-mounted devices, image processing devices, computing devices, and other electronic devices that include image sensors or are connected to image sensors. They may also include digital cameras, cellular phones, cameras, smart phones, personal digital assistants (PDAs), tablet computers, laptop computers, machine type communication (MTC) terminals, point of sales (POS), vehicle-mounted computers, head-mounted devices, image processing devices (such as bracelets, smart watches, etc.), security equipment, virtual reality (VR) devices, augmented reality (AR) devices, and other electronic devices with imaging capabilities.

[0084] Take a digital camera, for example. Digital cameras, short for digital cameras, are cameras that use photoelectric sensors to convert optical images into digital signals. Unlike traditional cameras, which rely on changes in photosensitive chemicals on film to record images, digital camera sensors are light-sensitive charge-coupled devices (CCDs) or complementary metal oxide semiconductors (CMOSs). Compared to traditional cameras, digital cameras offer advantages such as greater convenience, speed, repeatability, and timeliness, as they directly utilize image sensors that convert photoelectric energy. With the advancement of CMOS processing technology, digital cameras have become increasingly powerful, almost completely replacing traditional film cameras and enjoying widespread application in consumer electronics, human-computer interaction, computer vision, autonomous driving, and other fields.

[0085] For example, Figure 1 A schematic diagram of an electronic device provided by the present application is shown. As shown in the figure, the electronic device may include a lens group 110, an image sensor 120, and an electrical signal processor 130. The electrical signal processor 130 may include an analog-to-digital (A / D) converter 131 and a digital signal processor 132. The A / D converter 131 is an analog signal to digital signal converter, which is used to convert an analog electrical signal into a digital electrical signal.

[0086] It should be understood that Figure 1 The electronic device shown in is not limited to the above devices, and may also include more or fewer other devices, such as batteries, flashes, buttons, sensors, etc. The embodiment of the present application only uses the electronic device equipped with the image sensor 120 as an example for illustration, but the components installed on the electronic device are not limited to this.

[0087] In the embodiment of the present application, the aforementioned image sensor 120 may specifically include an image sensor, a multispectral image (MSI) sensor, etc.

[0088] The light signals reflected by the subject are focused by the lens assembly 110 and formed on the image sensor 120. The image sensor 120 converts the light signals into analog electrical signals. In the electrical signal processor 130, the analog electrical signals are converted into digital electrical signals by an analog-to-digital (A / D) converter 131. The digital electrical signals are then processed by the digital signal processor 132, for example, by performing a series of complex mathematical operations to optimize the digital electrical signals and ultimately output an image. The electrical signal processor 130 may also include an analog signal preprocessor 133, which is used to preprocess the analog electrical signals transmitted by the image sensor before outputting them to the analog-to-digital converter 131.

[0089] The performance of an image sensor affects the quality of the final output image. Image sensors, also known as photosensitive chips or photosensitive elements, contain hundreds of thousands to millions of photoelectric conversion elements. When exposed to light, they generate an electric charge, which is converted into a digital signal by an analog-to-digital converter chip. Image sensors consist of photosensitive elements with multiple pixels, and they achieve imaging through photoelectric response.

[0090] Among them, the MSI sensor can simultaneously collect image signals from multiple spectral bands. The multispectral band contains more frequency bands, and the color information contained in the MSI image is also richer, which has the potential to be used to improve the effect of RGB images or videos. Therefore, the method provided in the present application can be specifically deployed in the electronic signal processor 130 of the electronic device, for example, it can be specifically deployed in the digital signal processor 132, or it can be deployed in other processors of the electronic device. The method provided in the embodiment of the present application can use at least one multispectral image collected by the multispectral sensor and the first model to determine the first light source distribution information, and the first model can be used to predict the continuous light source distribution information of the image in the spatial domain and frequency domain. On the one hand, it can not only make full use of multi-channel multispectral images, but also improve the accuracy of light source decoupling, and is less susceptible to the influence of different spectra and incorrect decolorization of the scene. On the other hand, the first model can also have a high upper limit of end-to-end optimization accuracy, a more natural light source transition zone balancing effect, a simple inference process, and a low deployment cost. On the other hand, continuous light source distribution prediction can be achieved based on the first model.

[0091] The first model may be stored in the terminal device so as to improve the imaging efficiency of the terminal device.

[0092] Furthermore, the digital signal processor may specifically include a central processing unit (CPU), a graphics processing unit (GPU), a neural-network processing unit (NPU), a tensor processing unit (TPU) or an application-specific integrated circuit (ASIC), etc.

[0093] Deployment method 2: Deploy in the cloud.

[0094] The present application also provides a cloud platform, one or more terminals accessing the platform. The method provided in the present application can be deployed in the cloud (ie, the image processing device is a cloud device) to provide image processing services (such as image enhancement, image noise reduction, etc.) to the terminal.

[0095] For example, Figure 2 This is an application scenario of the method provided in this application, which may include a cloud 11 and a terminal 12. The cloud 11 and the terminal 12 may be connected via a wired or wireless connection.

[0096] The cloud 11 may specifically include a server cluster with storage and processing capabilities. The method provided in the embodiments of the present application may be deployed in the cloud 11 and may specifically receive at least one multispectral image and a first image from a terminal 12, determine first light source distribution information based on the at least one multispectral image and the first model, adjust the first image based on the first light source distribution information, and feed the adjusted image back to the terminal 12.

[0097] Among them, the first model can be stored in the cloud 11, which can reduce the storage of the terminal device and improve the imaging effect through the computing power of the cloud 11.

[0098] The terminal 12 can implement image processing by interacting with the cloud 11. The terminal may specifically include, but is not limited to, for example, a personal computer, a computer workstation, a smart phone, a tablet computer, a laptop computer, and a smart car. The terminal 12 can transmit an image (for example, including at least one multispectral image and / or a first image) to the cloud 11. The image can be an image captured by the terminal itself, or an image input by the user, or an image locally stored in the terminal. For example, the cloud can provide services to the user through a client deployed in the terminal or a web page in the terminal. Taking the deployment of the client in the terminal as an example, the user can send the image collected by the terminal to the cloud through the client deployed in the terminal, such as transmitting an RGB image and an MSI image. The cloud 11 adjusts the RGB image through the method provided in the embodiment of the present application, outputs the adjusted RGB image, and feeds back to the terminal 12.

[0099] In a possible scenario, it can also be applied to scenarios with multiple terminals. For example, a user can use a terminal other than the aforementioned terminal 12 to capture the aforementioned RGB image and MSI image, and transmit the RGB image and MSI image to the aforementioned terminal 12. After the terminal 12 uploads the RGB image and MSI image to the cloud 11 for image processing, the adjusted image can be fed back to the terminal 12, and the terminal 12 can feed back the adjusted image to the terminal that captured the aforementioned RGB image and MSI image.

[0100] As users' demand for taking photos gradually increases, the quality requirements for images captured by electronic devices are also getting higher and higher. When users use electronic devices to capture images, due to different shooting environments, there are differences between the images of the electronic devices and the real objects. For camera imaging, white balance is a key factor affecting the imaging effect. At present, the most commonly used white balance methods in the industry. One method is to perform white balance processing on the displayed image based on the red, green, and blue (RGB) linear image. Another method is: spatial discrete or frequency domain discrete adjustment method. For example, the spatial discrete method is to divide the input image into multiple sub-images and perform white balance on each of them. For another example, the frequency domain discrete method is to use the color temperature weight calculation model to infer the image to be adjusted, and the optimal weighting coefficient combination corresponding to different preset color temperatures can be obtained, and the weighted sum of each preset white balance effect is taken accordingly. That is, the overall white balance correction effect is achieved through spatial or frequency domain presets.

[0101] However, the two methods mentioned above have a low channel count for RGB images, resulting in poor white balance results. The other method uses a weighted recombination of discrete preset effects by predicting weighted coefficients, which has a low upper limit on accuracy.

[0102] To address the aforementioned technical issues, embodiments of the present application provide an image processing method and related apparatus. By using a first multispectral image acquired by a multispectral sensor and a first model to determine first light source distribution information, this method can fully utilize multi-channel multispectral images to obtain light source distribution information. Furthermore, because the first light source distribution information is continuous in both the spatial and frequency domains, adjusting the first image based on the first light source distribution information can optimize the image with a high upper limit of accuracy and achieve a more natural light source transition balance effect.

[0103] The image processing method provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0104] See also Figure 3 The present application provides an embodiment of an image processing method, which can be performed by an image processing device (also referred to as an image processing device, including the aforementioned Figure 1 Electronic devices and / or Figure 2 The method may be executed by a component of an image processing device (such as a processor, an image sensor, an electrical signal processor, a chip, or a chip system), and the method includes steps 301 to 304. Alternatively, it can be understood that the image processing method provided in the embodiment of the present application can be deployed in a terminal or in the cloud, and the specific details are not limited here. For the convenience of description, the image processing method will be described below using an image processing device as an example. In addition, the image processing method provided in the embodiment of the present application can be applied to local white balance in a multi-light source scene or a so-called mixed light source scene.

[0105] Due to the long intervals between each step, steps 301 to 304 will be briefly described here, and then described separately in detail later. Step 301: The image processing device collects a first multispectral image using a multispectral sensor. Step 302: The image processing device collects a first image using a first sensor. Step 303: The image processing device determines the first light source distribution information based on the first model. Step 304: The image processing device adjusts the first image based on the first light source distribution information to obtain a second image. The following is a detailed description of each step:

[0106] In step 301 , an image processing device collects a first multispectral image through a multispectral sensor.

[0107] In an embodiment of the present application, the image processing device includes a multispectral sensor. The number of multispectral sensors may be one or more, and the specific number is not limited herein. Similarly, the number of multispectral images collected by the multispectral sensor may also be one or more, and the specific number is not limited herein.

[0108] Optionally, at least one multispectral image can be captured by an MSI sensor. In some scenarios, the MSI mentioned in the embodiments of the present application may also include sensors such as HSI that can be used to capture multispectral images. Typically, the number of channels of the multispectral image is greater than the number of channels of the first image captured by the subsequent first sensor.

[0109] Optionally, the at least one multispectral image includes a first multispectral image. Further optionally, the at least one multispectral image includes a first multispectral image and a second multispectral image, and the first multispectral image and the second multispectral image are multispectral images acquired at different times.

[0110] Of course, in order to subsequently determine the accuracy of the first light source distribution information based on the multispectral image, the time interval between the acquisition time of the first multispectral image and the acquisition time of the second multispectral image must be less than a threshold. Alternatively, it can be understood that the first multispectral image and the second multispectral image are acquired at similar times. Alternatively, it can be understood that the first multispectral image and the second multispectral image are adjacent frames (which can be either completely separated or have a relatively small separation between them).

[0111] For example, the first multispectral image and the first image are acquired at the same time, and the second multispectral image is acquired at a time close to that of the first multispectral image.

[0112] In a possible implementation, the image processing device is a terminal device, and the terminal device collects at least one multispectral image through its own multispectral sensor.

[0113] In another possible implementation, the image processing device is a cloud device that acquires a multispectral image by receiving information sent by a terminal device. For example, after the terminal device acquires at least one multispectral image using a multispectral sensor, it transmits the at least one multispectral image to the cloud device. In response, the cloud device receives the at least one multispectral image sent by the terminal device.

[0114] For example, at least one multispectral image may be raw data. Figure 1 In the digital signal processor 132 of the electronic device, the multispectral image can be a digital signal output by the analog-to-digital converter 131, that is, raw data. For example, if the method provided in the embodiment of the present application is deployed in the aforementioned Figure 2 In the cloud 11, the at least one multispectral image may be an MSI image captured by the terminal 12, which may also be referred to as raw data. After capturing the MSI image, the terminal 12 sends the MSI image to the cloud 11. Accordingly, the cloud 11 receives the MSI image sent by the terminal 12.

[0115] It is understandable that the at least one multispectral image may also be raw data collected by the multispectral sensor and preprocessed, such as noise filtering or brightness correction, to obtain preprocessed multispectral raw data.

[0116] Step 302: The image processing device captures a first image through a first sensor.

[0117] In the embodiment of the present application, the image processing device includes a first sensor. Similarly, the number of the first sensors can be one or more, and the specific number is not limited here. Similarly, the number of the first images can also be one or more, and the specific number is not limited here.

[0118] The first sensor may also be referred to as a first image sensor or a three-channel sensor. The first sensor includes a multispectral sensor and / or a non-multispectral sensor.

[0119] Optionally, the first sensor includes at least one sensor. For example, the at least one sensor includes at least one of a first sensor, a second sensor, and a third sensor. The above sensors may include one or more of the following: a main camera (which may be referred to as a main camera), a wide-angle camera, or a telephoto camera, etc., and the specific details are not limited here. Alternatively, it can be understood that the at least one sensor includes at least one of a main camera sensor, a wide-angle sensor, and a telephoto sensor.

[0120] Further optionally, in order to improve the accuracy of subsequent partitioning parameters, the time when the first sensor collects the first image and the time when the multispectral sensor collects the multispectral image are the same, the position is the same, the weather is the same, the time is similar, the position is similar, or the weather is similar. For example, the time is similar, which can be specifically: the difference between the time when the first sensor collects the first image and the time when the multispectral sensor collects the multispectral image is less than a first threshold (for example, 2 hours, 10 minutes, or 5 seconds, etc.). For another example, the position is similar, which can be specifically: the difference between the position of the image processing device when the first sensor collects the first image and the position of the image processing device when the multispectral sensor collects the multispectral image is less than a second threshold (for example, 2 meters, 0.5 meters, or 0.1 meters, etc.). For another example, the weather is similar, which can be specifically: the weather is rainy when the first sensor collects the first image, and it is cloudy when the multispectral sensor collects the multispectral image.

[0121] In a possible implementation, the image processing device is a terminal device, and the terminal device collects the first image through its own first sensor.

[0122] In another possible implementation, the image processing device is a cloud device, and the cloud device obtains the first image by receiving a message from the terminal device. For example, after the terminal device captures the first image using a first sensor, it transmits the first image to the cloud device. In response, the cloud device receives the first image transmitted by the terminal device.

[0123] For example, the first image may be raw data. Figure 1 In the digital signal processor 132 of the electronic device, the first image may be a digital signal output by the analog-to-digital converter 131, that is, raw data. For another example, if the method provided in the embodiment of the present application is deployed in the aforementioned Figure 2 In the cloud 11, the first image can be an RGB image captured by the terminal 12, which can also be called raw data. After the terminal 12 captures the RGB image, it sends the RGB image to the cloud 11. Correspondingly, the cloud 11 receives the RGB image sent by the terminal 12.

[0124] It is understandable that the first image may also be raw data collected by the first sensor and preprocessed, such as noise filtering or brightness correction, to obtain preprocessed RGB image raw data.

[0125] Step 303: The image processing device determines first light source distribution information according to the first model.

[0126] After acquiring the first multispectral image, the image processing device determines first light source distribution information based on a first model. The first model is used to predict continuous light source distribution information for the image in the spatial and frequency domains. Specifically, the first light source distribution information is continuous light source distribution information for the first multispectral image in the spatial and frequency domains predicted by the first model.

[0127] Spatial continuity can be understood as each pixel corresponding to the light source information, rather than being divided into regions. Frequency continuity can be understood as the color temperature corresponding to different light sources has a transition.

[0128] Alternatively, it can be understood that the image processing device determines the first light source distribution information based on the first multispectral image and the first model. For example, the first multispectral image is input into the first model to obtain the first light source distribution information.

[0129] In addition, the image processing device can also determine the first light source distribution information end-to-end based on the first model without relying on pre-segmentation and color temperature preset. The inference process is simple and the deployment cost is low.

[0130] Among them, the first model may include one or more of the following: convolution layer, pooling layer, activation layer, upsampling layer, downsampling layer, self-attention layer, cross-attention layer, residual layer, etc. The structure of the first model, the number of layers and the loss function used in the training process are not specifically limited here.

[0131] Furthermore, during the training process, the first model is trained using the training data as input, with the goal of achieving a loss function value less than a threshold. The loss function represents the difference between the predicted value of the training data after the first model and the label value. The label value can be obtained by using a pixel-level light source distribution of a hand mark or multiple light source maps, etc., without limitation here.

[0132] The form of the light source distribution information (such as the first light source distribution information and the second light source distribution information) in the embodiments of the present application includes one or more of the following: pixel-by-pixel light source value distribution, K light source values ​​and corresponding weight distribution diagrams, etc., where K is a real number greater than 1, and is not specifically limited here.

[0133] For example, the first light source distribution information includes a plurality of light source values, and the plurality of light source values ​​correspond to a plurality of pixel values ​​in the first multispectral image, or the plurality of light source values ​​correspond to a plurality of pixel values ​​in the first image.

[0134] For another example, the input of the first model is at least one entire multispectral image, and the output is the light source distribution of the entire image. For example, the size of the first multispectral image is h1*w1*c1, and the size of the first light source distribution information is h2*w2*c2, where h1 represents the height of the first multispectral image, w1 represents the width of the first multispectral image, c1 represents the number of channels of the first multispectral image, h2 represents the height of the first light source distribution information, w2 represents the width of the first light source distribution information, and c2 represents the number of channels of the first light source distribution information. h1 is greater than 1, w1 is greater than 1, c1 is greater than 4, h2 is greater than 1, w2 is greater than 1, and c2 is greater than 1.

[0135] In one possible implementation, the image processing device is a terminal device, which can predict first light source distribution information for a first multispectral image based on a locally stored first model. Alternatively, the terminal device can send the first multispectral image to a cloud device, which can predict first light source distribution information for the first multispectral image based on the first model and send the first light source distribution information to the terminal device.

[0136] In another possible implementation, the image processing device is a cloud device, which receives the first multispectral image sent by the terminal device and predicts first light source distribution information of the first multispectral image based on the first model.

[0137] Optionally, the image processing device inputs the first multispectral image into the first model to obtain first light source distribution information.

[0138] Exemplarily, the first model includes a first module, a second module, and a third module. The first module is used to extract image features, the second module is used to extract spectral features of the image features, and the third module is used to extract first light source distribution information corresponding to the spectral features. The first module can also be referred to as a feature extraction module. The second module can also be referred to as a light source analysis module. The third module can also be referred to as a distribution recovery module.

[0139] For example, Figure 4 As shown, the image processing device determines image features corresponding to the first multispectral image based on the first module. The image processing device determines spectral features corresponding to the image features based on the second module. The image processing device determines first light source distribution information corresponding to the spectral features based on the third module. Alternatively, the image processing device restores the spectral features determined by the second module to the light source distribution corresponding to the input space to obtain the first light source distribution information.

[0140] Further optionally, the image processing device inputs the first multispectral image and the second multispectral image into a first model to obtain first light source distribution information.

[0141] For example, the first multispectral image and the first image were collected at the same time, and the second multispectral image was collected at a similar time. The image processing device inputs the first and second multispectral images into the first model, allowing the first model to refer to the light source distribution corresponding to the previous multispectral image during inference, thereby improving the accuracy of the first light source distribution information.

[0142] Exemplarily, the first model also includes a fourth module, that is, the first model includes a first module, a second module, a third module and a fourth module. The functions of the first module and the third module should be similar to those described above. The fourth module is used to fuse the information of the second multispectral image (for example, including the image features of the second multispectral image or the spectral features of the second multispectral image or the light source distribution information of the second multispectral image) with the information of the first multispectral image. The fourth module can also be called a temporal fusion module, which improves the correlation between frames by fusing multispectral images of different frames.

[0143] It should be noted that the temporal position of the fourth module in the first model can vary. For example, if the fourth module is used to fuse the image features of a multispectral image, the fourth module can be connected to either the first module or the second module. For another example, if the fourth module is used to fuse the spectral features of a multispectral image, the fourth module can be connected to either the third module or the second module. For another example, if the fourth module is used to fuse the light source distribution information of a multispectral image, the fourth module can be connected to the third module.

[0144] For example, an example of the first model is Figure 5A As shown, the image processing device determines the image features corresponding to the first multispectral image according to the first module. The image processing device fuses the image features of the first multispectral image with the image features of the second multispectral image according to the fourth module to obtain fused image features. The image processing device determines the spectral features corresponding to the fused image features according to the second module. The image processing device determines the first light source distribution information corresponding to the spectral features according to the third module. Because the spectral features take into account the correlation between different frames, the first light source distribution information determined based on the spectral features combines the image features of different frames. This improves the accuracy of the light source distribution prediction.

[0145] For another example, an example of the first model is Figure 5B As shown, the image processing device determines image features corresponding to the first multispectral image based on the first module. The image processing device determines spectral features corresponding to the image features based on the second module. The image processing device fuses the spectral features of the first multispectral image with the spectral features of the second multispectral image based on the fourth module to obtain a fused spectral feature. The image processing device determines first light source distribution information corresponding to the fused spectral feature based on the third module.

[0146] It is understandable that Figure 5A and Figure 5B These are just two examples of the position of the fourth module in the first model. In other embodiments, the fourth module may also be located in other positions to provide information of different frames, which is not limited here.

[0147] Furthermore, the input to the first model can include at least one multispectral image, i.e., one, two, or more multispectral images, with no specific limitation herein. Of course, the greater the number of multispectral images, the smoother the predicted light source distribution information, thereby reducing timing jumps or color instability.

[0148] Step 304: The image processing device adjusts the first image according to the first light source distribution information to obtain a second image.

[0149] After determining the first light source distribution information, the image processing device may adjust the first image according to the first light source distribution information to obtain the second image.

[0150] Among them, the first image can also be called the display image or the image to be displayed or the original image. The purpose of this step is to adjust the display image according to the light source distribution information determined by the multispectral image, so that the balance effect of the light source transition zone is more natural, and the accuracy of the light source decoupling is improved, and it is less susceptible to the influence of the same color and different spectrum and the scene is incorrectly decolorized.

[0151] In addition, the adjustment in the embodiment of the present application can be white balance adjustment or color adjustment, etc., which is not limited here.

[0152] In the embodiment of the present application, there are multiple ways for the image processing device to adjust the first image according to the first light source distribution information, which are described below respectively.

[0153] In the first case, fusion occurs first and then color adaptation.

[0154] In this case, the image processing device first determines the second light source distribution information corresponding to the first image based on the first light source distribution information, and then performs local light source compensation on the first image based on the second light source distribution information to obtain the second image.

[0155] Alternatively, it can be understood that the spatial distribution information of the multispectral image is first aligned point by point with the first image, and then the local light source compensation is performed on the first image using the aligned spatial distribution information to obtain the second image.

[0156] For example, taking the first multispectral image as an input of the first model, the image processing system can be as follows: Figure 6As shown. The image processing system includes a multispectral sensor, a first model, a fusion module and a color adaptation module. Specifically, the first multispectral image collected by the multispectral sensor is subjected to the first model to obtain first light source distribution information (for example, the light source value distribution of the first multispectral image pixel by pixel). The first light source distribution information and the first image are subjected to the fusion module to obtain second light source distribution information. Alternatively, it can be understood that the fusion module is used to determine the second light source distribution information of the first image based on the first light source distribution information. The color adaptation module performs local light source compensation on the first image based on the second light source distribution information to obtain the second image. It can be understood that other modules can be added between the modules for optimization. For example, there can be other operations after the color adaptation module, which are not limited here.

[0157] In the first case, the first light source distribution information of the first multispectral image is mapped onto the first image to obtain the second light source distribution information of the first image. Light source prediction does not require pixel-by-pixel alignment between the multispectral device and the imaging device, and light source learning accuracy is not affected by spatial misalignment. This reduces hardware requirements and power consumption.

[0158] In the second case, color adaptation occurs first and then fusion.

[0159] In this case, the image processing device first performs local light source compensation on the first multispectral image according to the first light source distribution information to obtain a third image, and then adjusts the first image according to the third image to obtain a second image.

[0160] Alternatively, the first multispectral image is locally adjusted based on its spatial distribution information, and then the locally adjusted multispectral image is mapped onto the first image to obtain the second image. Furthermore, the first model can predict the color style mapping table through inference, so that the correct color distribution after local white balancing of the multispectral image can be mapped onto the image to be displayed.

[0161] For example, taking the first model input as the first multispectral image and the second multispectral image, the image processing system can be as follows: Figure 7 As shown. The image processing system includes a multispectral sensor, a first model, a fusion module, and a color adaptation module. Specifically, the first multispectral image and the second multispectral image collected by the multispectral sensor are subjected to the first model to obtain first light source distribution information (for example, the light source value distribution of each pixel of the first multispectral image). The first light source distribution information is used to perform local adjustments to the first multispectral image through the color adaptation model to obtain a third image. The color adaptation module adjusts the first image according to the third image to obtain a second image. It is understandable that other modules can be added between the modules for optimization. For example, there can be other operations after the color adaptation module, which are not limited here.

[0162] It can be seen that the difference between the second case and the first case is that the first case is fusion first and then chromatic adaptation, while the second case is chromatic adaptation first and then fusion.

[0163] It should be noted that the input in the first case may also include the first multispectral image and the second multispectral image. The input in the second case may also be the first multispectral image, i.e., not including the second multispectral image. The light source distribution information in the first and second cases may be pixel-by-pixel light source value distribution, etc., and the specific details are not limited here.

[0164] In the second case, when local adjustment is performed for local white balance correction, the stability of the adjusted temporal partitions is significantly improved, avoiding local color inversion caused by non-smooth predicted partition weights. Fusion network color mapping further filters out extreme illuminant prediction errors, resulting in more robust color migration. Compared to the first case, in the second case, image alignment allows the adjusted second image to represent the illuminant distribution of the third image, improving the imaging quality of the second image. This significantly reduces local illuminant missegmentation caused by metamerism and avoids erroneous color loss.

[0165] It is understandable that the above two situations are just examples. In other embodiments, the light source distribution information obtained by multispectral images can be used to adjust the displayed image in other ways, which are not limited here.

[0166] It should be noted that Figure 3 There may be no time sequence relationship between the steps in the embodiment shown. For example, step 301 may be before step 302 or after step 302.

[0167] In an embodiment of the present application, the first multispectral image collected by the multispectral sensor and the first model are used to determine the partial information of the first light source end-to-end, and the first image is adjusted according to the first light source distribution information. The first light source distribution information is determined end-to-end by the multispectral image and the first model. On the one hand, the size of the first multispectral image is h1*w1*c1, and the size of the first light source distribution information is h2*w2*c2, which not only makes full use of the multi-channel multispectral image, but also improves the accuracy of light source decoupling, and is less susceptible to the influence of the same color and different spectrum to incorrectly decolorize the scene. On the other hand, the end-to-end optimization method of whole-image input and whole-image output does not rely on pre-segmentation and color temperature preset. Word cash box reasoning can achieve continuous prediction of light source distribution, and the upper limit of end-to-end optimization accuracy is high, the balance effect of the light source transition zone is more natural, the reasoning process is simple, and the deployment cost is low. On the other hand, after the parameter conversion of the fusion module, the main imaging path is guided to perform local color restoration, which can improve the accuracy of white balance.

[0168] The above describes the method flow provided by the embodiment of the present application. The following describes the device structure for executing the method flow.

[0169] See Figure 8 , a schematic diagram of the architecture of another image processing system provided in an embodiment of the present application, the image processing system can be used to perform the aforementioned Figures 3 to 7 The method steps shown. The image processing system includes: a first sensor 801, a multispectral sensor 802 and a processing unit 803;

[0170] The first sensor 801 is used to capture a first image;

[0171] The multispectral sensor 802 is used to collect multispectral images;

[0172] The processing unit 803 is configured to determine first light source distribution information according to the first model, where the first light source distribution information is continuous light source distribution information of the first multispectral image in the spatial domain and the frequency domain.

[0173] The processing unit 803 is further configured to adjust the first image according to the first light source distribution information to obtain a second image.

[0174] In one possible implementation, processing unit 803 is specifically configured to determine second light source distribution information corresponding to the first image based on the first light source distribution information. Processing unit 803 is specifically configured to perform local light source compensation on the first image based on the second light source distribution information to obtain the second image. The second light source distribution information is light source distribution information that is obtained by mapping the first light source distribution information onto continuous light source distribution information in the spatial and frequency domains of the first image.

[0175] In one possible implementation, the processing unit 803 is specifically configured to perform local light source compensation on the first multispectral image according to the first light source distribution information to obtain a third image. The processing unit 803 is specifically configured to adjust the first image according to the third image to obtain a second image.

[0176] In one possible implementation, multispectral sensor 802 is configured to capture a second multispectral image, where the first and second multispectral images are captured at different times. Processing unit 803 is specifically configured to determine first light source distribution information based on the first and second multispectral images.

[0177] In a possible implementation, the processing unit 803 is specifically configured to input the first multispectral image into the first model to obtain first light source distribution information.

[0178] In a possible implementation, the processing unit 803 is specifically configured to input the first multispectral image and the first image into the first model to obtain the first light source distribution information.

[0179] In one possible implementation, the first model includes a first module, a second module, and a third module. Processing unit 803 is specifically configured to determine image features corresponding to the first multispectral image based on the first module; processing unit 803 is specifically configured to determine spectral features corresponding to the image features based on the second module; and processing unit 803 is specifically configured to determine first light source distribution information corresponding to the spectral features based on the third module.

[0180] In one possible embodiment, the first model also includes a fourth module; a processing unit 803 is specifically used to fuse the image features of the first multispectral image with the image features of the second multispectral image according to the fourth module to obtain fused image features; and the processing unit 803 is specifically used to determine the spectral features based on the second module and the fused image features.

[0181] In a possible implementation, the processing unit 803 is specifically configured to determine the first light source distribution information end-to-end according to the first model.

[0182] In a possible implementation, the first sensor includes a multispectral sensor and / or a non-multispectral sensor.

[0183] In a possible implementation, the first light source distribution information includes a plurality of light source values, where the plurality of light source values ​​correspond to a plurality of pixel values ​​in the first multispectral image.

[0184] In one possible embodiment, the size of the first multispectral image is h1*w1*c1, and the size of the first light source distribution information is h2*w2*c2, where h1 represents the height of the first multispectral image, w1 represents the width of the first multispectral image, c1 represents the number of channels of the first multispectral image, h2 represents the height of the first light source distribution information, w2 represents the width of the first light source distribution information, c2 represents the number of channels of the first light source distribution information, h1 is greater than 1, w1 is greater than 1, c1 is greater than 4, h2 is greater than 1, w2 is greater than 1, and c2 is greater than 1.

[0185] The operations performed by each unit in this embodiment are the same as those described above. Figures 1 to 7 The description in the illustrated embodiment is similar and will not be repeated here.

[0186] like Figure 9 FIG. 1 is a schematic diagram of the hardware structure of an image processing system or electronic device 90 provided in an embodiment of the present application. The image processing system or electronic device 90 can be used to implement the aforementioned Figures 3 to 7 The steps of the method.

[0187] Figure 9The image processing system or electronic device 90 shown may include a processor 901 , a memory 902 , a communication interface 903 , and a bus 904 . The processor 901 , the memory 902 , and the communication interface 903 may be connected via a bus 904 .

[0188] The processor 901 is the control center of the image processing system or electronic device 90 and can be a general-purpose central processing unit (CPU) or other general-purpose processor. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adaptively configured according to the actual application scenario.

[0189] As an example, the processor 901 may include one or more CPUs, and may also include other processors, such as Figure 9 The CPU, NPU or GPU shown in .

[0190] The memory 902 may be a non-transitory memory, which may be a volatile memory or a non-volatile memory, or may include both volatile memory and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other types of static storage devices that can store static information and instructions. The volatile memory may be random access memory (RAM). By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct RAM (DR RAM). Memory 1405 may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices. It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0191] In one possible implementation, the memory 902 may exist independently of the processor 901. The memory 902 may be connected to the processor 901 via a bus 904 and used to store data, instructions, or program codes. When the processor 901 calls and executes the instructions or program codes stored in the memory 902, the method provided in the embodiment of the present application can be implemented, for example, Figures 3 to 7 The method shown.

[0192] In another possible implementation, the memory 902 may also be integrated with the processor 901 .

[0193] Communication interface 903 is used to connect the image processing system or electronic device 90 to other devices via a communication network. The communication network can be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. Communication interface 903 can include a receiving unit for receiving data and a transmitting unit for sending data.

[0194] The bus 904 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0195] It should be pointed out that Figure 9 The structure shown in the figure does not constitute a limitation on the image processing system or the electronic device 90, except Figure 9 In addition to the components shown, the image processing system or electronic device 90 may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0196] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a number of instructions for causing a device (which can be a personal computer, server, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0197] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0198] In an embodiment of the present application, a computer-readable storage medium is also provided. The computer-readable storage medium stores a program for training a model or performing an inference task. When the program is run on a computer, the computer performs the above-mentioned Figures 3 to 7 All or part of the steps in the method described in the embodiment shown in FIG.

[0199] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the above-mentioned processor or the functions of the processor. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more embodiments in the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the method steps of any one or more embodiments in the above-mentioned embodiments according to the program code stored in the external memory. For example, the chip can be specifically an ISP, see the above-mentioned Figure 1 The digital signal processor 132 is shown in FIG.

[0200] A computer program product is also provided in the embodiment of the present application, and the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0201] The data comparison device provided in the embodiment of the present application can be a chip, which includes: a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip in the server to execute the above Figures 3 to 7 The method described in the embodiment shown. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), etc.

[0202] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0203] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0204] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0205] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0206] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a server, or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0207] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can be changed in the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when implementing in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some ports, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application.

Claims

1. An image processing method, characterized in that: Applied to an image processing device, the image processing device includes a multispectral sensor and a first sensor, and the method includes: Acquire a first multispectral image through the multispectral sensor, and acquire a first image through the first sensor; determining first light source distribution information according to the first model, where the first light source distribution information is continuous light source distribution information of the first multispectral image in the spatial domain and the frequency domain; The first image is adjusted according to the first light source distribution information to obtain a second image.

2. The method according to claim 1, characterized in that The adjusting the first image according to the first light source distribution information to obtain a second image includes: determining second light source distribution information corresponding to the first image according to the first light source distribution information, where the second light source distribution information is light source distribution information obtained by mapping the first light source distribution information to continuous light source distribution information of the first image in the spatial domain and the frequency domain; Local light source compensation is performed on the first image according to the second light source distribution information to obtain the second image.

3. The method according to claim 1, characterized in that The adjusting the first image according to the first light source distribution information to obtain a second image includes: Performing local light source compensation on the first multispectral image according to the first light source distribution information to obtain a third image; The first image is adjusted according to the third image to obtain the second image.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Acquire a second multispectral image by the multispectral sensor, wherein the first multispectral image and the second multispectral image are multispectral images acquired at different times; The determining the first light source distribution information according to the first model includes: The first light source distribution information is determined according to the first model, the first multispectral image, and the second multispectral image.

5. The method according to any one of claims 1 to 4, characterized in that The determining the first light source distribution information according to the first model includes: The first multispectral image is input into the first model to obtain the first light source distribution information.

6. The method according to any one of claims 1 to 4, characterized in that The determining the first light source distribution information according to the first model includes: The first multispectral image and the first image are input into the first model to obtain the first light source distribution information.

7. The method according to any one of claims 1 to 6, characterized in that The first model includes a first module, a second module and a third module; The determining the first light source distribution information according to the first model includes: determining, according to the first module, image features corresponding to the first multispectral image; determining, according to the second module, a frequency spectrum feature corresponding to the image feature; The first light source distribution information corresponding to the spectrum feature is determined according to the third module.

8. The method according to claim 7, characterized in that The first model also includes a fourth module; The determining, according to the second module, the spectral feature corresponding to the image feature includes: fusing the image features of the first multispectral image and the image features of the second multispectral image according to the fourth module to obtain fused image features; The spectrum feature is determined according to the second module and the fused image feature.

9. The method according to any one of claims 1 to 8, characterized in that The determining the first light source distribution information according to the first model includes: The first light source distribution information is determined end-to-end according to the first model.

10. The method according to any one of claims 1 to 9, characterized in that The first sensor includes a multispectral sensor and / or a non-multispectral sensor.

11. The method according to any one of claims 1 to 10, characterized in that The first light source distribution information includes a plurality of light source values, and the plurality of light source values ​​correspond to a plurality of pixel values ​​in the first multispectral image.

12. The method according to any one of claims 1 to 11, characterized in that The size of the first multispectral image is h1*w1*c1, and the size of the first light source distribution information is h2*w2*c2, wherein h1 represents the height of the first multispectral image, w1 represents the width of the first multispectral image, c1 represents the number of channels of the first multispectral image, h2 represents the height of the first light source distribution information, w2 represents the width of the first light source distribution information, and c2 represents the number of channels of the first light source distribution information. h1 is greater than 1, w1 is greater than 1, c1 is greater than 4, h2 is greater than 1, w2 is greater than 1, and c2 is greater than 1.

13. An image processing device, characterized in that include: a first sensor, a multispectral sensor, and a processing unit; The first sensor is used to capture a first image; The multispectral sensor is used to collect multispectral images; The processing unit is used to implement the method according to any one of claims 1 to 12.

14. A chip, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method according to any one of claims 1 to 12 are implemented.

15. An image processing device, characterized in that: The method comprises a first sensor, a multispectral sensor, a processor and a memory, wherein the memory stores a program. When the program instructions stored in the memory are executed by the processor, the first sensor and the multispectral sensor implement the method according to any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that The invention comprises a program which, when executed by a processor of an image processing device, causes the image processing device to perform the method according to any one of claims 1 to 12.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a computer, the computer is caused to implement the method according to any one of claims 1 to 12.

Images