An image processing method and apparatus

Abstract

This application provides an image processing method and apparatus, relating to the field of image processing, which can be used to ensure the temporal stability of light source partitioning in image frame sequences in time-related scenes, thereby performing image partitioning processing and improving user experience. The method includes: obtaining a first fused spectral feature of the first multispectral signal based on the spectral features of a first multispectral signal and a reference spectral feature of a second multispectral signal; obtaining a first fused light source distribution feature of the first imaging image based on the first fused spectral feature of the first multispectral signal, the light source distribution feature of a first imaging image, and a reference light source distribution feature of the second imaging image; obtaining light source partitioning information of the first imaging image based on the first fused light source distribution feature of the first imaging image; and performing partitioning processing on the first imaging image based on the light source partitioning information. Wherein, the spectral features are single-point features without spatial resolution.

Description

[0001] This application claims priority to Chinese Patent Application No. 202511539324.7, filed on October 24, 2025, entitled "An Image Processing Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and more particularly to an image processing method and apparatus. Background Technology

[0003] White balance (WB) is a core module of the image signal processor (ISP). It refers to restoring white objects to white color under any light source by adding corresponding complementary colors. Traditional WB methods can effectively compensate for overall color cast in scenes with a single light source. However, in real-world scenes with multiple light sources (e.g., shooting outdoors through a window indoors, self-illuminating billboards and displays), using traditional WB for global white balance processing can easily result in outdoor areas appearing bluish and self-illuminating areas appearing bluish in the image.

[0004] To restore the true colors of objects in multi-light source scenes, zonal white balance processing was developed. In zonal white balance technology, different light source regions in the image are first identified, and then different primary and secondary white points are applied to define the white of different light source regions in order to restore the true colors of different light source regions in the image.

[0005] The color viewing experience of consecutive image frame sequences in time-dependent scenarios (such as video scenes, preview scenes, recording scenes, etc.) is also a key focus of the industry. However, if the light source partition prediction of different image frames in an image frame sequence is inaccurate, the temporal stability of the light source partition of the image frame sequence will be insufficient, which will lead to unstable color flickering in the picture and greatly reduce the user's viewing experience.

[0006] Therefore, ensuring the temporal stability of light source partitioning in image frame sequences in time-dependent scenarios for image partitioning processing is of great significance for improving user experience. Summary of the Invention

[0007] This application provides an image processing method and apparatus for ensuring the temporal stability of light source partitioning in an image frame sequence in a time-dependent scene, so as to perform image partitioning processing and improve user experience.

[0008] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, this application provides an image processing method applied to an electronic device, the electronic device including a first sensor and a second sensor for imaging. The method includes: obtaining a first fused spectral feature of the first multispectral signal based on spectral features of a first multispectral signal and reference spectral features of a second multispectral signal; obtaining a first fused light source distribution feature of the first imaging image based on the first fused spectral feature of the first multispectral signal, light source distribution features of a first imaging image, and reference light source distribution features of the second imaging image; obtaining light source partitioning information of the first imaging image based on the first fused light source distribution feature of the first imaging image; and performing partitioning processing on the first imaging image based on the light source partitioning information of the first imaging image. Wherein, the spectral features are single-point features without spatial resolution, and the first multispectral signal and the second multispectral signal are multispectral signals acquired by the first sensor at different times. The first imaging image and the second imaging image are imaging images acquired by the second sensor at different times.

[0009] In the solution provided in this application, compared to color images, multispectral signals have more channels and richer, more complete spectral features. Therefore, the spectral features of multispectral signals can more accurately distinguish the types of light sources in a scene. Thus, correcting the light source distribution features extracted from the imaging image based on these spectral features can improve the accuracy of the corrected light source distribution features.

[0010] Furthermore, if correction is based entirely on the light source distribution characteristics obtained from the imaging image, temporal local flicker is likely to occur as the input image changes over time. However, by relying on spectral features, which do not have spatial resolution, to perform global correction on the obtained light source distribution characteristics, the solution provided in this application can improve the robustness to changes in the acquired imaging image, reduce local flicker of light source partitioning information, and thus reduce local flicker in the imaging image obtained after partitioning processing.

[0011] On the other hand, the fused spectral features of the multispectral signals or the fused light source distribution features of the imaging images, obtained based on the spectral features of the multispectral signals acquired at different times or the light source distribution features of the imaging images, can reduce the inter-frame differences of consecutive image frames and ensure a smooth transition between adjacent frames in time-dependent scenes. Therefore, the light source partitioning information obtained through this scheme has both the characteristics of smooth transition between consecutive image frames and the characteristics of high light source partitioning accuracy, thus effectively ensuring the temporal stability of the light source partitioning.

[0012] Furthermore, in this application, since the multispectral signal used has no spatial resolution, the first sensor for acquiring the multispectral signal can be a single-point multispectral sensor, or a sensor with spatial resolution, such as a multi-point multispectral sensor or an array multispectral sensor, thereby improving the compatibility of the solution.

[0013] Furthermore, when the first sensor is a single-point multispectral sensor, the hardware cost of the solution is also reduced.

[0014] In one possible implementation, obtaining the first fused spectral feature of the first multispectral signal based on the spectral features of the first multispectral signal and the reference spectral features of the second multispectral signal includes: fusing the spectral features of the first multispectral signal and the reference spectral features of the second multispectral signal to obtain the first fused spectral feature of the first multispectral signal.

[0015] In one possible implementation, obtaining the light source partitioning information of the first imaging image based on the first fused light source distribution features of the first imaging image includes: obtaining the light source partitioning information based on the first fused light source distribution features of the first imaging image and the first imaging image.

[0016] In another possible implementation, obtaining light source partitioning information based on the first fused light source distribution features of the first imaging image and the first imaging image includes: interpolating and mapping the first fused light source distribution features of the first imaging image to obtain the light source partitioning information. Here, interpolation is an adjustment of the size of the first imaging image to obtain the desired size. Mapping is a mathematical transformation that establishes the coordinate correspondence between pixels in two images.

[0017] In another possible implementation, the method further includes generating a guide map of the first imaging image, the guide map indicating the distribution of color temperature and / or brightness. Correspondingly, obtaining the light source partitioning information of the first imaging image based on the first fused light source distribution features of the first imaging image includes obtaining the light source partitioning information based on the first fused light source distribution features and the guide map of the first imaging image. Through the solution provided in this application, light source partitioning information can be obtained based on the first fused light source distribution features of the first imaging image and the first imaging image or its guide map. This light source partitioning information includes the information of the first fused light source distribution features of the first imaging image and the guide map generated from the first imaging image, making the light source partitioning information accurate and ensuring the temporal stability of the light source partitions. Furthermore, pixel-level light source partitioning information can be obtained based on the first fused light source distribution features and the first imaging image or its guide map, making the light source partitioning information more accurate and further ensuring the temporal stability of the light source partitions.

[0018] In another possible implementation, the above-mentioned method of obtaining light source partitioning information based on the first fused light source distribution features and guide map of the first imaging image includes: performing interpolation mapping based on the first fused light source distribution features and guide map of the first imaging image to obtain light source partitioning information.

[0019] In another possible implementation, the above-mentioned interpolation mapping based on the first fused light source distribution features and the guide map of the first imaging image to obtain light source partitioning information includes: interpolating the first fused light source distribution features of the first imaging image based on the guide map to obtain light source partitioning information.

[0020] In another possible implementation, obtaining the first fused light source distribution feature of the first imaging image based on the first fused spectral feature of the first multispectral signal, the light source distribution feature of the first imaging image, and the light source distribution feature of the second imaging image includes: correcting the light source distribution feature of the first imaging image using the first fused spectral feature of the first multispectral signal; and obtaining the first fused light source distribution feature of the first imaging image based on the corrected light source distribution feature of the first imaging image and the reference light source distribution feature of the second imaging image.

[0021] In another possible implementation, the above-mentioned first fused light source distribution feature of the first imaging image obtained based on the light source distribution features of the corrected first imaging image and the reference light source distribution features of the second imaging image can be implemented as follows: the light source distribution features of the corrected first imaging image and the reference light source distribution features of the second imaging image are fused to obtain the first fused light source distribution feature of the first imaging image.

[0022] In another possible implementation, obtaining the first fused light source distribution feature of the first imaging image based on the first fused spectral feature of the first multispectral signal, the light source distribution feature of the first imaging image, and the reference light source distribution feature of the second imaging image includes: obtaining the second fused light source distribution feature based on the light source distribution feature of the first imaging image and the reference light source distribution feature of the second imaging image; and correcting the second fused light source distribution feature using the first fused spectral feature of the first multispectral signal to obtain the first fused light source distribution feature of the first imaging image.

[0023] In another possible implementation, the above-mentioned second fused light source distribution feature obtained based on the light source distribution features of the first imaging image and the reference light source distribution features of the second imaging image can be implemented as follows: the light source distribution features of the first imaging image and the reference light source distribution features of the second imaging image are fused to obtain the second fused light source distribution feature.

[0024] In another possible implementation, the above feature fusion includes: feature splicing fusion, which is used to fuse features by splicing them together.

[0025] In another possible implementation, the above feature fusion includes: element-level feature fusion, which is used to perform mathematical operations on features pixel by pixel.

[0026] In another possible implementation, the above feature fusion includes: attention-based feature fusion, which utilizes the attention mechanism for feature fusion.

[0027] In another possible implementation, the above feature fusion includes: coefficient-weighted feature fusion, which is used to perform weighted feature fusion based on fusion coefficients.

[0028] In another possible implementation, the above feature fusion includes: feature fusion based on gating mechanism, which is used to perform feature fusion using gating mechanism.

[0029] In another possible implementation, the method further includes: acquiring the feature differences of key features between the first and second imaging images. Based on the feature differences, a fusion coefficient is predicted. The fusion coefficient is used to fuse the light source distribution features of the first imaging image with the reference light source distribution features of the second imaging image. Through the scheme provided in this application, the fusion coefficient is predicted based on the feature differences of key features between the imaging images, making the fusion coefficient strongly correlated with the currently processed image content, thereby improving the temporal stability of the fused light source distribution features in different scenarios.

[0030] In another possible implementation, the above-mentioned prediction of fusion coefficients based on feature differences includes: inputting the feature differences into the coefficient prediction model to obtain the fusion coefficients output by the coefficient prediction model.

[0031] In another possible implementation, the aforementioned key feature is a feature with spatial resolution.

[0032] In another possible implementation, partitioning processing includes partition white balance or partition color correction.

[0033] In another possible implementation, the reference spectral features of the second multispectral signal include: the first fused spectral features of the cached second multispectral signal. Since the first fused spectral features of the second multispectral signal incorporate spectral features from other spectral signals, using the cached first fused spectral features of the second multispectral signal as the reference spectral features reduces inter-frame differences between consecutive image frames, ensuring smooth inter-frame transitions in time-dependent scenes. This, in turn, ensures better inter-frame smoothness and temporal stability of the obtained first fused spectral features of the first multispectral signal.

[0034] In another possible implementation, the reference light source distribution features of the second imaging image include: the first fused light source distribution features of the cached second imaging image. Since the first fused light source distribution features of the second imaging image incorporate the light source distribution features of other imaging images, using the cached first fused light source distribution features of the second imaging image as the reference light source distribution features of the second imaging image reduces inter-frame differences between consecutive image frames, ensuring smooth inter-frame transitions in time-dependent scenes. This, in turn, ensures better inter-frame smoothness and temporal stability of the first fused light source distribution features of the obtained first imaging image.

[0035] In another possible implementation, the method provided in this application further includes: caching the first fused spectral features of the first multispectral signal. This is used when acquiring the first fused spectral features of other multispectral signals to reduce inter-frame differences between consecutive image frames and ensure smooth inter-frame transitions in time-dependent scenes.

[0036] In another possible implementation, the method provided in this application further includes: caching the first fused light source distribution features of the first imaging image. This is used when acquiring the first fused light source distribution features of other imaging images to reduce inter-frame differences between consecutive image frames and ensure smooth inter-frame transitions in time-dependent scenes.

[0037] In another possible implementation, the reference spectral features of the second multispectral signal include: spectral features calculated based on the second multispectral signal, i.e., spectral features extracted from the second multispectral signal.

[0038] In another possible implementation, the reference light source distribution features of the second imaging image include: light source distribution features calculated based on the second imaging image. That is, light source distribution features extracted from the second imaging image.

[0039] In another possible implementation, the first multispectral signal is the current frame's multispectral signal, and the second multispectral signal is the previous frame's multispectral signal. The first image is the current frame's image, and the second image is the previous frame's image. The spectral features of the current frame's multispectral signal are corrected using the spectral features of the previous frame's multispectral signal. A first fused light source distribution feature is obtained by referencing the light source distribution features of the previous frame's image and the current frame's image, reducing inter-frame differences and ensuring smooth transitions between adjacent frames in time-dependent scenes.

[0040] In another possible implementation, the first multispectral signal is the current frame multispectral signal, and the second multispectral signal is one or more previous frame multispectral signals. The first imaging image is the current frame imaging image, and the second imaging image is one or more previous frame imaging images.

[0041] In another possible implementation, when the second multispectral signal is multiple previous frame multispectral signals, the above-mentioned feature fusion of the spectral features of the first multispectral signal and the reference spectral features of the second multispectral signal to obtain the first fused spectral features of the first multispectral signal includes: performing multi-frame feature fusion of the spectral features of the first multispectral signal and the reference spectral features of multiple second multispectral signals to obtain the first fused spectral features of the first multispectral signal.

[0042] Secondly, this application provides an image processing apparatus, which includes a processing module, an acquisition module, and a partitioning module. Wherein: The processing module is used to: obtain a first fused spectral feature of the first multispectral signal based on the spectral features of the first multispectral signal and the reference spectral features of the second multispectral signal. Here, the spectral features are single-point features without spatial resolution; the first multispectral signal and the second multispectral signal are multispectral signals acquired by the first sensor at different times.

[0043] The acquisition module is used to: acquire the first fused light source distribution features of the first imaging image based on the first fused spectral features of the first multispectral signal, the light source distribution features of the first imaging image, and the reference light source distribution features of the second imaging image. The first imaging image and the second imaging image are imaging images acquired by the second sensor at different times.

[0044] The partitioning module is used to: obtain the light source partitioning information of the first imaging image based on the first fused light source distribution characteristics of the first imaging image.

[0045] The processing module is also used to: perform partitioning processing on the first imaging image based on the light source partitioning information of the first imaging image.

[0046] In one possible implementation, the processing module is specifically used to: fuse the spectral features of the first multispectral signal with the reference spectral features of the second multispectral signal to obtain the first fused spectral features of the first multispectral signal.

[0047] In one possible implementation, the partitioning module is specifically used to: obtain light source partitioning information based on the first fused light source distribution features of the first imaging image and the first imaging image.

[0048] In another possible implementation, the partitioning module is specifically used to: interpolate and map the first fused light source distribution features of the first imaging image based on the first imaging image to obtain light source partitioning information.

[0049] In another possible implementation, the device further includes a generation module. This generation module is used to generate a guide map of the first imaging image, the guide map indicating the distribution of color temperature and / or brightness. Correspondingly, the partitioning module is specifically used to obtain light source partitioning information based on the first fused light source distribution features of the first imaging image and the guide map.

[0050] In another possible implementation, the partitioning module is specifically used to: interpolate and map the distribution features of the first fused light source in the first imaging image based on the guide map to obtain the light source partitioning information.

[0051] In another possible implementation, the partitioning module is specifically used to: correct the light source distribution characteristics of the first imaging image using the first fused spectral features of the first multispectral signal; and obtain the first fused light source distribution characteristics of the first imaging image based on the corrected light source distribution characteristics of the first imaging image and the reference light source distribution characteristics of the second imaging image.

[0052] In another possible implementation, the partitioning module is specifically used to: fuse the light source distribution features of the corrected first imaging image with the reference light source distribution features of the second imaging image to obtain the first fused light source distribution features of the first imaging image.

[0053] In another possible implementation, the partitioning module is specifically used to: obtain a second fused light source distribution feature based on the light source distribution features of the first imaging image and the reference light source distribution features of the second imaging image; and correct the second fused light source distribution feature using the first fused spectral features of the first multispectral signal to obtain the first fused light source distribution feature of the first imaging image.

[0054] In another possible implementation, the partitioning module is specifically used to: fuse the light source distribution features of the first imaging image with the reference light source distribution features of the second imaging image to obtain the second fused light source distribution features.

[0055] In another possible implementation, the above feature fusion includes: feature splicing fusion, which is used to fuse features by splicing them together.

[0056] In another possible implementation, the above feature fusion includes: element-level feature fusion, which is used to perform mathematical operations on features pixel by pixel.

[0057] In another possible implementation, the above feature fusion includes: attention-based feature fusion, which utilizes the attention mechanism for feature fusion.

[0058] In another possible implementation, the above feature fusion includes: coefficient-weighted feature fusion, which is used to perform weighted feature fusion based on fusion coefficients.

[0059] In another possible implementation, the above feature fusion includes: feature fusion based on gating mechanism, which is used to perform feature fusion using gating mechanism.

[0060] In another possible implementation, the device further includes a feature extraction module and a prediction module. The feature extraction module is used to obtain the feature differences between key features of the first imaging image and the second imaging image. The prediction module is used to predict a fusion coefficient based on the feature differences. The fusion coefficient is used to fuse the light source distribution features of the first imaging image with the reference light source distribution features of the second imaging image.

[0061] In another possible implementation, the key feature is a feature with spatial resolution.

[0062] In another possible implementation, partitioning processing includes partition white balance or partition color correction.

[0063] In another possible implementation, the reference spectral features of the second multispectral signal include: the first fused spectral features of the cached second multispectral signal. The reference light source distribution features of the second imaging image include: the first fused light source distribution features of the cached second imaging image.

[0064] In another possible implementation, the reference spectral features of the second multispectral signal include: spectral features calculated based on the second multispectral signal, i.e., spectral features extracted from the second multispectral signal. The reference light source distribution features of the second imaging image include: light source distribution features calculated based on the second imaging image, i.e., light source distribution features extracted from the second imaging image.

[0065] In another possible implementation, the first fused spectral features of the first multispectral signal are cached. The first fused light source distribution features of the first imaging image are also cached.

[0066] In another possible implementation, the first multispectral signal is the current frame multispectral signal, and the second multispectral signal is the previous frame multispectral signal. The first imaging image is the current frame imaging image, and the second imaging image is the previous frame imaging image.

[0067] In another possible implementation, the first multispectral signal is the current frame multispectral signal, and the second multispectral signal is one or more previous frame multispectral signals. The first imaging image is the current frame imaging image, and the second imaging image is one or more previous frame imaging images.

[0068] In another possible implementation, when the second multispectral signal is multiple previous frame multispectral signals, the processing module is specifically used to: fuse the spectral features of the first multispectral signal with the reference spectral features of multiple second multispectral signals to obtain the first fused spectral features of the first multispectral signal.

[0069] Thirdly, this application provides an electronic device including a memory and at least one processor, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions, it causes the electronic device to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0070] Fourthly, this application provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the method as described in the first aspect or any possible implementation.

[0071] Fifthly, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform a method as described in the first aspect or any possible implementation.

[0072] Sixthly, a chip system is provided, which includes a processor and may also include a memory for implementing the functions described above. The chip system may be composed of chips or may include chips and other discrete devices.

[0073] It should be noted that the technical effects of any of the design methods in aspects two through six can be found in the technical effects of different implementation methods in aspect one, and will not be repeated here.

[0074] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0075] Figure 1This is a flowchart illustrating an image fusion method based on a preset threshold. Figure 2 This is a flowchart illustrating an image fusion method based on historical motion information. Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application; Figure 7 A schematic flowchart of an image processing method provided in an embodiment of this application; Figure 8 A schematic flowchart illustrating a correction process provided in an embodiment of this application; Figure 9 A flowchart illustrating another correction process provided in an embodiment of this application; Figure 10 This is a schematic diagram of a process for obtaining light source partitioning information provided in an embodiment of this application; Figure 11 A flowchart illustrating a process for determining fusion coefficients, provided as an embodiment of this application; Figure 12 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application; Figure 13 This is a schematic diagram of another image processing apparatus provided in an embodiment of this application. Detailed Implementation

[0076] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0077] In the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0078] To facilitate understanding, the terms or related technologies involved in the embodiments of this application will be explained first.

[0079] A sensor (image sensor) is a semiconductor device that converts optical signals into electronic signals. Examples of sensors include color sensors, multispectral sensors, and others.

[0080] Multispectral refers to a spectral detection technology that can simultaneously acquire multiple optical spectral bands (usually more than 3) and extend beyond visible light into infrared and ultraviolet light. Multispectral signals refer to electromagnetic wave information collected simultaneously on the same target or scene at multiple (usually 3 to 10) discrete, non-continuous specific wavelength bands.

[0081] Image spatial resolution refers to the smallest detail or size that an image can distinguish; it measures the sharpness or fineness of an image in terms of spatial detail. Spectral resolution is a measure of an imaging system's ability to distinguish or differentiate adjacent wavelengths in the electromagnetic spectrum, that is, the number of different color bands it can distinguish. If an image is represented using H×W×C, where H×W represents the image's spatial resolution. For example, if H×W is 1×1, it means the image has no spatial resolution.

[0082] Spectral resolution is a measure of an imaging system's ability to distinguish or differentiate between adjacent wavelengths in the electromagnetic spectrum; in other words, it refers to the number of different color bands that can be distinguished. For example, if an image is represented by H×W×C, then C represents the image's spectral resolution. A 3-channel color image has a spectral resolution of 3, while a multispectral image typically has a spectral resolution greater than 3.

[0083] Light source zoning refers to the process of identifying different light source regions in an image. For example, different light source regions receive different lighting conditions. Lighting conditions can include information such as the direction, color (color temperature), intensity, and quantity of the light source.

[0084] Currently, in time-related scenarios (e.g., video, recordings, live images, previews), it is necessary to identify and partition each image frame in the obtained continuous image frame sequence, such as identifying foreground and background regions. Numerous image partitioning schemes have been proposed in the industry to ensure temporal stability.

[0085] One implementation scheme is an image fusion method based on a preset threshold. Figure 1 This illustrates the process of image fusion based on a preset threshold. For example... Figure 1 As shown, the method includes the following steps: S101. Obtain the current frame image from the video file.

[0086] S102. Perform image segmentation on the current frame image to obtain the segmented image of the current frame.

[0087] S103. Obtain the segmented image of the previous frame and determine whether the change between the segmented image of the current frame and the segmented image of the previous frame exceeds the first preset threshold.

[0088] If the change between the current frame segmented image and the previous frame segmented image exceeds the first preset threshold, then execute S104; if the change between the current frame segmented image and the previous frame segmented image does not exceed the first preset threshold, then execute S105.

[0089] S104. Optimize the segmented image of the current frame and output the optimized image of the current frame.

[0090] This optimization process includes Gaussian blurring, sharpening, and other enhancements.

[0091] S105. Add the current frame segmented image to the video segmentation queue and fuse it with the previous frame segmented image to obtain the fused image of the current frame.

[0092] In the above scheme, a preset threshold guides the fusion of the current frame image and the previous frame image, fusing the segmentation results of the current frame image and the previous frame image into a fused image of the current frame. Essentially, it corrects the segmentation result of the current frame image by referring to the segmentation result of the previous frame image, increasing the continuity of the segmentation results between two consecutive frames, thereby achieving temporal stability of the segmentation results of consecutive images in the video. However, in the implementation of this scheme, the preset threshold cannot guarantee that the image fusion operation will be satisfactory in all scenarios. If the preset threshold is not suitable for the actual scenario, it will lead to incorrect judgment of the difference between the current frame image and the previous frame image, resulting in the temporal stability of the fused current frame image not meeting the expected effect. Therefore, this scheme cannot guarantee the temporal stability of the partitioning results in time-dependent scenarios.

[0093] Another approach is to use image fusion based on historical motion information. Figure 2 This illustrates the process of image fusion based on historical motion information. For example... Figure 2 As shown, the method includes the following steps: S201. Obtain the current frame image from the video file.

[0094] S202. Perform image segmentation on the current frame image to obtain the mask image of the current frame image.

[0095] S203. Determine the historical motion information of the current frame image based on the current frame image and the previous frame image, and obtain the mask image corresponding to the current frame image based on the mask image of the previous frame image and the historical motion information of the current frame image.

[0096] S204. Calculate the fusion weight based on the historical motion information of the current frame image, and perform weighted fusion on the mask image of the current frame image and the mask image corresponding to the current frame image to obtain the fused mask image of the current frame.

[0097] In the above scheme, historical motion information is obtained through optical flow algorithm. Based on this historical motion information, a mask image and fusion weights corresponding to the current frame image are obtained. Then, the mask image corresponding to the current frame image is weighted and fused with the mask image of the current frame image. The segmentation results of the current frame image and the previous frame image are fused into the fused image of the current frame. Compared with the previous scheme, the temporal stability of the inter-frame segmentation results in the video is better. However, if there is a sudden fast movement or a change in the direction of movement in the scene, the optical flow algorithm may not be able to estimate it in a timely and accurate manner, resulting in a discrepancy between the historical motion information and the current actual motion, thus generating an incorrect mask. Furthermore, the incorrect mask image of the current frame will propagate to the mask image of the next frame, leading to the accumulation of errors. Therefore, this scheme cannot guarantee the temporal stability of the partitioning results in time-dependent scenes.

[0098] Based on this, this application proposes an image processing method that deploys a first sensor and a second sensor for imaging in an electronic device. A fused spectral feature is obtained based on the spectral features of the multispectral signals acquired by the first sensor at different times; the light source distribution features of the imaging image acquired by the second sensor at different times are used to obtain the light source distribution features of the current imaging image. Then, using the fused spectral feature and the light source distribution features of the current imaging image, the light source partitioning information of the imaging image is obtained, and the imaging image is further partitioned according to the light source partitioning information. Compared to color images, multispectral signals have more channels and richer, more complete spectral features; therefore, the spectral features of multispectral signals can more accurately distinguish the types of light sources in a scene. Therefore, obtaining the light source partitioning information of the current imaging image based on the spectral features and the light source distribution features extracted from the imaging image is equivalent to correcting the light source distribution features extracted from the imaging image based on the spectral features, which can improve the accuracy of the corrected light source distribution features. Furthermore, if the correction is based entirely on the light source distribution features obtained from the imaging image, temporal local flickering can easily occur as the input image time sequence changes. By relying on the spectral features without spatial resolution to perform global correction on the obtained light source distribution features, the solution provided in this application can improve the robustness to changes in the acquired imaging image, reduce local flicker of light source partitioning information, and thus reduce local flicker of the imaging image obtained after partitioning processing.

[0099] On the other hand, the fused spectral features of the multispectral signals or the fused light source distribution features of the imaging images, obtained based on the spectral features of the multispectral signals acquired at different times or the light source distribution features of the imaging images, can reduce the inter-frame differences of consecutive image frames and ensure a smooth transition between adjacent frames in time-dependent scenes. Therefore, the light source partitioning information obtained through this scheme has both the characteristics of smooth transition between consecutive image frames and the characteristics of high light source partitioning accuracy, thus effectively ensuring the temporal stability of the light source partitioning.

[0100] Furthermore, in this application, since the multispectral signal used has no spatial resolution, the first sensor for acquiring the multispectral signal can be a single-point multispectral sensor, or a sensor with spatial resolution, such as a multi-point multispectral sensor or an array multispectral sensor, thereby improving the compatibility of the solution.

[0101] Furthermore, when the first sensor is a single-point multispectral sensor, the hardware cost of the solution is also reduced.

[0102] The image processing method provided in this application can be applied to... Figure 3 An illustrated electronic device. (e.g.) Figure 3 As shown, the electronic device includes a lens 310, a first sensor 320, a second sensor 330, and a processor 340. The lens 310, the first sensor 320, the second sensor 330, and the processor 340 are communicatively connected.

[0103] The lens 310 is used to converge the light reflected from objects in the scene to the first sensor 320 and / or the second sensor 330.

[0104] The first sensor 320 is used to convert optical signals into electrical signals. The electrical signals are analog signals, which are converted into multispectral signals after analog-to-digital (A / D) conversion.

[0105] The second sensor 330 is used to convert optical signals into electrical signals. The electrical signals are analog signals, which are converted into an image after analog-to-digital (A / D) conversion.

[0106] The electrical signal obtained by the first sensor 320 is converted into a multispectral signal by an analog-to-digital converter (ADC or A / D converter). The A / D converter can be located inside the first sensor 320 or inside the processor 340, and is not limited thereto.

[0107] The electrical signal obtained by the second sensor 330 is converted into raw format (RAW) data by an A / D converter. The A / D converter can be located inside the second sensor 330 or inside the processor 340, and is not limited thereto.

[0108] Processor 340 processes the raw format (RAW) data to obtain an image. For example, the processing includes, but is not limited to: depixelation, automatic exposure, automatic white balance, lens shading removal, gamma correction, color space conversion, dynamic range correction, and image cropping.

[0109] The processor 340 executes the solution of this application, using the spectral characteristics of the multispectral signal acquired by the first sensor 320 and the imaging image acquired by the second sensor 330 to determine the light source partitioning information of the imaging image. Then, it uses the light source partitioning information to perform partitioning processing on the imaging image. The process by which the processor 340 executes the solution of this application is described in the following method embodiments, and will not be repeated here.

[0110] The first sensor 320 can acquire multispectral signals without spatial resolution, where the spectral characteristics are single-point features. Alternatively, the first sensor 320 can acquire multispectral signals with spatial resolution, which are then compressed by the processor 340 into multispectral signals without spatial resolution.

[0111] For example, the first sensor 320 is a multispectral sensor that acquires electrical signals without spatial resolution, or electrical signals with spatial resolution. The second sensor 330 can be a multispectral sensor, a color sensor, or a panchromatic sensor. When the second sensor 330 is a multispectral sensor, it acquires electrical signals with spatial resolution.

[0112] For example, electronic devices include, but are not limited to, mobile phones, cameras, tablets, smart cars, robot vacuum cleaners, or virtual reality (VR) glasses, which are terminal devices that can have imaging capabilities.

[0113] For example, processor 340 may be an image signal processor (ISP).

[0114] In one possible implementation, Figure 3 The architecture of the illustrated electronic device can be as follows Figure 4 As shown. Figure 4 As shown, the processor 340 includes a spectrum analysis module 3401, a feature extraction module 3402, a feature processing module 3403, a correction module 3404, and a partition processing module 3405.

[0115] The spectrum analysis module 3401 is used to perform spectrum analysis on multispectral signals to obtain spectral features.

[0116] The feature extraction module 3402 is used to extract the light source distribution features of the imaging image. For example, the light source distribution features include, but are not limited to, the color temperature, color, and brightness features of the imaging image.

[0117] The feature processing module 3403 is used to obtain a first fused spectral feature based on the spectral features of multispectral signals acquired at different times. The feature processing module 3403 is also used to obtain a first fused light source distribution feature based on the light source distribution features of imaging images acquired at different times.

[0118] The correction module 3404 is used to correct the first fused light source distribution characteristics of the imaging image based on the first fused spectral characteristics to obtain light source partitioning information.

[0119] The partitioning module 3405 is used to perform partitioning processing on the imaging image based on light source partitioning information. For example, partitioning processing includes partitioned white balance and / or partitioned color correction.

[0120] It should be noted that, Figure 3 or Figure 4 The architecture shown is merely an example; in practical applications, it can be more complex than... Figure 3 or Figure 4 The more or fewer components shown can be combined into two or more components, or they can have different component configurations.

[0121] Figure 5 This is a schematic diagram of a computing device provided in this application. This computing device can... Figure 3 or Figure 4 The electronic devices shown in the image. For example... Figure 5 As shown, the computing device 50 may include a processor 510, a bus 520, a memory 530, and a communication interface 540. The processor 510, the memory 530, and the communication interface 540 are connected via the bus 520.

[0122] It should be understood that in this embodiment, the processor 510 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0123] The processor 510 may also be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the program of the present application.

[0124] The communication interface 540 is used to enable communication between the computing device 50 and external devices or components.

[0125] Bus 520 may include a pathway for transferring information between the aforementioned components (such as processor 510 and memory 530). In addition to a data bus, bus 520 may also include a power bus, control bus, and status signal bus. However, for clarity, all buses are labeled as bus 520 in the diagram. Bus 520 may be a peripheral component interconnect express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cachecoherent interconnect for accelerators (CCIX), etc. Bus 520 can be divided into address bus, data bus, control bus, etc.

[0126] As an example, computing device 50 may include multiple processors. A processor may be a multi-core (multi-CPU) processor. Here, "processor" can refer to one or more devices, circuits, and / or computing units used to process data (e.g., computer program instructions).

[0127] It is worth noting that, Figure 5 Taking computing device 50 as an example, which includes one processor 510 and one memory 530, the processor 510 and memory 530 are used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment can be determined according to business needs.

[0128] The memory 530 can be a pool of volatile memory or a pool of non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0129] For example, the electronic device described in this application can be a terminal device. Figure 6 This is a structural schematic diagram of a terminal device provided in this application. Figure 6As shown, the terminal device 60 may include: a processor 610, an external memory interface 620, an internal memory 621, a universal serial bus (USB) interface 630, a power management module 640, an antenna, a wireless communication module 660, an audio module 670, a speaker 670A, a speaker interface 670B, a microphone 670C, a sensor module 680, buttons 690, an indicator 691, a display screen 692, a camera 693, etc. The sensor module 680 may include sensors such as a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, and an ambient light sensor.

[0130] Processor 610 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0131] In this embodiment, the processor 610 is used to execute the image processing method provided in this application.

[0132] The processor 610 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 610 is a cache memory. This memory can store instructions or data that the processor 610 has just used or that are used repeatedly. If the processor 610 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 610, and thus improves the efficiency of the system.

[0133] In some embodiments, the processor 610 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and / or a USB interface, etc.

[0134] The power management module 640 is used to connect to a power source. The power management module 640 can also be connected to the processor 610, internal memory 621, display screen 692, camera 693, and wireless communication module 660, etc. The power management module 640 receives power input and supplies power to the processor 610, internal memory 621, display screen 692, camera 693, and wireless communication module 660, etc. In some embodiments, the power management module 640 may also be located within the processor 610.

[0135] The wireless communication function of the terminal device 60 can be implemented through an antenna and a wireless communication module 660. The wireless communication module 660 can provide solutions for wireless communication applications on the terminal device 60, including wireless local area networks (WLANs) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0136] The wireless communication module 660 may be one or more devices integrating at least one communication processing module. The wireless communication module 660 receives electromagnetic waves via an antenna, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to the processor 610. The wireless communication module 660 can also receive signals to be transmitted from the processor 610, perform frequency modulation and amplification on them, and then convert them into electromagnetic waves for radiation via the antenna. In some embodiments, the antenna of the terminal device 60 is coupled to the wireless communication module 660, enabling the terminal device 60 to communicate with networks and other devices via wireless communication technology.

[0137] Terminal device 60 implements display functions through a GPU, display screen 692, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 192 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 610 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0138] The display screen 692 is used to display text, images, and videos, etc. The display screen 692 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0139] The terminal device 60 can implement shooting functions through an ISP, a camera 693, a video codec, a GPU, a display 692, and an application processor. The ISP is used to process the data fed back by the camera 693. In some embodiments, the ISP can be set in the camera 693. In some embodiments, the ISP can execute the scheme provided in this application to obtain time-stable light source partitions for partitioning the image.

[0140] Camera 693 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the terminal device 60 may include one or N cameras 693, where N is a positive integer greater than 1. This embodiment does not limit the position of the cameras 693 on the terminal device 60.

[0141] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0142] The external storage interface 620 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device 60. The external storage card communicates with the processor 610 through the external storage interface 620 to perform data storage functions. For example, text, images, and video files can be saved on the external storage card.

[0143] Internal memory 621 can be used to store computer executable program code, which includes instructions. Processor 610 executes various functional applications and data processing of terminal device 60 by running the instructions stored in internal memory 621. Internal memory 621 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of terminal device 60 (such as audio data, etc.). Furthermore, internal memory 621 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0144] Terminal device 60 can implement audio functions through audio module 670, speaker 670A, microphone 670C, speaker interface 670B, and application processor, such as music playback and recording. In this application, microphone 670C can be used to receive voice commands issued by the user to terminal device 60. Speaker 670A can be used to provide feedback on the decision commands of terminal device 60 to the user.

[0145] Audio module 670 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 670 can also be used for encoding and decoding audio signals. In some embodiments, audio module 670 may be located in processor 610, or some functional modules of audio module 170 may be located in processor 610. Speaker 670A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Microphone 670C, also called a "microphone" or "microphone," is used to convert sound signals into electrical signals.

[0146] The speaker jack 670B is used to connect wired speakers. The speaker jack 670B can be a USB 630 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0147] Buttons 690 include a power button, volume buttons, etc. Buttons 690 can be mechanical buttons or touch-sensitive buttons. Terminal device 60 can receive button input and generate key signal inputs related to user settings and function control of the terminal device 60.

[0148] Indicator 691 can be an indicator light, which can be used to indicate whether the terminal device 60 is in a powered-on state, a standby state, or a powered-off state. For example, an indicator light that is off indicates that the terminal device 60 is powered off; an indicator light that is green or blue indicates that the terminal device 60 is powered on; and an indicator light that is red indicates that the terminal device 60 is in a standby state.

[0149] It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the terminal device 60. It may have more than Figure 6 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. For example, the terminal device 60 may also include components such as speakers. Figure 6The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing or application-specific integrated circuits.

[0150] The solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0151] On one hand, embodiments of this application provide an image processing method that can be executed by an electronic device, the electronic device including a first sensor and a second sensor for imaging. This electronic device can be... Figure 3 or Figure 4 The illustrated electronic device. Alternatively, the method can be executed by a computing device, which can be... Figure 3 or Figure 4 The processor 340 in the illustrated electronic device, or, Figure 5 The illustrated computing device 50.

[0152] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that in the description of the image processing methods provided in the embodiments of this application, the preceding or subsequent operations are not necessarily performed precisely in sequence; instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these method processes, or one or more steps can be removed from these method processes.

[0153] like Figure 7 As shown, the image processing method provided in this application may include the following steps: S701. Based on the spectral features of the first multispectral signal and the reference spectral features of the second multispectral signal, a first fused spectral feature of the first multispectral signal is obtained. Here, the spectral feature is a single-point feature without spatial resolution.

[0154] Specifically, in S701, the first multispectral signal and the second multispectral signal collected by the first sensor at different times are first acquired.

[0155] The second multispectral signal is a referenced previous frame multispectral signal. There can be one or more second multispectral signals, which is not limited in this application embodiment.

[0156] For example, the first multispectral signal is the multispectral signal of the current frame, and the second multispectral signal is the multispectral signal of the previous frame; or, the first multispectral signal is the multispectral signal of the current frame, and the second multispectral signal is the multispectral signal of the previous frame. Wherein, the previous frame is the Nth frame before the current frame, and N is greater than or equal to 2.

[0157] One possible implementation is that the first sensor can be a single-point multispectral sensor. The first multispectral signal and the second multispectral signal are single-point multispectral signals collected by the single-point multispectral sensor at different times, which are further calculated and converted into spectral features with a spatial resolution of 1×1.

[0158] Another possible implementation is that the first sensor can be a multi-point multispectral sensor or an array multispectral sensor, or other multispectral sensors with spatial resolution, which is not limited in the embodiments of this application. The first multispectral signal and the second multispectral signal are multi-point multispectral signals or two-dimensional image multispectral signals collected by the first sensor at different times, which are further calculated and compressed into spectral features with a spatial resolution of 1×1.

[0159] One possible implementation involves the reference spectral features of the second multispectral signal including: a first fused spectral feature of the cached second multispectral signal. The first fused spectral feature of the second multispectral signal is obtained based on spectral features extracted from the second multispectral signal and spectral features of the third multispectral signal. The second and third multispectral signals are multispectral signals acquired by the first sensor at different times. There can be one or more third multispectral signals.

[0160] The process of determining the light source partition information of each imaging image using the scheme provided in this application can cache the first fused spectral features of the first multispectral signal obtained by executing the S701 process, so that they can be used as reference spectral features of the second multispectral signal in the process of determining the light source partition information of subsequent imaging images.

[0161] Correspondingly, after S701, the first fused spectral features of the first multispectral signal obtained in S701 can be cached.

[0162] Another possible implementation is that the reference spectral features of the second multispectral signal can be calculated based on the second multispectral signal. That is, the reference spectral features of the second multispectral signal are extracted from the second multispectral signal.

[0163] Another possible implementation is that the first multispectral signal is the first frame in the image frame sequence, or the first multispectral signal is a single frame, and the second multispectral signal can be a preset spectral signal. This application does not limit the content of the preset spectral signal; it can be configured according to actual needs.

[0164] Here, spectral features refer to intermediate features calculated based on multispectral signals, which have a spatial resolution of 1x1 (i.e., the height (H) and width (W) of the intermediate feature are 1). In the embodiments of this application, the intermediate features are used to assist in the calculation of light source distribution in the imaging image to obtain light source partitioning information.

[0165] One possible implementation is that the intermediate feature can be calculated using machine learning methods (Support Vector Machine (SVM) or neural network models).

[0166] Another possible implementation is that the intermediate feature can be calculated using inverse spectral analysis. For example, the inverse spectral analysis can be performed using least squares, basis function fitting, or principal component analysis (PCA), and this application does not limit the specific methods used.

[0167] Furthermore, after obtaining the spectral features of the first multispectral signal and the spectral features of the second multispectral signal, a first fused spectral feature is obtained based on the spectral features of the first multispectral signal and the spectral features of the second multispectral signal.

[0168] It should be understood that the first fused spectral feature of a multispectral signal is obtained based on the spectral features of the multispectral signal and the reference spectral features of the preceding frame multispectral signal. The first fused spectral feature of the multispectral signal includes the characteristics of both the spectral features of the multispectral signal and the reference spectral features of the preceding frame multispectral signal. The embodiments of this application do not limit the process for obtaining the first fused spectral feature. The first fused spectral feature can be obtained by feature fusion of the spectral features of the multispectral signal and the reference spectral features of the preceding frame multispectral signal, or other methods may be used; the embodiments of this application do not limit this approach.

[0169] For example, S701 can be specifically implemented as follows: the spectral features of the first multispectral signal and the reference spectral features of the second multispectral signal are fused to obtain the first fused spectral features of the first multispectral signal.

[0170] The result of feature fusion includes the characteristics of the features used in the feature fusion process. The specific implementation of feature fusion can be configured according to actual needs, and this application embodiment does not limit it in this regard.

[0171] In one possible implementation, the feature fusion described in this application can be temporal feature fusion, or temporal feature fusion.

[0172] In another possible implementation, when the second multispectral signal is composed of multiple preceding multispectral signals, the feature fusion described in S701 can be multi-frame feature fusion. That is, when the second multispectral signal is composed of multiple preceding multispectral signals, S701 can specifically be implemented as follows: performing multi-frame feature fusion of the spectral features of the first multispectral signal and the reference spectral features of multiple second multispectral signals to obtain the first fused spectral features of the first multispectral signal.

[0173] The following examples illustrate several implementation methods for feature fusion, but are not intended to limit the implementation process of feature fusion: Method 1: Feature splicing and fusion. Feature splicing and fusion is used to fuse features by splicing them together.

[0174] For example, in S701, the spectral features of the first multispectral signal and the spectral features of the second multispectral signal can be spliced ​​and fused along the channel dimension to obtain a first fused spectral feature. For instance, if the dimensions of the spectral features of the first multispectral signal are (B, C1, H, W) and the dimensions of the spectral features of the second multispectral signal are (B, C2, H, W), then the dimensions of the first fused spectral feature obtained after splicing are (B, C1+C2, H, W). Here, B represents batch, C represents channel, H represents height, and W represents weight.

[0175] Method 2: Element-level feature fusion. Element-level feature fusion is used to perform mathematical operations on features pixel by pixel.

[0176] For example, in S701, the spectral features of the first multispectral signal and the spectral features of the second multispectral signal can be mathematically operated on pixel by pixel to achieve feature fusion and obtain the first fused spectral feature. Exemplarily, the mathematical operation may include addition, multiplication or other mathematical operations, which are not limited in the embodiments of this application.

[0177] Method 3: Feature fusion based on attention mechanism. Feature fusion based on attention mechanism is used to perform feature fusion using attention mechanism.

[0178] For example, attention weights for the spectral features of a first multispectral signal and the spectral features of a second multispectral signal can be predicted using neural network models (such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and residual networks (ResNets)). Based on these attention weights, feature fusion is then performed between the spectral features of the first and second multispectral signals.

[0179] For example, the spatial attention weights of the spectral features of the first multispectral signal and the spectral features of the second multispectral signal can be calculated using a neural network model (e.g., calculating the attention correlation of the spectral features in the width and height dimensions), and the features can be fused according to the spatial attention weights to obtain the first fused spectral features.

[0180] For example, the channel attention weights of the spectral features of the first multispectral signal and the spectral features of the second multispectral signal can be calculated using a neural network model (e.g., calculating the attention correlation of spectral features in the channel dimension), and the features can be fused according to the channel attention weights to obtain the first fused spectral features.

[0181] Of course, the attention weight of spectral features can also be other dimensions besides space or channels, and this application embodiment does not limit this.

[0182] Method 4: Feature fusion based on gating mechanism. Feature fusion based on gating mechanism is used to perform feature fusion using gating mechanism.

[0183] For example, in S701, the fusion weights of the spectral features of the first multispectral signal and the spectral features of the second multispectral signal can be dynamically determined through the learnable gating mechanism in deep learning (such as long short-term memory network (LSTM) or gated recurrent unit (GRU)). Based on the fusion weights, the spectral features of the first multispectral signal and the spectral features of the second multispectral signal are fused by feature weighting to obtain the first fused spectral features.

[0184] Method 5: Coefficient-weighted feature fusion. Coefficient-weighted feature fusion is used to perform weighted feature fusion based on fusion coefficients.

[0185] For example, the coefficient-weighted feature fusion process can satisfy the following expression: F ‘ i =αF i +(1-α)F i-1 .

[0186] Among them, F i For the spectral characteristics of the first multispectral signal, F i-1 F serves as the reference spectral feature for the second multispectral signal. ’ i α represents the first fused spectral feature of the first multispectral signal, and α is the fusion coefficient of the coefficient-weighted feature fusion.

[0187] One possible implementation is that the fusion coefficient of the coefficient-weighted feature fusion is a preset value, which can be statically configured according to actual needs.

[0188] Another possible implementation is that the fusion coefficients of the coefficient-weighted feature fusion are preset values, which can be dynamically configured according to actual needs. For example, the fusion coefficients can be dynamically configured with reference to the content of the multispectral signal. The embodiments of this application do not limit the process of configuring the fusion coefficients.

[0189] Of course, the process of determining the fusion coefficients can also be configured according to actual needs, which will not be described in detail in this embodiment. In practical applications, the specific implementation of feature fusion can be configured according to actual needs, and this embodiment does not limit it.

[0190] For example, the operations in S701 are performed by the spectral analysis module and the feature fusion module configured in the processor.

[0191] Furthermore, after obtaining the first fused spectral feature of the first multispectral signal, the process of S702 is executed to correct the light source distribution features obtained from the imaging path (the imaging image acquired by the second sensor) based on the first fused spectral feature of the first multispectral signal.

[0192] S702. Based on the first fused spectral features of the first multispectral signal, the light source distribution features of the first imaging image, and the reference light source distribution features of the second imaging image, obtain the first fused light source distribution features of the first imaging image.

[0193] Specifically, in S702, the first imaging image and the second imaging image acquired by the second sensor at different times are first obtained.

[0194] In one possible implementation, the first imaging image and the first multispectral signal are obtained from the same scene. For example, the same scene is a scene at a specific moment. The first imaging image is the display image at the moment of the current frame.

[0195] For example, the first imaging image is the current frame imaging image, and the second imaging image is the previous frame imaging image; or, the first imaging image is the current frame imaging image, and the second imaging image is the previous frame imaging image. The second imaging image can be one or more, and this application embodiment does not limit this.

[0196] In one possible implementation, the reference light source distribution features of the second imaging image include: cached first fused light source distribution features of the second imaging image. That is, during the process of determining the light source partitioning information of each imaging image using the scheme provided in this application, the first fused light source distribution features of the first imaging image obtained by executing process S702 are cached so that they can be used as reference light source distribution features of the second imaging image during the process of determining the light source partitioning information of subsequent imaging images.

[0197] It should be understood that the first fused light source distribution feature of an imaging image is obtained using the first fused spectral feature of the multispectral signal corresponding to the imaging image, the light source distribution feature of the imaging image, and the reference light source distribution feature of the previous frame imaging image. The acquisition process is the execution of process S702. Specifically, the imaging image acquired from the same scene corresponds to the multispectral signal.

[0198] Correspondingly, after S702, the first fused light source distribution features of the first imaging image obtained in S702 can be cached.

[0199] Another possible implementation is that the reference light source distribution characteristics of the second imaging image can be calculated based on the second imaging image. That is, the reference light source distribution characteristics of the second imaging image are extracted from the second imaging image.

[0200] Another possible implementation is that the first image is the first frame of the current image frame sequence, or the first image is a single frame image, and the second image can be a preset image. For example, the pixel values ​​of the preset image are preset pixel values. The values ​​of these preset pixel values ​​can be configured according to actual needs. For example, the preset pixel values ​​can be all 0s or all 1s.

[0201] After acquiring the first imaging image and the second imaging image, the light source distribution characteristics of the first imaging image and the reference light source distribution characteristics of the second imaging image are obtained.

[0202] The light source distribution features include feature information characterizing the distribution of light sources in the image acquisition scene. For example, the light source distribution features can take the form of a pixel-by-pixel light source value distribution, K light source values, and / or a weighted distribution map of the corresponding light source values. Here, K is a real number greater than 1, and this embodiment of the application does not limit the definition.

[0203] For example, light source values ​​may include color temperature, luminance, color coordinates (CIE chromaticity values), luminous flux, and / or spectral power distribution (SPD) values, which are not limited in the embodiments of this application.

[0204] For example, the light source distribution features may include multiple light source values, which correspond to multiple pixel values ​​in the first imaging image.

[0205] For example, the size of the light source distribution feature can be h 1× w 1× c 1. Among them, h 1 indicates a high level of light source distribution characteristics. w 1 indicates the width of the light source distribution characteristics. ，c 1 represents the number of channels representing the light source distribution characteristics. h 1 is greater than 1. w 1 is greater than 1. c 1 is greater than or equal to 1.

[0206] The specific process of obtaining the light source distribution characteristics from the imaging image is not limited in the embodiments of this application, and can be configured according to actual needs.

[0207] One possible implementation is to obtain the light source distribution features based on feature extraction methods in traditional image processing. For example, the image can be converted to a color space, its color distribution can be analyzed in a suitable color space (such as XYZ, LAB space, etc.), and the light source distribution features of the image can be calculated using a color temperature estimation formula.

[0208] Another possible implementation is to obtain the light source distribution features based on deep learning feature extraction methods. For example, convolutional neural networks (such as ResNet, LeNet, and AlexNet), recurrent neural networks, or attention-based mechanisms can be used to obtain the light source distribution features of an image.

[0209] Specifically, the process in S702 of obtaining the first fused light source distribution feature of the first imaging image based on the first fused spectral feature of the first multispectral signal, the light source distribution feature of the first imaging image, and the reference light source distribution feature of the second imaging image can be configured according to actual needs, and this application does not limit it. This application provides the following two processes for implementing S702, but they do not constitute a specific limitation: Implementation 1: The light source distribution characteristics of the first imaging image are corrected by using the first fused spectral characteristics of the first multispectral signal. Then, based on the corrected light source distribution characteristics of the first imaging image and the reference light source distribution characteristics of the second imaging image, the first fused light source distribution characteristics of the first imaging image are obtained.

[0210] For example, the light source distribution features of the corrected first imaging image can be fused with the reference light source distribution features of the second imaging image to obtain the first fused light source distribution features of the first imaging image. The feature fusion process can be referred to the description of feature fusion in S701 above, and will not be repeated here.

[0211] In one possible implementation, the feature fusion described in 1 can be implemented as temporal feature fusion, also known as temporal feature fusion.

[0212] In another possible implementation, when the second imaging image consists of multiple previous imaging images, the feature fusion described in step 1 can be implemented as multi-frame feature fusion. Specifically, when the second imaging image consists of multiple previous imaging images, implementation 1 can be implemented as follows: using the first fused spectral features of the first multispectral signal, the light source distribution features of the first imaging image are corrected; then, the corrected light source distribution features of the first imaging image are fused with the reference light source distribution features of multiple second imaging images to obtain the first fused light source distribution features of the first imaging image.

[0213] like Figure 8As shown, based on the first fused spectral features of the first multispectral signal, the light source distribution features of the first imaging image are corrected. Then, the corrected light source distribution features of the first imaging image are fused with the reference light source distribution features of the second imaging image to obtain the first fused light source distribution features of the first imaging image.

[0214] One possible implementation, based on the first fused spectral features of the first multispectral signal, involves correcting the light source distribution features of the first imaging image as follows: mapping the first fused spectral features of the first multispectral signal into multiple modulation terms of dimension 1×1×C. Multiplying and / or adding each modulation term pixel-by-pixel with the light source distribution features of the first imaging image to obtain the corrected light source distribution features of the first imaging image. Then, performing distribution feature recovery on the corrected light source distribution features of the first imaging image to obtain light source distribution features with the same target resolution, which are used as the corrected light source distribution features of the first imaging image. For example, the target resolution can be the resolution of the first imaging image or a preset resolution; this embodiment is not limited thereto.

[0215] For example, if the first fused spectral feature of the first multispectral signal is A, and the dimension of A is 1×1×C, the first fused spectral feature A of the first multispectral signal is mapped to multiple modulation terms with a dimension of 1×1×C through a neural network model. The light source distribution feature of the first imaging image is B, and the dimension of B is H×W×C. For example, if the modulation terms are M and N, then the modulation terms M and N are multiplied and / or added pixel by pixel with the light source distribution feature B in the spatial dimension (H, W). For example, M*B+N, M+B, N*B+M, and / or N+B. The light source distribution feature B' of the corrected first imaging image is obtained, and the dimension of B' is also H×W×C. Then, the light source distribution feature B' of the corrected first imaging image is restored to obtain the light source distribution feature with the same resolution as the target resolution, which is used as the light source distribution feature of the corrected first imaging image.

[0216] For example, the process of fusing the light source distribution features of the corrected first imaging image with the reference light source distribution features of the second imaging image can be a feature fusion based on a fusion coefficient, i.e., the coefficient-weighted feature fusion described above. This fusion coefficient can be statically configured or dynamically predicted; this embodiment of the application does not limit this. For example, the process of dynamically predicting the fusion coefficient can refer to the following... Figure 11 The process is illustrated.

[0217] Implementation 2: Based on the light source distribution characteristics of the first imaging image and the reference light source distribution characteristics of the second imaging image, a second fused light source distribution characteristic is obtained. Using the first fused spectral characteristics of the first multispectral signal, the second fused light source distribution characteristic is corrected to obtain the first fused light source distribution characteristic of the first imaging image.

[0218] For example, the light source distribution features of the first imaging image can be fused with the reference light source distribution features of the second imaging image to obtain the second fused light source distribution features. The feature fusion process can be referred to the description of feature fusion in S701 above, and will not be repeated here.

[0219] In one possible implementation, the feature fusion described in section 2 can be implemented as temporal feature fusion, also known as temporal feature fusion.

[0220] In another possible implementation, when the second imaging image consists of multiple previous imaging images, the feature fusion described in step 2 can be implemented as multi-frame feature fusion. Specifically, when the second imaging image consists of multiple previous imaging images, step 2 can be implemented as follows: The light source distribution features of the first imaging image are fused with the reference light source distribution features of multiple second imaging images to obtain a second fused light source distribution feature. The second fused light source distribution feature is then corrected using the first fused spectral features of the first multispectral signal to obtain the first fused light source distribution feature of the first imaging image.

[0221] like Figure 9 As shown, the light source distribution features of the first imaging image and the reference light source distribution features of the second imaging image are fused to obtain the second fused light source distribution features. The second fused light source distribution features are then corrected using the first fused spectral features of the first multispectral signal to obtain the first fused light source distribution features of the first imaging image.

[0222] For example, the process of fusing the light source distribution features of the first imaging image with the reference light source distribution features of the second imaging image can be a feature fusion based on a fusion coefficient, i.e., the coefficient-weighted feature fusion described above. This fusion coefficient can be statically configured or dynamically predicted; this embodiment of the application does not limit this. For example, the process of dynamically predicting the fusion coefficient can refer to the following... Figure 11 The process is illustrated.

[0223] One possible implementation, using the first fused spectral features of the first multispectral signal to correct the second fused light source distribution features, can be as follows: Feature extraction is performed on the second fused light source distribution features to obtain the second fused light source distribution features; the first fused spectral features of the first multispectral signal are mapped to multiple modulation terms with a dimension of 1×1×C. Each modulation term is multiplied and / or added pixel-by-pixel with the second fused light source distribution features to obtain the corrected second fused light source distribution features. Then, the corrected second fused light source distribution features are used for distribution feature recovery to obtain the first fused light source distribution features of the first imaging image with the same target resolution. For example, the target resolution can be the resolution of the first imaging image or a preset resolution; this embodiment is not limited thereto.

[0224] For example, if the first fused spectral feature of the first multispectral signal is A, and the dimension of A is 1×1×C, the first fused spectral feature A of the first multispectral signal is mapped to multiple modulation terms with a dimension of 1×1×C through a neural network model. The second fused light source distribution feature is B, and the dimension of B is H×W×C. For example, if the modulation terms are X and Y, then the modulation terms X and Y are multiplied and / or added pixel by pixel with the light source distribution feature B in the spatial dimension of (H, W). For example, X*B+Y, X+B, Y*B+X and / or Y+B. The corrected second fused light source distribution feature B' is obtained, and the dimension of B' is also H×W×C. Then, the distribution feature of the corrected second fused light source distribution feature B' is restored to obtain the first fused light source distribution feature of the first imaging image with the same resolution as the target image.

[0225] For example, the operation of S702 is performed by the feature extraction module and the correction module configured in the processor.

[0226] S703. Based on the first fused light source distribution characteristics of the first imaging image, obtain the light source partitioning information of the first imaging image.

[0227] Specifically, the first fused light source distribution feature of the first imaging image can reflect the regions of different light sources in the imaging image. S703 refers to the regions of different light sources reflected by the first fused light source distribution feature of the first imaging image to divide the first imaging image into light source partitions to obtain the light source partition information of the first imaging image.

[0228] The implementation process of S703 can be configured according to actual needs, and the embodiments of this application do not limit it. The following embodiments of this application provide two different implementation processes of S703, but do not constitute a specific limitation.

[0229] Implementation process A: Based on the first fused light source distribution characteristics of the first imaging image and the first imaging image, the light source partitioning information of the first imaging image is obtained.

[0230] In one possible implementation, when the size of the first fused light source distribution feature of the first imaging image is different from the size of the first imaging image, in implementation process A, the first fused light source distribution feature of the first imaging image can be interpolated and mapped with reference to the first imaging image to obtain the light source partitioning information of the first imaging image.

[0231] For example, such as Figure 10 As shown in (a), referring to the first imaging image, interpolation mapping is performed on the first fused light source distribution features of the first imaging image to obtain the light source partitioning information of the first imaging image. Interpolation involves adjusting the size of the first fused light source distribution features of the first imaging image to obtain light source partitioning information that meets the expected size. For example, the interpolation method can be bilinear interpolation or nearest neighbor interpolation, which is not limited in this embodiment. Mapping is a mathematical transformation method for establishing the coordinate correspondence between pixels of two images. It defines the coordinate correspondence between pixels of the first fused light source distribution features of the first imaging image and pixels of the first imaging image, thereby obtaining the corresponding position of each pixel in the first imaging image within the first fused light source distribution features of the first imaging image. For example, the mapping method can be forward mapping or backward mapping, which is not limited in this embodiment.

[0232] In another possible implementation, if the size of the first fused light source distribution feature of the first imaging image is the same as the size of the first imaging image, then there is no need to interpolate and map the first fused light source distribution feature of the first imaging image. In this case, the first fused light source distribution feature of the first imaging image can be directly used as the light source partitioning information of the first imaging image.

[0233] For example, the light source partitioning information obtained through process A can achieve pixel-level light source partitioning.

[0234] Implementation process B: In implementation process B, a guidance map of the first imaging image is first generated. Then, based on the first fused light source distribution characteristics and the guidance map of the first imaging image, the light source partitioning information of the first imaging image is obtained.

[0235] The guide map is used to indicate the distribution of color temperature and / or brightness. For example, the guide map can be generated using color space conversion, chromaticity coordinate calculation, McCamy formula calculation, blackbody radiation model, weighted average method, white point estimation method or deep learning model, or other methods, and the embodiments of this application are not limited thereto.

[0236] For example, such as Figure 10As shown in (b), after generating the guide map of the first imaging image, interpolation mapping is performed based on the guide map and the first fused light source distribution characteristics of the first imaging image to obtain the light source partition information of the first imaging image.

[0237] In one possible implementation, the above-mentioned interpolation mapping based on the first fused light source distribution features and the guide map of the first imaging image to obtain light source partitioning information includes: interpolating the first fused light source distribution features of the first imaging image based on the guide map to obtain the light source partitioning information of the first imaging image.

[0238] For example, the light source partitioning information obtained through process B can achieve pixel-level light source partitioning.

[0239] S704. Based on the light source partitioning information of the first imaging image, perform partitioning processing on the first imaging image.

[0240] Specifically, the zoning process involves applying different processing methods to different light source regions in the first imaging image based on light source zoning information, in order to obtain a scene in the imaging image that approximates the true colors. For example, the zoning process includes zoning white balance or zoning color correction.

[0241] For example, the image is a computer screen photographed by a user indoors. The indoor scene has a warm color tone, while the computer screen has a cool color tone. If the image is not processed by the above S701-S704 steps, the color of the computer screen in the image will be bluish. However, after white balance processing is performed on the image based on the light source partition information obtained by S701-S703, the color of the computer screen in the resulting image will be the same as its true color (the computer screen is white).

[0242] For example, the operation of S704 is performed by the partitioning module configured in the processor.

[0243] In the solution provided in this application, compared to color images, multispectral signals have more channels and richer, more complete spectral features. Therefore, the spectral features of multispectral signals can more accurately distinguish the types of light sources in a scene. Thus, correcting the light source distribution features extracted from the imaging image based on these spectral features can improve the accuracy of the corrected light source distribution features.

[0244] Furthermore, if correction is based entirely on the light source distribution features extracted from the imaging image, temporal local flicker is likely to occur as the input image changes over time. However, by relying on spectral features without spatial resolution to perform global correction on the obtained light source distribution features, the solution provided in this application can improve the robustness to changes in the acquired imaging image, reduce local flicker of light source partitioning information, and thus reduce local flicker in the imaging image obtained after partitioning processing.

[0245] On the other hand, the fused spectral features of the multispectral signals or the fused light source distribution features of the imaging images, obtained based on the spectral features of the multispectral signals acquired at different times or the light source distribution features of the imaging images, can reduce the inter-frame differences of consecutive image frames and ensure a smooth transition between adjacent frames in time-dependent scenes. Therefore, the light source partitioning information obtained through this scheme has both the characteristics of smooth transition between consecutive image frames and the characteristics of high light source partitioning accuracy, thus effectively ensuring the temporal stability of the light source partitioning.

[0246] Furthermore, in this application, since the multispectral signal used has no spatial resolution, the first sensor for acquiring the multispectral signal can be a single-point multispectral sensor, or a sensor with spatial resolution, such as a multi-point multispectral sensor or an array multispectral sensor, thereby improving the compatibility of the solution.

[0247] Furthermore, when the first sensor is a single-point multispectral sensor, the hardware cost of the solution is also reduced.

[0248] Furthermore, this application also provides a process for determining a fusion coefficient, which is used to fuse the light source distribution features of a first imaging image with the light source distribution features of a second imaging image. For example, this fusion coefficient can be used to perform feature fusion in S702.

[0249] For example, the process for determining the fusion coefficient provided in this application includes: acquiring the feature differences of key features between a first imaging image and a second imaging image; predicting the fusion coefficient based on the feature differences; and using the fusion coefficient to fuse the light source distribution features of the first imaging image with the reference light source distribution features of the second imaging image.

[0250] Among them, the key feature is the feature with spatial resolution.

[0251] For example, the feature difference can be obtained by subtracting key features from the first imaging image and the second imaging image or by other operations, which is not limited in the embodiments of this application.

[0252] For example, the fusion coefficient can be predicted using the coefficient prediction module.

[0253] In one possible implementation, the coefficient prediction module can utilize traditional image processing methods (Laplacian pyramid, Gaussian pyramid, minimum mean square error (MSE), etc.) to predict fusion coefficients based on feature differences.

[0254] In another possible implementation, the coefficient prediction module can be based on deep learning methods (CNN, ResNet, etc.) to predict fusion coefficients based on feature differences. For example, the coefficient prediction module can be a deep learning model.

[0255] Figure 11 This illustrates a process for determining fusion coefficients. This process is used to determine the fusion coefficients for feature fusion of the light source distribution features of a first imaging image and the reference light source distribution features of a second imaging image. For example... Figure 11 As shown, the process may include: extracting features from the first and second imaging images respectively to obtain key features of the first and second imaging images; calculating (e.g., subtracting) the key features of the first and second imaging images to obtain the feature differences of the key features; and inputting the feature differences into the coefficient prediction module to obtain the fusion coefficients output by the coefficient prediction module.

[0256] It is understood that, in order to achieve the above functions, the apparatus for running these functions includes software modules that perform the respective functions. Those skilled in the art should readily recognize that, based on the method steps of the examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in a combination of hardware and computer software. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0257] This application embodiment can divide the image processing apparatus containing image processing functions into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of software functional modules. It should be noted that the module division in this application embodiment is illustrative and is only a logical functional division. In actual implementation, there may be other division methods.

[0258] When dividing each function into modules according to its corresponding function. Figure 12 This is a schematic diagram of a possible structure of the image processing device involved in the above embodiments. For example... Figure 12 As shown, the image processing device 120 includes: a processing module 1201, an acquisition module 1202, and a partitioning module 1203. The processing module 1201 is used to perform... Figure 7The illustrated method refers to any one of operations S701, S703, or S704. The acquisition module 1202 is used to execute... Figure 7 The illustrated method shows the operation in S702. The partition module 1203 is used to execute... Figure 7 The operation in S703 is illustrated in the method.

[0259] Optional, such as Figure 13 As shown, the image processing apparatus 120 also includes a generation module 1204. The generation module 1204 is used to perform the operation of generating a guide map in S703.

[0260] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the steps or operations of any of the methods described above, for example, to perform the aforementioned... Figure 7 The steps or instructions for image processing methods.

[0261] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the steps or operations of the methods provided in the above embodiments of this application, for example, to perform the aforementioned... Figure 7 The steps or instructions for image processing methods.

[0262] This application also provides a chip. The chip system includes a processor and may further include a memory for executing the aforementioned functions. Figure 7 The image processing method refers to the steps or instructions for operation. The chip system can consist of chips or include chips and other discrete components.

[0263] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0264] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, and can be electrical, mechanical or other forms.

[0265] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0266] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0267] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, Applied to an electronic device, the electronic device including a first sensor and a second sensor for imaging, the method includes: Based on the spectral features of the first multispectral signal and the reference spectral features of the second multispectral signal, a first fused spectral feature of the first multispectral signal is obtained; wherein, the spectral feature is a single-point feature without spatial resolution; the first multispectral signal and the second multispectral signal are multispectral signals collected by the first sensor at different times; Based on the first fused spectral features, the light source distribution features of the first imaging image, and the reference light source distribution features of the second imaging image, the first fused light source distribution features of the first imaging image are obtained; the first imaging image and the second imaging image are imaging images acquired by the second sensor at different times. Based on the first fused light source distribution characteristics, the light source partitioning information of the first imaging image is obtained; Based on the light source partitioning information, the first imaging image is partitioned.

2. The method of claim 1, wherein, The first fused spectral feature of the first multispectral signal is obtained based on the spectral features of the first multispectral signal and the reference spectral features of the second multispectral signal, including: The spectral features of the first multispectral signal and the reference spectral features of the second multispectral signal are fused to obtain the first fused spectral features.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the light source partitioning information of the first imaging image based on the first fused light source distribution characteristics includes: Based on the first fused light source distribution characteristics and the first imaging image, the light source partitioning information is obtained.

4. The method according to claim 1, characterized in that, The method further includes: generating a guide map of the first imaging image, the guide map being used to indicate the distribution of color temperature and / or brightness; The step of obtaining the light source partitioning information of the first imaging image based on the first fused light source distribution characteristics includes: Based on the first fused light source distribution characteristics and the guide map, the light source partitioning information is obtained.

5. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining the first fused light source distribution feature of the first imaging image based on the first fused spectral feature, the light source distribution feature of the first imaging image, and the light source distribution feature of the second imaging image includes: The light source distribution characteristics of the first imaging image are corrected using the first fused spectral features; Based on the light source distribution characteristics of the corrected first imaging image and the light source distribution characteristics of the second imaging image, the first fused light source distribution characteristics are obtained.

6. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining the first fused light source distribution feature of the first imaging image based on the first fused spectral feature, the light source distribution feature of the first imaging image, and the light source distribution feature of the second imaging image includes: Based on the light source distribution characteristics of the first imaging image and the light source distribution characteristics of the second imaging image, a second fused light source distribution characteristic is obtained; Using the first fused spectral characteristics, the second fused light source distribution characteristics are corrected to obtain the first fused light source distribution characteristics.

7. The method of claim 2, wherein, The feature fusion includes: Feature splicing and fusion, wherein the feature splicing and fusion is used to fuse features by splicing; or, Element-level feature fusion, wherein the element-level feature fusion is used to perform mathematical operations on features pixel by pixel; or, Feature fusion based on attention mechanism, wherein the feature fusion based on attention mechanism is used to perform feature fusion using attention mechanism; or, Coefficient-weighted feature fusion, wherein the coefficient-weighted feature fusion is used to perform weighted feature fusion based on fusion coefficients; or, Feature fusion based on gating mechanism, wherein the feature fusion based on gating mechanism is used to perform feature fusion using gating mechanism.

8. The method according to claim 2 or 7, characterized in that, The method further includes: Obtain the feature differences of key features between the first imaging image and the second imaging image; Based on the aforementioned feature differences, a fusion coefficient is predicted; the fusion coefficient is used to fuse the light source distribution features of the first imaging image and the light source distribution features of the second imaging image.

9. The method of claim 8, wherein, The key feature is a feature with spatial resolution.

10. The method according to any one of claims 1 to 9, characterized in that, The partitioning process includes: partition white balance or partition color correction.

11. The method according to any one of claims 1-10, characterized in that, The reference spectral features of the second multispectral signal include: the first fused spectral features of the cached second multispectral signal; The reference light source distribution features of the second imaging image include: the first fused light source distribution features of the cached second imaging image.

12. The method according to any one of claims 1 to 11, characterized in that, The method further includes: Cache the first fused spectral features of the first multispectral signal; The first fused light source distribution features of the first imaging image are cached.

13. The method according to any one of claims 1 to 12, characterized in that, The first multispectral signal is the current frame multispectral signal, and the second multispectral signal is the previous frame multispectral signal; the first imaging image is the current frame imaging image, and the second imaging image is the previous frame imaging image.

14. The method of claim 2, wherein, The first multispectral signal is the current frame multispectral signal, and the second multispectral signal is one or more previous frame multispectral signals; the first imaging image is the current frame imaging image, and the second imaging image is one or more previous frame imaging images; When the second multispectral signal is composed of multiple previous frame multispectral signals, the step of fusing the spectral features of the first multispectral signal with the reference spectral features of the second multispectral signal to obtain the first fused spectral feature includes: The spectral features of the first multispectral signal are fused with the reference spectral features of multiple second multispectral signals to obtain the first fused spectral features.

15. An image processing apparatus characterized by comprising: The device includes: The processing module is used to obtain a first fused spectral feature of the first multispectral signal based on the spectral features of the first multispectral signal and the reference spectral features of the second multispectral signal; wherein the spectral feature is a single-point feature without spatial resolution; the first multispectral signal and the second multispectral signal are multispectral signals collected by the first sensor at different times; The acquisition module is used to acquire the first fused light source distribution feature of the first imaging image based on the first fused spectral feature, the light source distribution feature of the first imaging image, and the reference light source distribution feature of the second imaging image; the first imaging image and the second imaging image are imaging images acquired by the second sensor at different times; The partitioning module is used to obtain the light source partitioning information of the first imaging image based on the first fused light source distribution characteristics; The processing module is also used to perform partitioning processing on the first imaging image based on the light source partitioning information.

16. An electronic device, comprising: The electronic device includes a processor and a memory; the processor is configured to execute instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the method as described in any one of claims 1-14.

18. A computer program product comprising instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-14.

Images