Image processing method and electronic device

Abstract

Disclosed in the embodiments of the present application are an image processing method and an electronic device. The electronic device comprises a first image sensor, wherein the first image sensor is a multi-spectral image sensor. The method comprises: acquiring a first initial image collected by a first image sensor, wherein the first initial image is a multi-spectral image; and performing first image adjustment processing on the first initial image, so as to obtain a first target image, wherein the first image adjustment processing comprises first color rendition processing. Thus, a multi-spectral image collected by a multi-spectral image sensor can be acquired, and image adjustment processing comprising color rendition processing is performed on the multi-spectral image; the image color rendition accuracy can be improved, so as to obtain an image having a better degree of color rendition, thereby effectively improving the image quality; and an image that better meets user requirements can be obtained, thereby effectively improving the user experience.

Description

Image processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on March 15, 2024, with application number 202410305870.3 and application name “Image Processing Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic equipment. Background Art

[0003] Currently, widely used electronic devices such as cameras and mobile phones are equipped with color image sensors, such as RGB (red (Red, R), green (Green, G), blue (Blue, B)) image sensors. However, the response of RGB image sensors to the spectrum is the sampling or integration of the complete spectral signal. RGB image sensors cannot fully capture the spectral signal, which easily causes problems such as metamerism. As a result, in some scenarios, the image color restoration performance is limited and the image color cannot be accurately restored, affecting the image quality and user experience. Compared with RGB image sensors, multispectral image sensors can capture the original spectral signals more completely. Therefore, based on the original spectral signals captured by multispectral image sensors, how to obtain images with better color restoration and improve image quality is a problem that needs to be solved urgently. Furthermore, different users often have different requirements for images. Therefore, how to obtain images that better meet user needs is also very important in the field of image processing. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method and device for image processing model training, as well as electronic equipment, which can solve the above-mentioned technical problems, namely, based on the original spectral signals captured by the multispectral image sensor, images with better color reproduction can be obtained, effectively improving image quality, and images that better meet user needs can be obtained, thereby effectively improving user experience.

[0005] To solve the above-mentioned technical problems, in a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, wherein the electronic device includes a first image sensor, which is a multispectral image sensor. The method includes: obtaining a first initial image captured by the first image sensor, which is a multispectral image, performing a first image adjustment process on the first initial image to obtain a first target image, wherein the first image adjustment process includes a first color restoration process.

[0006] In a possible implementation of the first aspect above, the method further includes: displaying the first target image.

[0007] In a possible implementation of the first aspect above, the electronic device further includes a second image sensor, where the second image sensor is an RGB image sensor or a grayscale image sensor. The method further includes: acquiring a second initial image captured by the second image sensor, where the second initial image is an RGB image or a grayscale image, and performing a second image adjustment process on the second initial image based on the first initial image to obtain a second target image, where the second image adjustment process includes a second color restoration process.

[0008] In a possible implementation of the first aspect above, the method further includes: displaying the first target image and / or the second target image.

[0009] The electronic device mentioned above may be, for example, a mobile phone, a watch, a camera, a tablet computer, or other electronic device with an image processing function.

[0010] The first initial image is a multispectral image acquired by a multispectral image sensor. A multispectral image includes spectral information of multiple (3 or more) bands, so the number of channels of the multispectral image is greater than or equal to 3. The second initial image is an RGB image acquired by an RGB image sensor or a grayscale image acquired by a grayscale image sensor.

[0011] Since the multispectral image includes a more complete original spectral signal, the above-mentioned multispectral image is obtained, or the above-mentioned multispectral image and RGB image (or grayscale image) are obtained, and image adjustment processing including color restoration processing is performed. Image color restoration processing can be performed based on the multispectral image to obtain a target image with better color restoration, effectively improving the image quality, and obtaining an image that better meets user needs, better meeting user needs, thereby effectively improving the user experience.

[0012] In a possible implementation of the first aspect, a spatial resolution of the first initial image is smaller than a spatial resolution of the second initial image, and a number of channels of the first initial image is larger than a number of channels of the second initial image.

[0013] In a possible implementation of the first aspect, the spatial resolution of the first target image is smaller than the spatial resolution of the second target image. For example, the second target image is a color image, and the second target image is a high-resolution color image.

[0014] In a possible implementation of the first aspect, performing the first color restoration processing on the first initial image may be performing the first color restoration processing on the first initial image using a pre-trained image processing model.

[0015] In a possible implementation of the first aspect, performing the second color restoration processing on the second initial image may be performing the second color restoration processing on the second initial image using a pre-trained image processing model.

[0016] In a possible implementation of the first aspect above, the first image adjustment processing may also include adjustment processing of other image features or parameters other than the first color restoration processing for the image color, such as image brightness, resolution, clarity, etc.

[0017] In a possible implementation of the first aspect above, the second image adjustment processing may also include adjustment processing of other image features or parameters other than the second color restoration processing for the image color, such as image brightness, resolution, clarity, etc.

[0018] In a possible implementation of the first aspect above, the first image adjustment processing and / or the second image adjustment processing further includes any one of the following processing: image preprocessing; image stylization processing; image spatial resolution enhancement processing.

[0019] In one possible implementation of the first aspect, the electronic device may acquire the first initial image, or the first initial image and the second initial image, based on a user operation. Taking a mobile phone as an example, the user operation may include, for example, the user taking a photo in a photo-taking scenario, or the user selecting a photo-taking scenario, photo-taking mode, or other image usage scenario, or other operations.

[0020] In a possible implementation of the first aspect above, the electronic device may display the first target image and / or the second target image on a display screen of the electronic device, where the display screen may be, for example, a main screen, a secondary screen, or the like.

[0021] In a possible implementation of the first aspect above, the method further includes: displaying the first target image and / or the second target image according to target information, where the target information includes information related to the image usage scenario and / or device information of the electronic device.

[0022] The image usage scenario-related information corresponding to the image usage scenario may include, for example, information about normal photography scenarios, strong ambient light photography scenarios, weak ambient light photography scenarios, and other photography scenario information, as well as information about the image resolution or clarity requirements of the image usage scenario, ambient light intensity information, etc. Since users have different image requirements in different scenarios, the image processing method can be determined based on the image usage scenario-related information.

[0023] Device information of an electronic device may include, for example, screen information such as the number of screens, e.g., whether the electronic device has only one display screen, or two display screens, a primary screen and a secondary screen, etc., or screen size information. Thus, image processing methods may vary based on the number of screens, screen size, etc., to better meet user needs and enhance user experience.

[0024] In a possible implementation of the first aspect above, the method further includes: displaying the first target image and / or the second target image according to a third user operation.

[0025] The third user operation may be, for example, a user selecting or setting the display mode of the first target image and / or the second target image. For example, the user may select to display the first target image on the primary screen and the second target image on the secondary screen. The selection or setting mode can be configured as needed. In addition, the electronic device may display a fourth interface for the user to select or set the display mode of the first target image and / or the second target image.

[0026] In a possible implementation of the first aspect above, the electronic device further includes a second image sensor, which is an RGB image sensor or a grayscale image sensor. The method further includes: in response to the first user operation, determining whether to start the second image sensor; if it is determined that the second image sensor is not started, performing the aforementioned first image adjustment processing on the first initial image, that is, only starting the first image sensor to capture the first initial image, and performing the first image adjustment processing on the first initial image to obtain the first target image; if it is determined that the second image sensor is started, performing the aforementioned second image adjustment processing on the second initial image based on the first initial image, or performing the aforementioned second image adjustment processing on the second initial image based on the first initial image and the aforementioned first image adjustment processing on the first initial image, that is, starting the first image sensor to capture the first initial image, starting the second image sensor to capture the second initial image, performing the second image adjustment processing on the second initial image based on the first initial image to obtain the second target image, or performing the second image adjustment processing on the second initial image based on the first initial image to obtain the second target image, and performing the first image adjustment processing on the first initial image to obtain the first target image.

[0027] In a possible implementation of the first aspect above, the first user operation is a user selection operation for an image usage scenario, and the method further includes: displaying a first interface, the first interface including a selection control for an image usage scenario; receiving a user selection operation for an image usage scenario through the selection control; then, in response to the first user operation, determining whether to start the second image sensor, including: in response to the first user operation, determining the image usage scenario selected by the user, and determining whether to start the second image sensor based on the image usage scenario selected by the user.

[0028] Based on the image usage scenario selected by the user, determining whether to start the second image sensor to obtain the first target image and / or the second target image can obtain an image that better meets the user's needs, effectively improving the user experience.

[0029] In a possible implementation of the first aspect above, determining whether to start the second image sensor in response to a first user operation includes: obtaining target information in response to the first user operation, the target information including information related to the image usage scenario and / or device information of the electronic device; and determining whether to start the second image sensor based on the target information.

[0030] Taking a mobile phone as an example, the first user action could be a click on a camera control. When the electronic device determines that a photo is needed, it can automatically obtain information related to the current photo scene, such as required image resolution, ambient light information, and / or device information such as the number of screens on the phone, to determine whether to activate the second image sensor. This allows for an image that is more appropriate for the current photo scene or the phone, better meeting user needs.

[0031] In a possible implementation of the first aspect above, performing a first image adjustment process on the first initial image to obtain a first target image includes: performing a first color restoration process on the image of the first area in the first initial image to obtain a fourth target image, performing target processing on the image of the second area in the first initial image to obtain a fifth target image; performing image fusion processing on the fourth target image and the fifth target image to obtain the first target image; or performing a first image adjustment process on the first initial image to obtain the first target image includes: performing a first color restoration process on the entire first initial image to obtain a third target image, performing a first color restoration process on the image of the first area in the first initial image to obtain a fourth target image, performing target processing on the image of the second area in the first initial image to obtain a fifth target image; and performing image fusion processing on the third target image, the fourth target image, and the fifth target image to obtain the first target image.

[0032] By performing different image adjustment processing on images in different spectral characteristic areas in the image and then fusing the images that have undergone different image adjustment processing, better adjustment of image features or parameters can be achieved, resulting in images with better image quality and more in line with user needs.

[0033] In a possible implementation of the first aspect above, the first region is a region corresponding to a sub-channel in the first initial image that does not conform to the target spectral characteristics, and the second region is a region corresponding to a sub-channel in the first initial image that conforms to the target spectral characteristics.

[0034] In one possible implementation of the first aspect, the target spectral characteristic is a spectral response of a first subchannel that is greater than or equal to a preset first response threshold or less than a preset second response threshold. The first subchannel can be a channel corresponding to any wavelength band, such as an infrared band or an ultraviolet band.

[0035] In a possible implementation of the first aspect above, the method further includes: determining the target spectral feature in response to a second user operation.

[0036] In a possible implementation of the first aspect above, the second user operation is a user setting operation on the target spectral characteristics, and the method also includes: displaying a second interface, the second interface including a setting control for the target spectral characteristics; receiving the user setting operation on the target spectral characteristics through the setting control; then, in response to the second user operation, determining the target spectral characteristics, including: determining the target spectral characteristics set by the user in response to the second user operation.

[0037] For example, the mobile phone may display a display interface including content such as the infrared band corresponding channel response being greater than the response threshold, so that the user can select or set the target spectral characteristics through the display interface, thereby implementing the setting operation of the target spectral characteristics.

[0038] In a possible implementation of the first aspect above, the second user operation is a user selection operation for an image usage scenario, and the method further includes: displaying a third interface, the third interface including a selection control for the image usage scenario; receiving the user's selection operation for the image usage scenario through the selection control; then, in response to the second user operation, determining the target spectral characteristics, including: determining the image usage scenario selected by the user in response to the second user operation, and determining the target spectral characteristics according to the image usage scenario selected by the user and a preset correspondence between the image usage scenario and the target spectral characteristics.

[0039] For example, a mobile phone can pre-set the correspondence between image usage scenarios and target spectral characteristics. For example, image usage scenarios can include camera recognition and leather recognition. For camera recognition, the target spectral characteristic corresponding to the infrared channel spectral response greater than X, while for leather recognition, the target spectral characteristic corresponding to the ultraviolet channel spectral response greater than Y. This way, users only need to select the image usage scenario they need, and the mobile phone can conveniently determine the corresponding target spectral characteristic based on the selected image usage scenario.

[0040] In one possible implementation of the first aspect, the target processing (i.e., target image adjustment processing) is any one of the following: masking an overexposed area; enhancing brightness of a target area; or grayscaling the target area. Of course, the target processing may also include adjusting other image features or parameters.

[0041] In one possible implementation of the first aspect, performing a first image adjustment process on the first initial image to obtain a first target image includes: performing a first color restoration process on the first initial image to obtain a third target image; determining image brightness information based on spectral response information of a second subchannel in the first initial image; and performing a brightness shift process on the third target image based on the image brightness information to obtain the first target image. The second subchannel may, for example, be a channel having a spectral response greater than or less than a preset response threshold.

[0042] In a possible implementation of the first aspect above, performing a first image adjustment process on the first initial image to obtain a first target image includes: determining spectral characteristics of the first initial image; determining first image adjustment information based on the spectral characteristics; and performing a first color restoration process on the first initial image based on the first image adjustment information to obtain the first target image.

[0043] In a possible implementation of the first aspect above, the first image adjustment information includes a color subspace projection matrix, an original domain response of a light source power spectrum, and a subspace color correction matrix. According to the first image adjustment information, a first color restoration process is performed on the first initial image to obtain a first target image, including: performing color subspace projection process on the first initial image according to the color subspace projection matrix to obtain a subspace response image; obtaining a white balance diagonal matrix according to the color subspace projection matrix and the original domain response of the light source power spectrum, and performing white balance process on the subspace response image according to the white balance diagonal matrix to obtain a white balanced image; performing color correction process on the white balanced image according to the subspace color correction matrix to obtain a color adapted image as the first target image, or performing color gamut transformation process on the color adapted image to obtain the first target image.

[0044] In a possible implementation of the first aspect above, performing a second image adjustment process on the second initial image based on the first initial image to obtain a second target image includes: determining spectral characteristics of the first initial image; determining second image adjustment information based on the spectral characteristics; and performing a second color restoration process on the second initial image based on the second image adjustment information to obtain a second target image.

[0045] In a possible implementation of the first aspect above, if the second initial image is an RGB image, the second image adjustment information includes white balance adjustment parameters and a subspace color correction matrix, the white balance adjustment parameters include a white balance diagonal matrix, and according to the second image adjustment information, a second color restoration process is performed on the second initial image to obtain a second target image, including: performing white balance processing on the second initial image according to the white balance adjustment parameters to obtain a white balanced image; and performing color correction processing on the white balanced image according to the subspace color correction matrix to obtain a chromatic adaptation image as the second target image.

[0046] In a possible implementation of the first aspect above, the second image adjustment information is a third image determined based on the first initial image, and according to the second image adjustment information, a second color restoration process is performed on the second initial image to obtain a second target image, including: according to the third image, a color migration process is performed on the second initial image to obtain a second target image.

[0047] In a possible implementation of the first aspect above, performing a first image adjustment process on the first initial image to obtain a first target image includes: using at least one neural network to perform a first color restoration process on the first initial image to obtain the first target image.

[0048] In a possible implementation of the first aspect above, the user operation may be a touch operation on the display interface, or other operations such as voice and gesture.

[0049] The aforementioned multispectral image may be, for example, a multispectral raw (RAW) image, the RGB image may be, for example, a high-resolution color RAW image, and the grayscale image may be, for example, a high-resolution grayscale RAW image.

[0050] Based on this, in a possible implementation of the first aspect above, an embodiment of the present application provides an image processing method, which includes: inputting a first image into an image processing module, the first image includes a first sub-image (that is, a first initial image), the first sub-image is a multispectral original image collected by a first sensor, and the image processing module includes an image processing model pre-trained based on machine learning; using the image processing model to perform a first color restoration process on the first image to obtain a second image (as an example of a first target image), the first color restoration process includes an operation of adjusting the color of the first image based on the first image adjustment information obtained by the image processing model through the first sub-image, and the second image is a color image.

[0051] The first image sensor is a multispectral image sensor. The image corresponding to the raw spectral signal captured by the multispectral image sensor can be referred to as a multispectral raw (RAW) image. The first image can, for example, include only the multispectral RAW image, a multispectral RAW image and a high-resolution color RAW image, or a multispectral RAW image and a high-resolution grayscale RAW image. The multispectral RAW image is an example of the first sub-image included in the first image (i.e., an example of the first initial image), the high-resolution color RAW image is an example of the second sub-image included in the first image (i.e., an example of the second initial image), and the high-resolution grayscale RAW image is another example of the second sub-image included in the first image (i.e., another example of the second initial image). That is, the first image can include only the first sub-image or both the first and second sub-images. Of course, the first image can also include other images, which can be selected and configured as needed. Furthermore, the high-resolution color (or grayscale) RAW image can be captured by the second image sensor. The second image sensor can, for example, be a conventional image sensor, such as a conventional three-channel color image sensor (i.e., an RGB image sensor) or a single-channel grayscale image sensor. Among them, the ordinary 3-channel color image sensor is used to acquire high-resolution color RAW images with a spatial resolution greater than that of the multispectral RAW image, and the 1-channel grayscale image sensor is used to acquire high-resolution grayscale RAW images with a spatial resolution greater than that of the multispectral RAW image.

[0052] If the first image only includes a multispectral RAW image, the second image is a color image (i.e., a color image with lower resolution, referred to as a color image for short); if the first image includes a multispectral RAW image and a high-resolution color (or grayscale) RAW image, the second image is a high-resolution color image, or the second image includes a color image and a high-resolution color image.

[0053] In a possible implementation of the first aspect above, the first color restoration process may include, for example, color subspace projection processing, white balance processing, and subspace color correction processing, etc. Of course, other processing may also be included, which can be set as needed.

[0054] In a possible implementation of the first aspect above, the first image adjustment information includes, for example, a color subspace projection matrix, an original domain response of the light source power spectrum, a subspace color correction matrix, an image for color migration, etc. Of course, it may also include other information, which can be set as needed.

[0055] The image processing model is a learnable image processing model with pre-trained parameters based on machine learning or deep learning. Therefore, in this embodiment, the image processing model is used to perform color restoration processing on the first image, which can improve the accuracy of image color restoration and obtain a color image with better color restoration, effectively improving the image quality and thereby improving the user experience.

[0056] Moreover, through the image processing model, the first image is subjected to color restoration processing, and corresponding image adjustment information is obtained based on the multispectral RAW image, and color restoration processing is performed on the multispectral RAW image according to the image adjustment information corresponding to the multispectral RAW image, or color restoration processing is performed on the high-resolution color (or grayscale) RAW image, which can also effectively improve the image color restoration accuracy and obtain a color image with better color restoration, effectively improve the image quality, and thus improve the user experience.

[0057] In a possible implementation of the first aspect above, a first color restoration process is performed on a first image using an image processing model to obtain a second image, including: based on the first sub-image, using the image processing model to perform processing to obtain first image adjustment information; and according to the first image adjustment information, performing a first color restoration process on the first sub-image to obtain the second image.

[0058] In this embodiment, by using the image adjustment information corresponding to the multispectral RAW image obtained by the image processing model, the multispectral RAW image is subjected to color restoration processing, which can improve the accuracy of image color restoration and obtain a color image with better color restoration, effectively improving the image quality and thereby improving the user experience.

[0059] In a possible implementation of the first aspect above, based on the first sub-image, processing is performed using an image processing model to obtain first image adjustment information, including: based on the first sub-image, processing is performed using an image processing model to obtain spectral characteristics of the first sub-image; based on the spectral characteristics, obtaining the first image adjustment information using the image processing model.

[0060] In this way, based on the spectral characteristics of the multispectral RAW image, image adjustment information can be obtained conveniently and accurately to perform color restoration processing on the multispectral RAW image, which can improve the accuracy of image color restoration and obtain a color image with better color restoration, effectively improving the image quality and thus improving the user experience.

[0061] In a possible implementation of the first aspect above, the first image adjustment information includes a color subspace projection matrix, an original domain response of a light source power spectrum, and a subspace color correction matrix. According to the first image adjustment information, a first color restoration process is performed on the first sub-image to obtain a second image, including: performing a color subspace projection process on the first sub-image according to the color subspace projection matrix to obtain a subspace response image; obtaining a white balance diagonal matrix according to the color subspace projection matrix and the original domain response of the light source power spectrum, and performing a white balance process on the subspace response image according to the white balance diagonal matrix to obtain a white balanced image; performing a color correction process on the white balanced image according to the subspace color correction matrix to obtain a color adapted image as the second image, or performing a color gamut transformation process on the color adapted image to obtain the second image.

[0062] In this embodiment, the image processing model can be a model that implements light source estimation and color adaptation functions, that is, a light source estimation and color adaptation model. Furthermore, in this embodiment, the first sub-image can be the aforementioned multispectral RAW image, and the second image is a color image. Furthermore, the light source can refer to an ambient light source, and therefore the light source estimation and color adaptation model can also be referred to as an ambient light source estimation and color adaptation model.

[0063] In a possible implementation of the first aspect above, the first image also includes a second sub-image, the second sub-image being a color image or a grayscale image captured by a second image sensor, the spatial resolution of the second sub-image being greater than the spatial resolution of the first sub-image, and performing a first color restoration process on the first image using an image processing model to obtain the second image, including: processing the first sub-image using the image processing model to obtain first image adjustment information; and performing the first color restoration process on the second sub-image based on the first image adjustment information to obtain the second image. The second image sensor may be a color image sensor or a grayscale image sensor, wherein a color image sensor captures a color image, and a grayscale image sensor captures a grayscale image.

[0064] In this embodiment, the first sub-image can be the aforementioned multispectral RAW image, the second sub-image can be the aforementioned high-resolution color RAW image, and the second image is a high-resolution color image. In this manner, a high-resolution color image is generated based on the multispectral RAW image and the high-resolution color RAW image. Specifically, based on the image adjustment information obtained from the multispectral RAW image, the high-resolution color RAW image is subjected to color restoration processing to obtain a high-resolution color image. This improves the accuracy of image color restoration and produces a color image with better color restoration, effectively enhancing image quality and, in turn, the user experience.

[0065] In a possible implementation of the first aspect above, the first adjustment information includes white balance adjustment parameters and a subspace color correction matrix, the white balance adjustment parameters include a white balance diagonal matrix, the second sub-image is a color image, and according to the first image adjustment information, a first color restoration process is performed on the second sub-image to obtain a second image, including: performing white balance processing on the second sub-image according to the white balance adjustment parameters to obtain a white balanced image; and performing color correction processing on the white balanced image according to the subspace color correction matrix to obtain a chromatic adaptation image as the second image.

[0066] In a possible implementation of the first aspect, the white balance diagonal matrix is ​​obtained according to the color subspace projection matrix and the original domain response of the light source power spectrum.

[0067] In a possible implementation of the first aspect above, the first image adjustment information includes a third image, and according to the first image adjustment information, a first color restoration process is performed on the second sub-image to obtain a second image, including: according to the third image, a color migration process is performed on the second sub-image to obtain a second image.

[0068] In this way, the color transfer process can further enhance the image color reproduction effect, resulting in a color image with better color reproduction, effectively improving image quality and, in turn, enhancing the user experience. Furthermore, the second sub-image can be a color image or grayscale image having a spatial resolution greater than that of the first sub-image.

[0069] In addition, in this embodiment, the image processing model may include, for example, at least one neural network. After the first image is input, the at least one neural network performs color restoration and other processing on the first image to obtain a third image. The multispectral RAW image is subjected to end-to-end color restoration processing through the third image, such as color migration processing, which can improve the accuracy of color restoration and obtain a color image with better color restoration, thereby effectively improving the image quality and thereby improving the user experience.

[0070] In a possible implementation of the first aspect above, the image processing model includes at least one neural network, and the image processing model is used to perform a first color restoration process on the first image to obtain a second image, including: using at least one neural network to perform a first color restoration process on the first sub-image to obtain the second image.

[0071] In this embodiment, by including an image processing model of at least one neural network, end-to-end color restoration processing can be conveniently performed on multispectral RAW images, which can improve the accuracy of color restoration and obtain color images with better color restoration, effectively improving image quality and thereby improving user experience.

[0072] In a possible implementation of the first aspect above, inputting the first image into an image processing module includes: preprocessing the first image to adjust the image quality of the first image to obtain a preprocessed first image; and inputting the preprocessed first image into the image processing module.

[0073] Preprocessing can include image demosaicing and other processing. Through preprocessing, the noise in the multispectral RAW image can be removed, the useful information in the image can be enhanced, and the image quality can be improved. As a result, a color image with better color reproduction can be obtained, which effectively improves the image quality and thus improves the user experience.

[0074] In a possible implementation of the first aspect above, the method further includes: performing stylization processing on the second image to obtain a fourth image (as another example of the first target image) having an image style of a preset image style.

[0075] In this way, a corresponding stylized color image can be obtained, which further effectively improves the image quality, meets the different needs of users, and improves the user experience.

[0076] In a possible implementation of the first aspect above, the method further includes: obtaining second image adjustment information based on the first sub-image; performing second color restoration processing on the target image based on the second image adjustment information to obtain a fifth image, where the target image includes the second image or the fourth image.

[0077] In this way, the obtained color image can be further subjected to color restoration processing to obtain a color image with better color restoration, thereby effectively improving the image quality and further improving the user experience.

[0078] In a possible implementation of the first aspect above, the second image adjustment information includes image overexposed area masking information and a sixth image, the image overexposed area masking information is the image overexposed area masking information corresponding to the overexposed area of ​​the sub-channel spectral response in the first sub-image, the sixth image is an image obtained by performing a third color restoration processing based on the spectral response of the non-overexposed sub-channel in the first sub-image, and according to the second image adjustment information, a second adjustment processing is performed on the target image to obtain a fifth image, including: performing image fusion processing based on the target image, the image overexposed area masking information and the sixth image to obtain the fifth image.

[0079] In this embodiment, by processing the overexposed area, a color image with better color reproduction can be obtained, which effectively improves the image quality and further enhances the user experience.

[0080] In a possible implementation of the first aspect above, the second image adjustment information includes image brightness information, the image brightness information is determined based on the spectral response of the target subchannel in the first sub-image, and a second adjustment process is performed on the target image based on the second image adjustment information to obtain the fifth image, including: adjusting the brightness of the target image based on the image brightness information to obtain the fifth image.

[0081] In this embodiment, by processing the image brightness, a color image with better color reproduction can be obtained, which effectively improves the image quality and further enhances the user experience.

[0082] In second aspect, an embodiment of the present application provides an image processing model training method, the method comprising: inputting a training sample image into an initial image processing model, the training sample image comprising a multispectral original image captured by a first image sensor; the initial image processing model obtains an output image based on the training sample image, and the output image is a color image; the initial image processing model is trained based on the output image to obtain an image processing model, and the image processing model is applied to the aforementioned image processing method.

[0083] In this way, based on the image processing model obtained through model training, color restoration processing is performed on multispectral RAW images, which can improve the accuracy of color restoration and obtain color images with better color restoration, effectively improving image quality and thus improving user experience.

[0084] In a third aspect, an embodiment of the present application provides an image processing device for executing the aforementioned image processing method.

[0085] In a possible implementation of the third aspect above, an embodiment of the present application provides an image processing device, including: a first input module, used to input a first image into an image processing module, the first image includes a first sub-image, the first sub-image is a multispectral original image captured by a first image sensor, and the image processing module includes an image processing model pre-trained based on machine learning; an image processing module, used to perform a first color restoration process on the first image using the image processing model to obtain a second image, the first color restoration process including an operation of adjusting the color of the first image based on the first image adjustment information obtained by the image processing model through the first sub-image, and the second image is a color image.

[0086] In a fourth aspect, an embodiment of the present application provides an image processing model training device, comprising: a second input module, used to input the training sample image into the initial image processing model, the initial image processing model module includes an initial image processing model, and the training sample image includes a multispectral original image captured by a first image sensor; the initial image processing model module is used to obtain an output image based on the training sample image, and the output image is a color image; a training module is used to train the initial image processing model based on the output image to obtain an image processing model, and the image processing model is applied to the aforementioned image processing method.

[0087] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing a computer program, the computer program including program instructions; a processor for executing program instructions so that the electronic device performs the aforementioned image processing method, or so that the electronic device performs the aforementioned image processing model training method.

[0088] In a sixth aspect, an embodiment of the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs the aforementioned image processing method, or so that the computing device cluster performs the aforementioned image processing model training method.

[0089] In a seventh aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute the aforementioned image processing method, or enables the computing device cluster to execute the aforementioned image processing model training method.

[0090] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the aforementioned image processing method, or the computing device cluster executes the aforementioned image processing model training method.

[0091] The relevant beneficial effects of the third to eighth aspects mentioned above can be found in the relevant descriptions in the first or second aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings used in the description of the implementation methods.

[0093] FIG1 is a schematic diagram showing a CFA arrangement corresponding to an RGB image sensor and the response of the RGB image sensor to a spectrum;

[0094] FIG2 shows a schematic diagram of cone cells in the human eye and the response of cone cells to light spectrum;

[0095] FIG3 shows a schematic diagram of a CFA arrangement corresponding to a multispectral image sensor and the response of the multispectral image sensor to a spectrum;

[0096] FIG4 is a flow chart showing an image processing system and a corresponding image processing method according to some embodiments of the present application;

[0097] FIG5 is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0098] FIG6A is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0099] FIG6B is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0100] FIG7 is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0101] FIG8 is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0102] FIG9 is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0103] FIG10 is a schematic diagram showing the structure of a global light source estimation and color adaptation model and the corresponding process of generating image adjustment parameters according to some embodiments of the present application;

[0104] FIG11 is a schematic diagram showing a structure of a local light source estimation and color adaptation model and a flow chart of corresponding image adjustment parameter generation according to some embodiments of the present application;

[0105] FIG12 is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0106] FIG13 is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0107] FIG14 is a schematic diagram showing a comparison of image processing effects according to some embodiments of the present application;

[0108] FIG15 is a schematic diagram showing some different images obtained by global and local estimation processes according to some embodiments of the present application;

[0109] FIG16 is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0110] FIG17 is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0111] FIG18 is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0112] FIG19 is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0113] FIG20A is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0114] FIG20B is a schematic diagram showing a comparison of image effects of disabling and enabling response truncation compensation according to some embodiments of the present application;

[0115] FIG20C is a schematic diagram showing some display interfaces of a mobile phone according to some embodiments of the present application;

[0116] FIG20D is a schematic diagram showing other display interfaces of a mobile phone according to some embodiments of the present application;

[0117] FIG20E is a schematic diagram showing other display interfaces of a mobile phone according to some embodiments of the present application;

[0118] FIG21A is a flowchart illustrating another image processing system and a corresponding image processing method according to some embodiments of the present application;

[0119] FIG21B is a schematic diagram showing a comparison of image effects of disabling and enabling image brightness migration according to some embodiments of the present application;

[0120] FIG22A is a schematic flow chart illustrating another image processing method according to some embodiments of the present application;

[0121] FIG22B is a schematic diagram showing other display interfaces of a mobile phone according to some embodiments of the present application;

[0122] FIG23 is a schematic structural diagram of an image processing device according to some embodiments of the present application;

[0123] FIG24 is a schematic structural diagram illustrating an image processing model training device according to some embodiments of the present application;

[0124] 25A and 25B are schematic diagrams showing some structural diagrams of computing devices according to some embodiments of the present application;

[0125] 26A and 26B are schematic diagrams showing some structures of a computing device cluster according to some embodiments of the present application. DETAILED DESCRIPTION

[0126] The technical solutions provided by the embodiments of this application will be further described below with reference to the accompanying drawings.

[0127] As mentioned above, currently widely used electronic devices with photography or image processing capabilities, such as cameras and mobile phones, are all equipped with color image sensors. Common color imaging systems that include color image sensors typically use a color filter array (CFA) arranged above the photosensitive area. CFA is also called a Bayer filter. The photosensitive area can be implemented based on technologies such as complementary metal-oxide semiconductor (CMOS).

[0128] Figure 1 (a) shows a common arrangement of CFA, which consists of a 2×2 cyclic unit of four pixels, that is, the cyclic unit can be understood as a 2×2 pixel arrangement. Each cyclic unit contains three primary colors: R, G, and B. Therefore, color image sensors are often also called RGB image sensors. In addition, the response of the RGB image sensor to the spectrum is shown in Figure 1 (b). The response of the image sensor to the spectrum can be reflected by the relative QE curve, which describes the photoelectric conversion efficiency of the image sensor. Since the absorption efficiency of semiconductor materials for light signals is related to the wavelength, the horizontal axis of the QE curve is the wavelength, for example, in nanometers (nm), and the vertical axis is the percentage corresponding to the photoelectric conversion efficiency, for example, 0.2 corresponds to 20%.

[0129] The human eye typically has three types of cone cells (S, M, and L, or L / M / S), each of which responds to light of different wavelengths. RGB image sensors are designed to directly simulate the spectral responses of the three types of cone cells in the human eye (which can be referred to as LMS cone cells). The structure of the cone cells in the human eye is shown in Figure 2(a). The responses of the three types of cone cells in the human eye to the spectrum are shown in Figure 2(b). For example, S cone cells respond to light with a wavelength in the range of 400-500nm, with a peak at 420-440nm; M cone cells respond to light with a wavelength in the range of 450-630nm, with a peak at 534-555nm; and L cone cells respond to light with a wavelength in the range of 500-700nm, with a peak at 564-580nm. Furthermore, the responses of the three types of cone cells to the spectrum correspond to the three primary colors: R, G, and B.

[0130] As can be seen in Figures 1(b) and 2(b), the QE curves of the RGB image sensor's spectral response deviate from the QE curves of the cone cells of the human eye. Therefore, the raw (RAW) response of an RGB image sensor typically requires color correction to restore the image's colors, thereby accurately reproducing the image's colors as seen by the human eye at the time of capture. The raw response of an RGB image sensor refers to the camera sensor's raw imaging signal, often referred to as the RAW image response or simply the RAW image signal. Furthermore, due to the color constancy of the human eye, the RAW image response is typically corrected through operations such as white balancing. For example, the paper "Hakki Can Karaimer, et al. A Software Platform for Manipulating the Camera Imaging Pipeline" describes a typical image color processing workflow for RGB image sensors. Color constancy refers to the perceptual property that maintains the perception of an object's surface color even when the color of the light illuminating the surface changes. In the field of image processing science, based on the human eye's cognitive characteristics of scenery, separating the background elements from the lighting elements in the image is the key to solving the problem.

[0131] Moreover, since the RGB image sensor's response to the spectrum is the sampling or integration of the complete spectral signal (also called spectral information), it is unable to fully capture the spectral signal, which can easily cause problems such as metamerism. As a result, the RGB camera including the RGB image sensor has limited image color reproduction performance in some difficult scenes, such as those containing large areas of pure color, and cannot accurately restore the image color, affecting image quality and user experience.

[0132] Compared to RGB image sensors, multispectral image sensors based on multispectral technology are designed to capture the original spectral signal more completely. Multispectral refers to spectral detection technology that can simultaneously acquire multiple optical spectrum bands (usually more than three) and can expand beyond visible light to infrared and ultraviolet light.

[0133] As shown in Figure 3 (a), in one implementation, the CFA included in the multispectral image sensor is composed of 4×4 sixteen pixels forming a circular unit above the photosensitive area, where each pixel corresponds to a color filter or color filter coating. Through this design, the spectral resolution of the multispectral image sensor is improved from 3 dimensions to 16 dimensions. The response of the multispectral image sensor to the spectrum, as shown in Figure 3 (b), includes more response curves. Therefore, the multispectral image sensor can capture the spectral signal more completely, thereby improving the image color restoration performance in difficult scenes. In addition, the circular unit included in the CFA included in the multispectral image sensor can be understood as an N×N pixel arrangement, where N is greater than or equal to 3.

[0134] Based on this, an embodiment of the present application provides an image processing method, which can be applied to electronic devices such as cameras and mobile phones. As shown in Figure 4, in one embodiment of the present application, the electronic device may include an image acquisition system and an image processing system. The image acquisition system may include a multispectral image sensor (i.e., a first image sensor), and the multispectral image sensor may acquire an image to obtain a multispectral RAW image (i.e., a first sub-image or a first initial image). RAW represents the original image signal read from the image sensor, and thus the multispectral RAW image represents the original image signal read from the multispectral image sensor. In addition, the multispectral image sensor may send the acquired multispectral RAW image to the image processing system, and the image processing system obtains a corresponding color image (as an example of a second image or a first target image) based on the multispectral RAW image.

[0135] In addition, in one embodiment of the present application, the image acquisition system may further include a common image sensor (i.e., a second image sensor), such as a common 3-channel color image sensor (i.e., an RGB image sensor) or a 1-channel grayscale image sensor. The common 3-channel color image sensor is used to acquire a high-resolution color RAW image having a spatial resolution greater than that of the multispectral RAW image, and the 1-channel grayscale image sensor is used to acquire a high-resolution grayscale RAW image having a spatial resolution greater than that of the multispectral RAW image. The high-resolution color (or grayscale) RAW image represents the original image signal read out from the common image sensor. Furthermore, the common image sensor may send the acquired high-resolution color (or grayscale) RAW image (i.e., a second sub-image or a second initial image) to the image processing system, and the image processing system may obtain a corresponding high-resolution color image (as another example of a second image, or as an example of a second target image) based on the multispectral RAW image and the high-resolution color (or grayscale) RAW image, or obtain a color image and a high-resolution color image.

[0136] Furthermore, as shown in FIG4 , in an embodiment of the present application, the image processing system includes a parameter-learnable image processing module (as an example of an image processing module), and the parameter-learnable image processing module includes a parameter-learnable image processing model (as an example of an image processing model). The parameter-learnable image processing model is an image processing model that has been pre-trained based on machine learning or deep learning. That is, the parameter-learnable image processing model is an image processing model obtained based on model training. By using the parameter-learnable image processing model and the multispectral RAW image sent by the multispectral image sensor, a color restoration process (also referred to as color adjustment process) is performed on the multispectral RAW image to obtain a corresponding color image. Alternatively, by using the parameter-learnable image processing model and the multispectral RAW image sent by the multispectral image sensor, a color restoration process is performed on the high-resolution color (or grayscale) RAW image sent by the ordinary image sensor to obtain a corresponding high-resolution color image.

[0137] That is, the embodiments of the present application provide a learning-based multispectral image color restoration method, or a learning-based new multispectral image color restoration path. The electronic device can perform color restoration processing on multispectral RAW images or high-resolution color (or grayscale) RAW images based on a parameter-learnable image processing model and a multispectral RAW image, which can improve the accuracy of image color restoration and obtain a color image with better color restoration, effectively improving the image quality and thereby improving the user experience.

[0138] Furthermore, the image processing model with learnable parameters can be composed of at least one of a neural network, a differentiable image processing operator (e.g., image processing based on a color space projection matrix, a color correction matrix, etc.), and a non-differentiable image processing operator (e.g., image processing based on a color space projection matrix, a color correction matrix, etc. that is processed by other modules). The neural network and / or the differentiable image processing operator contain learnable parameters.

[0139] Furthermore, as shown in Figure 5, in one embodiment of the present application, the multispectral image sensor included in the image acquisition system acquires a multispectral RAW image 101 (i.e., a multispectral RAW image with lower resolution), and inputs the multispectral RAW image 101 into a parameter-learnable image processing module 301. The parameter-learnable image processing module 301 can obtain a color image 401 based on the multispectral RAW image 101.

[0140] Image resolution is often expressed using spatial / spectral resolution. For example, an image can be represented as H'×W'×C', where H'×W' represents the spatial resolution and C' represents the spectral resolution. For example, a common three-channel color image has a spectral resolution of 3, while for multispectral images, C' is typically greater than 3.

[0141] In this embodiment, the resolution of the multispectral RAW image 101 may be, for example, h×w×N, where h×w represent the width and height of the image (i.e., spatial resolution), and N represents the number of image channels (i.e., spectral resolution), and N is typically greater than 3. The resolution of the color image 401 may be, for example, h×w×3.

[0142] In this way, the electronic device performs color restoration processing on the multispectral RAW image 101 based on the parameter-learnable image processing model and the multispectral RAW image 101, which can improve the color restoration accuracy and obtain a color image 401 with better color restoration, effectively improving the image quality and thus improving the user experience.

[0143] As shown in Figure 6A, in one embodiment of the present application, the image acquisition system includes a multispectral image sensor that acquires a multispectral RAW image 101 (i.e., a multispectral RAW image with lower resolution), and a common image sensor that acquires a high-resolution color (or grayscale) RAW image 102, and the multispectral RAW image 101 and the high-resolution color (or grayscale) RAW image 102 are input into an image processing module 301 with learnable parameters to obtain a high-resolution color image 402.

[0144] In this embodiment, the resolution of the high-resolution color (or grayscale) RAW image 102 can be, for example, H×W×C, where H×W represent the image width and height (i.e., spatial resolution), respectively, and H>h and W>w. For a high-resolution grayscale RAW image 102, its resolution can be, for example, H×W×1, i.e., C=1. Similarly, for a high-resolution color RAW image 102, its resolution can be, for example, H×W×3, i.e., C=3. Furthermore, the resolution of the high-resolution color image 402 can be, for example, H×W×3.

[0145] In this way, the electronic device performs color restoration processing on the high-resolution color (or grayscale) RAW image 102 based on the parameter-learnable image processing model and the multispectral RAW image 101, thereby obtaining a high-resolution color image 402. This improves color restoration accuracy, resulting in a high-resolution color image 402 with better color restoration, effectively improving image quality and, in turn, enhancing the user experience.

[0146] As shown in Figure 6B, in one embodiment of the present application, the image acquisition system includes a multispectral image sensor that acquires a multispectral RAW image 101 (i.e., a multispectral RAW image with lower resolution), and a common image sensor that acquires a high-resolution color (or grayscale) RAW image 102, and the multispectral RAW image 101 and the high-resolution color (or grayscale) RAW image 102 are input into a parameter-learnable image processing module 301, so that a color image 401 and a high-resolution color image 402 can be obtained at the same time.

[0147] In this manner, the electronic device performs color restoration processing on the multispectral RAW image 101 and the high-resolution color (or grayscale) RAW image 102, respectively, based on the parameter-learnable image processing model 301, the multispectral RAW image 101, and the high-resolution color (or grayscale) RAW image 102, to obtain a color image 401 and a high-resolution color image 402. This improves the accuracy of image color restoration, resulting in a color image 401 and a high-resolution color image 402 with better color restoration, effectively improving image quality and, in turn, enhancing the user experience.

[0148] Here, multispectral RAW image 101 is used as an example of a first initial image, high-resolution color (or grayscale) RAW image 102 is used as an example of a second initial image, color image 401 is used as an example of a first target image, and high-resolution color image 402 is used as an example of a second target image. Alternatively, multispectral RAW image 101 is used as an example of a first sub-image, high-resolution color (or grayscale) RAW image 102 is used as an example of a second sub-image, and color image 401 and / or high-resolution color image 402 are used as examples of a second image.

[0149] Furthermore, as shown in FIG7 , in some embodiments of the present application, the image processing system may further include at least one of a pre-processing module and a stylized post-processing module.

[0150] The preprocessing module, for example, includes a preprocessing module 201 and / or a preprocessing module 202. The preprocessing module 201 is used to preprocess the multispectral RAW image 101, and the preprocessing module 202 is used to preprocess the high-resolution color (or grayscale) RAW image 102. Furthermore, preprocessing includes at least one of image processing operations such as image demosaicing, image denoising, image super-resolution, sensor black level correction, lens shading correction, lens distortion correction, sensor spectral response correction, and sensor response normalization. Of course, preprocessing may also include other image processing operations, which can be selected and configured as needed. Preprocessing the multispectral RAW image using the preprocessing module can effectively improve the image quality of the multispectral RAW image, thereby improving the quality of the resulting color image.

[0151] The stylized post-processing module is used to perform stylized processing on the color image obtained by the parameter-learnable image processing module 301, resulting in, for example, a final color image 401 or a high-resolution color image 402. Stylized processing can include, for example, adjusting the image's saturation and contrast, or other processing. The image processing operations and stylized processing methods included in stylized processing can be selected and configured as needed. By stylizing color images using the stylized post-processing module, images can be obtained that better meet user needs, while also improving image quality and effectively enhancing the user experience.

[0152] In summary, in some embodiments of the present application, it can be understood that the electronic device mainly includes the following five parts:

[0153] Module 101 is used to input a multispectral RAW image 101 .

[0154] Module 102 is used to input a high-resolution color (or grayscale) RAW image 102 .

[0155] Modules 201 and 202 correspond to preprocessing module 201 and preprocessing module 202 respectively, and are used to preprocess the image.

[0156] Module 301 corresponds to the parameter-learnable image processing module 301 and includes the aforementioned parameter-learnable image processing model. If only the multispectral RAW image 101, or only the image corresponding to the multispectral RAW image 101 processed by the preprocessing module 201, is input, the parameter-learnable image processing module 301 outputs a color image 401. If both the multispectral RAW image 101 (or the image corresponding to the multispectral RAW image 101 processed by the preprocessing module 201) and the high-resolution color (or grayscale) RAW image 102 (or the image corresponding to the high-resolution color (or grayscale) RAW image 102 processed by the preprocessing module 202) are input, the parameter-learnable image processing model module 301 outputs a high-resolution color image 402, or simultaneously outputs both the high-resolution color image 402 and the color image 401. Furthermore, the processing performed by module 301 can be referred to as color restoration processing.

[0157] Modules 401 and 402 correspond to the color image 401 output and the high-resolution color image 402 output, respectively.

[0158] In this way, electronic devices can perform color restoration on multispectral RAW images based on a parameter-learnable image processing model, which can improve the accuracy of image color restoration and obtain color images with better color restoration, effectively improving image quality and thus improving user experience.

[0159] In other embodiments of the present application, the above-mentioned multispectral RAW image 101 and the parameter-learnable image processing module 301 are necessary, and others such as the high-resolution color (or grayscale) RAW image 102, the preprocessing module, the stylized post-processing module, the high-resolution color image 402, etc. are optional and can be selected and set as needed.

[0160] The structure of the image processing system and the image processing process related to obtaining the color image 401 based on the multispectral RAW image 101 will be further described below.

[0161] As shown in Figure 8, in one embodiment of the present application, the image processing system includes a preprocessing module 201 and a parameter-learnable image processing module 301, and the parameter-learnable image processing module 301 includes a parameter-learnable image processing model. The parameter-learnable image processing model includes at least one neural network, that is, the parameter-learnable image processing model can be implemented by a neural network or a collection of multiple neural networks. The parameter-learnable image processing model can include one or more general neural network components such as convolutional layers, pooling layers, activation layers, upsampling layers, downsampling layers, self-attention layers, residual connections, dense connections, etc., which can be selected and set as needed. In addition, the parameter-learnable image processing model can be obtained through model training, and its structure and training method can be set as needed.

[0162] In this embodiment, the multispectral RAW image 101 is first input into the preprocessing module 201. The preprocessing module 201 performs the aforementioned preprocessing, such as image demosaicing, on the multispectral RAW image 101 to obtain a preprocessed multispectral RAW image 101'. The preprocessing module 201 then inputs the preprocessed multispectral RAW image 101' into the parameter-learnable image processing module 301, i.e., into a parameter-learnable image processing model (also referred to as a neural network or a neural network set). The parameter-learnable image processing model performs image color restoration processing (as an example of a first color restoration process) on the preprocessed multispectral RAW image 101' to obtain a color image 401. Image color restoration processing can, for example, adjust the image's color through filtering, matrix aggregation, or other methods. In this manner, at least one neural network can be used to perform the first color restoration process, or both the preprocessing and the first color restoration process, on the multispectral RAW image 101 to obtain a color image 401 (as an example of a first target image).

[0163] Of course, in other embodiments of the present application, the electronic device may also include only the parameter-learnable image processing module 301 in FIG8 , without including the preprocessing module 201. The multispectral RAW image 101 is directly input into the parameter-learnable image processing module 301 to obtain the corresponding color image 401.

[0164] This embodiment is an end-to-end image color restoration solution based on the multispectral RAW image 101. Based on this embodiment, the electronic device can perform color restoration processing on the multispectral RAW image 101 based on a parameter-learnable image processing model and the multispectral RAW image 101, which can improve the accuracy of image color restoration and obtain a color image with better color restoration, effectively improving image quality and thereby improving the user experience.

[0165] As shown in Figure 9, in one embodiment of the present application, the image processing system includes a parameter-learnable image processing module 301 and a stylized post-processing module, wherein the parameter-learnable image processing module 301 includes a light source estimation and color adaptation model (as an example of a parameter-learnable image processing model) module, a matrix module and a color gamut (Gamut) transformation module, and the matrix module may include a single matrix or a matrix set including multiple matrices.

[0166] Among them, the light source estimation and color adaptation model module includes a light source estimation and color adaptation model, which is used to extract spectral features from the multispectral RAW image 101, obtain image adjustment parameters (as an example of the first image adjustment information), and input the image adjustment parameters into the matrix module. In this way, the estimation and adjustment of the image adjustment parameters can be achieved. The image adjustment parameters can be, for example, at least one of the parameters such as the color subspace projection matrix, the RAW response of the light source power spectrum (i.e., the light source power spectrum RAW response), and the subspace color correction matrix (Color Calibration Matrix, CCM). Of course, the image adjustment parameters can also be other parameters, which can be selected and set as needed. In addition, the method for extracting the spectral features corresponding to the light source estimation and color adaptation model can be through a model with learnable parameters such as a neural network, or through a non-parametric feature extraction method such as histogram statistics.

[0167] The matrix module is used to perform color restoration processing (as an example of the first color restoration processing) on ​​the multispectral RAW image 101 according to the image adjustment parameters to obtain a chromatic adaptation image, and input the chromatic adaptation image into the Gamut transformation module.

[0168] The Gamut transform module is used to perform color gamut transformation processing on the color-adapted image to obtain a color image 401′ (as an example of the second image, or as an example of the first target image). The color image 401′ is then input into the stylization post-processing module for further stylization processing to obtain a color image 401 (as an example of the fourth image, or as another example of the first target image).

[0169] Furthermore, in one embodiment of the present application, the image adjustment parameters include a color subspace projection matrix, a RAW response of a light source power spectrum, and a subspace color correction matrix. For example, the color subspace projection matrix is ​​T N×3 , the RAW response of the light source power spectrum is L, the subspace color correction matrix T M×3. Among them, N and M can be determined according to the channel dimension of the multispectral RAW image, that is, N is the number of multispectral image channels, which represents the spectral resolution, and M is usually a multiple of 3, which can be determined according to actual conditions. Then, the light source estimation and color adaptation model can be implemented by a neural network, and the input of the light source estimation and color adaptation model is a multispectral RAW image. After spectral feature extraction, spectral features are obtained, and the spectral features are output to multiple branches. The multiple branches have differentiated feature processing processes, such as feature dimensionality reduction processing, global average pooling processing, activation processing, scale scaling, and normalization processing. Of course, the feature processing process can also include other processing, which can be selected and set as needed.

[0170] For example, in one embodiment of the present application, the light source estimation and color adaptation model can be a global light source estimation and color adaptation model. The structure of the global light source estimation and color adaptation model is shown in FIG10 , which includes a backbone model module and three branch modules. The backbone model module includes a backbone model. The backbone model can be any neural network based on a convolutional neural network (CNN) or a Transformer. The backbone model is used for multispectral RAW images 101, such as X∈R h×w×N , perform spectral feature extraction to obtain spectral features, such as F∈R h′×w′×N′ . Then, the spectral features are respectively input into the three branch modules, namely, branch 1 module (as an example of the first branch module), branch 2 module (as an example of the second branch module) and branch 3 module (as an example of the third branch module). Each branch module includes a feature dimensionality reduction module, a global average pooling and an Exp activation module. The feature dimensionality reduction modules included in each branch module may be the same or different, and the global average pooling and Exp activation modules included in each branch module may be the same or different. Among them, after the feature dimensionality reduction module performs dimensionality reduction processing on the spectral features, it is input into the global average pooling and Exp activation modules for global pooling processing and activation processing.

[0171] Furthermore, in this embodiment, the output of the global average pooling and Exp activation module in the branch 1 module is the RAW response of the light source power spectrum, for example, L∈R N .

[0172] In this embodiment, the branch 2 module may further include a scale scaling module. The output of the global average pooling and Exp activation module in the branch 2 module is input to the scale scaling module for scale scaling processing to obtain a color subspace projection matrix, for example, T1∈R N×3 .

[0173] In this embodiment, the branch 3 module may further include a scale scaling and normalization module. The output of the global average pooling and Exp activation module in the branch 3 module is input to the scale scaling and normalization module for scale scaling and normalization processing to obtain a subspace color correction matrix, for example, T2∈R 3×3 Or T2∈R 6×3 .

[0174] In this embodiment, each branch module may not include the feature dimension reduction module, that is, the feature dimension reduction module is optional. In addition, each branch module may also include other modules, which can be selected and configured as needed.

[0175] In one embodiment of the present application, the number of channels corresponding to the spectral feature may be greater than the number of channels of the multispectral RAW image 101. For example, the number of channels of the multispectral RAW image 101 may be 9, 16, 25, etc., and the number of channels corresponding to the spectral feature may be 32, 64, 128, etc. Of course, in other embodiments of the present application, the number of channels corresponding to the spectral feature and the number of channels of the multispectral RAW image 101 may also be other values, which can be set as needed.

[0176] For example, in one embodiment of the present application, the light source estimation and color adaptation model can be a local light source estimation and color adaptation model. The structure of the local light source estimation and color adaptation model is shown in FIG11. Compared with the global light source estimation and color adaptation model shown in FIG10, the local light source estimation and color adaptation model includes an Exp activation module, that is, only Exp activation processing is performed on the features after dimensionality reduction, and no global pooling processing is performed. In addition, each branch module also includes a reverse pooling (or replication) module.

[0177] In this embodiment, the output of the Exp activation module in the branch 1 module is the RAW response of the light source power spectrum, for example, L∈R h′×w′×N Then, the RAW response L∈R of the light source power spectrum is h′×w′×N , input to the reverse pooling (or copying) module, perform reverse pooling processing, and obtain the final RAW response L∈R of the light source power spectrum h×w×N .

[0178] In this embodiment, the color subspace projection matrix obtained by the branch 2 module scaling module is, for example, T1∈R h′×w′×N×3 Afterwards, the color subspace projection matrix T1∈R h′×w′×N×3 , input to the reverse pooling (or copying) module, perform reverse pooling processing, and obtain the final color subspace projection matrix T1∈R h×w×N×3 .

[0179] In this embodiment, the branch 3 module scale scaling and normalization module obtains the subspace color correction matrix, for example, T2∈R h′×w′×M×3 Afterwards, the subspace color correction matrix T2∈R h′×w′×M×3 , input to the reverse pooling (or copying) module, perform reverse pooling processing, and obtain the final subspace color correction matrix T2∈R h×w×M×3 .

[0180] In this embodiment, each branch module may not include the feature dimension reduction module, that is, the feature dimension reduction module is optional. In addition, each branch module may also include other modules, which can be selected and configured as needed.

[0181] This implementation method uses the source estimation and color adaptive model structure of the backbone network + multi-branch structure. Due to the differentiated feature processing flow in the multi-branch, multiple different image adjustment parameters can be obtained, so that the image color restoration processing can be performed conveniently and accurately, which can improve the image color restoration accuracy and obtain color images with better color restoration, effectively improving the image quality and thus improving the user experience.

[0182] Furthermore, as shown in FIG12 and FIG13, corresponding to FIG9, in one embodiment of the present application, the matrix module performs color restoration processing on the multispectral RAW image 101 according to the image adjustment parameters to obtain a chromatic adaptation image, including the following process: according to the color subspace projection matrix T N×3 , i.e., the three-dimensional color subspace projection matrix, projects the h×w×N-dimensional multispectral RAW image 101 into the h×w×3-dimensional color subspace, adapting the data dimensions required by the three primary colors (R / G / B) display device (i.e., electronic device), or performing dimensionality reduction processing, to obtain a subspace response image. Then, according to the color subspace projection matrix T N×3 The white balance diagonal matrix is ​​obtained by combining the RAW response of the projected light source and the power spectrum of the light source. The subspace response image is subjected to white balance (WB) processing or automatic white balance (AWB) processing according to the white balance diagonal matrix to obtain a white balanced image. That is, the RAW domain response of the projected light source is aligned to the neutral color (R=G=B) response of the display device, decoupling the colors of different light sources to achieve color consistency. Then, in the projected 3-channel color subspace, according to the normalized subspace color correction matrix T M×3 , that is, the three-dimensional color subspace correction matrix, performs color correction (Color Calibration) on the white balance image to achieve light source related color correction and obtain a chromatic adaptation image.

[0183] The basic concept of white balance is to restore white objects to white regardless of the light source. The color cast that occurs when shooting under a specific light source is compensated by strengthening the corresponding complementary color.

[0184] The purpose of color correction is to ensure that the colors of the image can be accurately reproduced as seen by the human eye at the scene of the shooting. Color correction technology is the key to image color restoration.

[0185] Furthermore, as shown in FIG13 , in one embodiment of the present application, the multispectral RAW image 101 may first pass through the pre-processing module 201 and then be input into the parameter-learnable image processing module 301 .

[0186] As shown in FIG13 , in one embodiment of the present application, the multispectral RAW response is obtained by a multispectral image sensor based on a filter array (or filter coating), and a cyclic unit is composed of nine pixels of 3×3, that is, the spectral resolution is 9. Moreover, the color subspace projection matrix T N×3 For example, it can be a 9×3 dimensional matrix, such as The RAW response L of the light source power spectrum can be, for example, a 9-dimensional vector, such as [0.564, 0.362, ..., ..., 0.498, 0.548], and the subspace color correction matrix T M×3 For example, it can be a 3×3 dimensional matrix, such as The obtained white balance diagonal matrix can be, for example, a 3×3 dimensional matrix, such as

[0187] Of course, in other embodiments of the present application, the color subspace projection matrix, the RAW response of the light source power spectrum, and the subspace color correction matrix may also be in other formats or values, which can be selected and set as needed. Furthermore, the RAW response of the light source power spectrum can be understood as the output vector of the light source estimation.

[0188] The above matrix module can be understood as an abstract expression without considering the specific physical meanings of the above color subspace projection matrix, white balance diagonal matrix and subspace color correction matrix.

[0189] The present embodiment is a modular image color restoration solution based on a multispectral RAW image 101. The image processing method provided by the present embodiment has a better effect on the color image than some image processing methods in the prior art. For example, the image effect can be reflected by the color difference (Color Difference) of the images obtained under different light sources. Color difference, also known as color distance, is a focus in color science. It quantifies a concept. Color difference can be simply calculated by the Euclidean distance in the color space (such as Angular Error (AE)), or it can be calculated using the International Commission on Illumination's more complex and uniform human perception formula (such as Delta E (Empfindung, dE), dE is a measure that describes the difference between two colors.

[0190] For example, the AE and dE (e.g., dE 2000) of a color image, as well as their corresponding average values, are used as evaluation metrics. Light sources may include, for example, fluorescent tube lamps (CFLs), incandescent lamps (INCs), light-emitting diodes (LEDs), and sunlight (SUNs).

[0191] Exemplarily, as shown in FIG14 , a comparison method of image processing effects is shown. As can be seen from FIG14 , in the solution of the present application, the average value (Average) of the AE of the color image obtained is reduced compared to the average value of the AE in the prior art method. In some scenarios, it can be reduced by more than 50%. In addition, the average value of the dE of the color image is reduced compared to the average value of the dE in the prior art. In some scenarios, it can also be reduced by more than 50%. That is, the electronic device can perform color restoration processing on the multispectral RAW image based on the parameter-learnable image processing model, which can improve the accuracy of image color restoration, obtain a color image with better color restoration, effectively improve the image quality, and thus improve the user experience.

[0192] In the embodiments of this application, in some scenarios, the effects of local illuminant estimation and global illuminant estimation are essentially the same. However, in some typical scenarios, local illuminant estimation provides better results than global illuminant estimation because the image adjustment parameters obtained by local illuminant estimation have higher resolution than those obtained by global illuminant estimation. This means that local illuminant estimation can perform different processing for different areas compared to global illuminant estimation. This is especially true in challenging scenarios such as low color temperature and mixed color temperature.

[0193] For example, please refer to Figure 15, which shows some color images obtained based on global light source estimation and local light source estimation. Among them, in outdoor sunlight scenes, the color saturation of the color image obtained based on local light source estimation is better than the color saturation of the color image obtained based on global light source estimation. In low color temperature difficult scenes, the color image obtained based on global light source estimation is yellowish, and the color of the color image obtained based on local light source estimation is closer to the color seen by the human eye. Therefore, the white balance of local light source estimation is more accurate than that of global light source estimation. In mixed color temperature difficult scenes, the color image obtained based on global light source estimation is overall blue, and the color of the color image obtained based on local light source estimation is closer to the color seen by the human eye. Therefore, the color restoration of local light source estimation is more accurate than that of global light source estimation.

[0194] Furthermore, in a large-area monochrome background scene, for example, placing a white ball on a green background, if the estimation accuracy of the light source is insufficient, the white ball will appear green or yellow. The image processing method provided in the embodiment of the present application significantly improves the light source estimation accuracy of such difficult scenes, making the color restoration of neutral colors (R=G=B) more accurate.

[0195] In low color temperature (warm yellow) lighting scenes, insufficient light source estimation accuracy can cause the overall image color to appear yellowish (for example, white objects appear yellowish). The image processing method provided by the embodiments of this application significantly improves the light source estimation accuracy in low color temperature scenes, making neutral color (R=G=B) reproduction more accurate.

[0196] In scenes with mixed color temperatures and shadows, if the light source estimation accuracy is insufficient, the overall image color will be bluish. However, the image processing method provided by the embodiments of the present application significantly improves the light source estimation accuracy in such scenes, making the color reproduction of neutral colors (R=G=B) more accurate.

[0197] In scenes with mixed color temperatures and shadows, if the light source estimation accuracy is insufficient, the overall color will be bluish. However, the image processing method provided by the embodiments of the present application significantly improves the light source estimation accuracy in such scenes, making the color reproduction of neutral colors (R=G=B) more accurate.

[0198] In scenes with a solid-color background and a dark-skinned face, there's a problem with consistently capturing the darker face as a true black, resulting in poor color consistency. However, the image processing method provided in the present application can accurately restore the color of dark-skinned faces against a variety of solid-color backgrounds, achieving better color consistency.

[0199] In addition, the image processing method provided in the embodiment of the present application can accurately restore the image color under different light source conditions and has better color consistency.

[0200] In summary, in the embodiments of the present application, light source estimation is implemented in the multispectral RAW domain space. On the one hand, it can alleviate the problem of heterochromatic color and improve the accuracy of light source estimation. On the other hand, adaptive color correction related to the light source is implemented through matrices T1 and T2, which improves the accuracy of color reproduction and obtains a color image with better color reproduction. It also effectively improves the consistency of image color under different light sources, effectively improves image quality, and thus improves user experience.

[0201] Furthermore, the image processing method provided by the embodiments of this application can achieve end-to-end processing of multispectral RAW images into color images (e.g., RGB images), reducing pipeline complexity. It enables end-to-end joint optimization between modules, leveraging spectral information to mitigate metamerism and improve color discrimination performance. Furthermore, based on light source estimation and color adaptation models, it significantly improves color reproduction accuracy. It can also improve light source estimation accuracy and color reproduction accuracy in difficult scenarios, and improves color reproduction accuracy by implementing light source-dependent adaptive color correction.

[0202] The structure of the image processing system and the image processing process related to obtaining the high-resolution color image 402 based on the multispectral RAW image 101 and the high-resolution color (or grayscale) RAW image 102 will be further described below.

[0203] In another embodiment of the present application, as shown in Figure 16, the image processing system includes a preprocessing module 202, a parameter-learnable image processing module 301 and a stylized post-processing module, wherein the parameter-learnable image processing module 301 includes a light source estimation and color adaptation model (as an example of a parameter-learnable image processing model) module and a matrix module.

[0204] The light source estimation and color adaptation model module is used to obtain the multi-spectral RAW image 101 (e.g. X2∈R h×w×N ) to obtain image adjustment parameters (as an example of first image adjustment information corresponding to color restoration processing of the high-resolution color RAW image 102, or as an example of second image adjustment information). The image adjustment parameters are input into the matrix module. The image adjustment parameters may be, for example, at least one of white balance parameters and a subspace color correction matrix. The white balance parameters may be the aforementioned white balance diagonal matrix obtained based on the color subspace projection moment and the RAW response of the light source power spectrum, or other parameters that can be set as needed. Of course, the image adjustment parameters may also be other parameters that can be selected and set as needed.

[0205] The matrix module is used to adjust the high-resolution color RAW image 102 (e.g. X1∈R H×W×3 ) is subjected to color restoration processing to obtain a high-resolution color image 402' (as another example of the second image, or as an example of the second target image) as an intermediate image processing result, and then the high-resolution color image 402' is input into the stylization post-processing module for further stylization processing to obtain a high-resolution color image 402 (as another example of the fourth image, or as another example of the second target image). The high-resolution color image 402 can be, for example, Y∈R H×W×3 .

[0206] Furthermore, as shown in FIG17 , in one embodiment of the present application, the image adjustment parameter may be, for example, a white balance parameter T1∈R 3 and subspace color correction matrix T2∈R 3×3 The matrix module performs color restoration processing on the high-resolution color RAW image 102 according to the image adjustment parameters to obtain a high-resolution color image 402', including the following process: according to the white balance parameter T1∈R 3 , perform white balance processing on the high-resolution color RAW image 102 to obtain a white balanced image. Then, according to the subspace color correction matrix T2∈R 3×3 Color correction is performed on the white-balanced image to achieve light source-dependent color correction and obtain a chromatically adapted image. The white balance parameters may include, for example, a white balance diagonal matrix, which is obtained based on the color subspace projection matrix and the original domain response of the light source power spectrum.

[0207] In this embodiment, the white balance and color correction function in the same manner as described above, or they may be different, and they may be selected and set as needed.

[0208] This embodiment is a modular image color restoration solution based on a multispectral RAW image 101 and a high-resolution color RAW image 102. Based on this embodiment, the electronic device can perform color restoration processing on the high-resolution color RAW image 102 based on a parameter-learnable image processing model and the multispectral RAW image 101, thereby improving the accuracy of image color restoration and obtaining a color image with better color restoration, effectively improving image quality and thereby enhancing the user experience.

[0209] In addition, the white balance parameter T1 and color correction matrix T2 are obtained through light source estimation and color adaptation model, and are applied to the 3-channel high-resolution color RAW (or preprocessed RAW) image respectively. While maintaining the H×W high resolution, the color reproduction quality of the image can be improved.

[0210] In another embodiment of the present application, as shown in FIG18 , a parameter-learnable image processing module 301 includes a light source estimation and color adaptation model module and a color migration module. The light source estimation and color adaptation model module is used to extract spectral features from the multispectral RAW image 101, obtain image adjustment parameters (as another example of first image adjustment information corresponding to color restoration processing of the high-resolution color RAW image 102, or as another example of second image adjustment information), and input the image adjustment parameters into the color migration module. The image adjustment parameters can be, for example, a color image A (as an example of a third image, the color image A is determined based on the multispectral RAW image 101), and the resolution of the color image A is less than the resolution of the high-resolution color (or grayscale) RAW image 102. Of course, the image adjustment parameters can also be other parameters, which can be selected and set as needed.

[0211] The color migration module is used to transfer the high-resolution color (or grayscale) RAW image 102 (eg X1∈R H×W×C ) performs color restoration processing, that is, performs color migration processing, to obtain a high-resolution color image 402' (as another example of the second image, or as another example of the second target image), and then inputs the high-resolution color image 402' into the stylization post-processing module for further stylization processing to obtain a high-resolution color image 402 (as another example of the fourth image, or as another example of the second target image).

[0212] The color migration module can be implemented by a classic image filtering algorithm (such as a bilateral filtering algorithm) or by one (or more) neural networks.

[0213] This embodiment is a modular image color restoration solution based on a multispectral RAW image 101 and a high-resolution color (or grayscale) RAW image 102. Based on this embodiment, the color image A obtained from the multispectral RAW image 101 based on the light source estimation and color adaptive model module is used to perform color restoration processing on the high-resolution color (or grayscale) RAW image 102, which can improve the color restoration accuracy and obtain a color image with better color restoration, effectively improving the image quality and thereby improving the user experience.

[0214] In some scenarios, users may have different image requirements depending on the image usage scenario. Based on this, in another embodiment of the present application, the image processing system may further perform processing such as image fusion on the aforementioned color image 401 to obtain a further processed image to meet user needs.

[0215] For example, in one embodiment of the present application, as shown in FIG19 , further processing of color image 1 (e.g., color image 401, as an example of the third target image) may be performed as follows: the image processing system analyzes the spectral characteristics of each channel of multispectral RAW image 101 (as an example of the first initial image), divides multispectral RAW image 101 into a plurality of sub-regions (i.e., a plurality of sub-images) based on the spectral characteristics of each channel of multispectral RAW image 101, and performs different processing on different sub-regions. For example, the image processing system analyzes whether there are sub-channels in multispectral RAW image 101 that meet the target spectral graph characteristics. If so, the region corresponding to the sub-channel that meets the target spectral graph characteristics (i.e., the second region) is subjected to image adjustment processing (as an example of target processing) using a specified special processing method to obtain image 4 (as an example of the fifth target image). The region corresponding to the sub-channel that does not meet the target spectral graph characteristics (i.e., the first region) is subjected to corresponding processing (e.g., the processing process for obtaining color image 401 as an example of the first color restoration processing) based on the aforementioned parameter-learnable image processing module 301 to obtain color image 2 (as an example of the fourth target image). Then, image fusion processing is performed based on the color image 1, the image 4 and the color image 2 to obtain a combined color image 3 (as an example of the first target image).

[0216] In another embodiment of the present application, the image processing system may directly analyze the spectral characteristics of each channel of the multispectral RAW image 101 to obtain the aforementioned image 4 and color image 2, and perform image fusion processing based on image 4 and color image 2 to obtain a combined color image 3. Alternatively, the image processing system may directly analyze the spectral characteristics of each channel of the multispectral RAW image 101 to obtain the aforementioned color image 1, image 4, and color image 2, and perform image fusion processing based on color image 1, image 4, and color image 2 to obtain a combined color image 3.

[0217] That is, the multispectral RAW image 101 can be divided into several sub-regions according to the channel spectral response, and different processing is performed on different sub-regions to achieve different beneficial effects, and then image fusion is performed to obtain a complete color image.

[0218] In some embodiments of the present application, the aforementioned further processing of the color image 1 can be implemented by a target processing module included in the image processing system. The target processing module includes a region determination module, an image sub-region processing module, and an image combination module. The region determination module is used to determine sub-regions in the multispectral RAW image 101 that do and do not meet the target spectral characteristics. The image sub-region processing module is used to perform the aforementioned special processing on sub-regions that meet the target spectral characteristics and to perform the aforementioned processing based on the parameter-learnable image processing module 301 on sub-regions that do not meet the target spectral characteristics. The image combination module is used to perform image fusion processing.

[0219] In some embodiments of the present application, the above-mentioned target spectral characteristics may be spectral characteristics related to the spectral response of the image channel. The channel conforming to the target spectral characteristics may, for example, refer to the spectral response of one or more sub-channels in the image being greater than or equal to or less than a preset response threshold. For example, the target spectral characteristics are that the spectral response of the target sub-channel is greater than or equal to a preset first response threshold or less than a preset second response threshold. The first response threshold (for example, 50,000 lux) and the second response threshold (for example, 5 lux) can be set as needed. Of course, the target spectral characteristics may also be other mathematical relationships related to the channel spectral response.

[0220] In addition, the region determination module can obtain spectral feature information corresponding to the target spectral feature from the application programming interface (API) parameters corresponding to the multispectral image sensor (used to capture the multispectral RAW image 101) in the image acquisition system (for example, determine the target spectral feature and obtain the spectral feature information based on the "target spectral feature for sub-region screening" parameter in the API parameters), and determine whether there is a sub-channel in the multispectral RAW image 101 whose spectral feature meets the target spectral graph feature. The spectral feature information is a parameter corresponding to the target spectral feature for sub-region screening in the API parameters, such as a channel response value. Based on the spectral feature information and the target spectral feature, the sub-regions in the multispectral RAW image 101 that meet or do not meet the target spectral feature can be determined. For example, the image sub-region processing module can also obtain the "special image processing method" parameter corresponding to the target spectral feature from the API parameters to determine the aforementioned special processing method and perform corresponding processing.

[0221] In some embodiments of the present application, exemplary target spectral features, special processing methods corresponding to the target spectral features, and beneficial effects that can be achieved by the processed images can be shown in Table 1 below.

[0222] Table 1

[0223] Among them, the target spectral characteristics can be, for example, the super-dynamic range of the spectral response of any channel, the spectral response of the channel corresponding to the infrared band is too high, the spectral response of the channel corresponding to the ultraviolet band is too high, the spectral response of some channels is too high, the spectral response of some channels is too high or too low, etc. Among them, the dynamic range can be, for example, 5lux-50000lux. Of course, it can also be other values ​​set as needed. The channel spectral response is too high, for example, the channel spectral response is greater than the preset response threshold. The response threshold can be, for example, 50000lux. Of course, the response threshold can also be other values ​​set as needed. In addition, the response thresholds corresponding to different channels can be the same or different, and can be set as needed. The channel spectral response is too low, for example, the channel spectral response is less than the preset response threshold. The response threshold can be, for example, 5lux. Of course, the response threshold can also be other values ​​set as needed. In addition, the response thresholds corresponding to different channels can be the same or different, and can be set as needed.

[0224] In addition, the special processing corresponding to the above-mentioned spectral characteristics can be, for example, the area of ​​the super dynamic range of the Mask channel spectral response (this area is the overexposed area, as an example of the masking processing of the overexposed area image), the highlight corresponding area (as an example of the brightness enhancement processing of the target area image, where the enhanced brightness value can be set as needed), the grayscale corresponding area (as an example of the grayscale processing of the target area image), etc.

[0225] Furthermore, if the spectral response of any channel exceeds the dynamic range, the area of ​​the channel spectral response exceeding the dynamic range is masked. This avoids image color distortion in that area, thereby improving the accuracy of image color reproduction and obtaining a color image with better color reproduction, effectively improving image quality. Furthermore, when the obtained color image 3 is displayed, the user experience can be effectively enhanced.

[0226] If the spectral response of the channel corresponding to the infrared band is too high, it means that the temperature in the corresponding area is too high. Therefore, by highlighting the area where the spectral response of the channel corresponding to the infrared band is too high and displaying the resulting color image 3, the user can intuitively identify the highlighted area as a high-temperature danger area through the highlighted area on the color image 3, or the user can intuitively identify the highlighted area as a hidden camera, etc.

[0227] If the spectral response of the channel corresponding to the ultraviolet band is too high, it indicates that the corresponding area is a high ultraviolet (UV) region. Therefore, by highlighting the area where the spectral response of the channel corresponding to the ultraviolet band is too high and displaying the resulting color image 3, the user can intuitively identify the highlighted area as a high UV region through the highlighted area on the color image 3. The user can then avoid the area corresponding to the highlighted area or take sun protection measures to prevent sunburn.

[0228] Furthermore, optical identification of certain chemicals or materials can be achieved based on the spectral responses of certain channels. Therefore, by highlighting areas with excessively high spectral responses (which can be set as needed) and displaying the resulting color image 3, users can intuitively identify the corresponding chemical substances or materials through the highlighted areas on the color image 3. For example, this allows for the identification of chemicals or materials such as blood and leather.

[0229] Furthermore, based on the spectral response of certain channels (e.g., if the spectral response of certain channels (which can be set as needed) is too high or too low), grayscale processing can be performed on these areas to achieve an artistic filter effect. The resulting color image 3 is then displayed, allowing the user to see an image with an artistic filter effect through color image 3.

[0230] For example, in some scenarios, during the picture-taking process, there may be problems such as the channel spectral response being too high, resulting in overexposure of the image. In the case of overexposure, when the response of some channels exceeds the dynamic range of the sensor, the response will be truncated, which destroys the subspace projection relationship established by the linear matrix and causes color distortion in the overexposed area.

[0231] Based on this, in one embodiment of the present application, as shown in FIG20A , for the obtained color image 1 (such as the aforementioned color image 401 or the high-resolution color image 402 ), multi-spectral response truncation compensation can also be performed to obtain an overexposed color image 3.

[0232] As shown in FIG20A , in one embodiment of the present application, the region determination module determines overexposed and non-overexposed regions in the multispectral RAW image 101. For example, if the response of a channel exceeds a preset threshold, the region is considered overexposed; otherwise, the region is considered non-overexposed. The image region adjustment module processes the overexposed regions by masking the overexposed regions, for example, to obtain an adjusted image corresponding to the overexposed regions (as an example of masking information for the overexposed regions). Masking the overexposed regions can be achieved, for example, through binarization. The image region adjustment module inputs the image corresponding to the non-overexposed sub-channel spectral response into the parameter-learnable image processing module 301, which performs color restoration processing, such as the aforementioned processing (as an example of the first or third color restoration processing), to obtain a color image 2 (as an example of a sixth image). The image combination module then performs image fusion processing based on the color image 1, the overexposed region adjustment image, and the color image 2 (the overexposed region adjustment image and the color image 2 are examples of the second image adjustment information corresponding to the color restoration processing performed on the multispectral RAW image 101), to obtain a combined color image 3.

[0233] Based on this, in another embodiment of the present application, as shown in Figure 20A, the image processing system also includes an overexposure processing module for processing the obtained color image 1 (such as the aforementioned color image 401 or the high-resolution color image 402), compensating for the multi-spectral response truncation, and obtaining the overexposed color image 3.

[0234] As shown in FIG20A , in one embodiment of the present application, the overexposure processing module includes an overexposed region determination module, an overexposed region adjustment module, a non-overexposed channel determination module, a parameter-learnable image processing module 301, and an image combination module. The overexposed region determination module is used to determine the region in the multispectral RAW image 101 that responds to overexposure. For example, if the response of a channel is greater than a preset response threshold, it is considered an overexposed region. The overexposed region adjustment module is used to process the overexposed region by masking the overexposed region, etc., to obtain a corresponding overexposed region adjusted image (as an example of image overexposed region masking information). The overexposed region masking can be achieved, for example, by binarization processing. The non-overexposed channel determination module is used to determine the corresponding image of the non-overexposed sub-channel spectral response, input it into the parameter-learnable image processing module 301, and perform, for example, the aforementioned color restoration processing (as an example of the third color restoration processing) to obtain a color image 2 (as an example of the sixth image). The image combination module then performs image fusion processing based on the color image 1, the overexposed area adjustment image and the color image 2 (the overexposed area adjustment image and the color image 2 are examples of the second image adjustment information corresponding to the color restoration processing of the multispectral RAW image 101), obtains the combined color image 3 (as an example of the fifth image) and outputs it.

[0235] As shown in Figure 20B, the image on the left is the image without the response truncation compensation scheme (i.e., masking the overexposed areas), while the image on the right is the image after the response truncation compensation scheme is applied. Therefore, after the response truncation compensation scheme is applied, color artifacts in the overexposed areas are eliminated. As a result, a color image with better color reproduction is obtained, effectively improving image quality and, in turn, enhancing the user experience.

[0236] In other implementations of the present application, the color image 2 may be obtained according to the spectral response of the sub-channel that is not overexposed by other methods such as 3-channel color restoration (such as AWB and color correction).

[0237] Furthermore, in some implementations of the present application, the above-mentioned target spectral characteristics can be pre-set in the operating system of the electronic device that executes the image processing method provided by the present application. During the image processing process, the electronic device can perform the aforementioned processing based on the target spectral characteristics preset by the system.

[0238] For example, taking the application of the image processing method to a mobile phone as an example, in a photo-taking scenario, the mobile phone can automatically determine whether there is an area in the captured image that meets the above-mentioned target spectral characteristics based on the target spectral characteristics preset by the mobile phone system, and perform the above-mentioned corresponding processing on the captured image, and store or display the final color image 3 for the user to view.

[0239] In other implementations of the present application, the target spectral characteristics may also be determined by an electronic device executing the image processing method provided herein based on a user operation, such as a user selection operation on the target spectral characteristics. Specifically, the electronic device displays the target spectral characteristics or identification information corresponding to the target spectral characteristics on its display interface for user selection, and the electronic device determines the target spectral characteristics based on the user selection operation and performs corresponding processing.

[0240] Exemplarily, the image processing method can be applied in a mobile phone, for example, in a mobile phone photo taking scenario or a mobile phone album image processing scenario.

[0241] For example, in a scenario where a user opens a mobile phone's camera app to take a photo, as shown in FIG20C(a), the mobile phone displays a camera interface, which includes a "More" control. If the mobile phone receives a user click operation on the "More" control, the mobile phone displays a display interface as shown in FIG20C(b) (as a type of third interface), which includes controls corresponding to target spectral characteristics, such as an "overexposure processing" control, a "high temperature / hidden camera recognition" control, a "high UV area recognition" control, a "**chemical substance / material (e.g., blood, leather, etc.) recognition" control, an "artistic filter" control, and other setting controls (as some examples of selection controls for image usage scenarios). As shown in Table 2 below, the "overexposure processing" control, the "high temperature / hidden camera recognition" control, the "high UV area recognition" control, the "**chemical substance / material recognition" control, and the "artistic filter" control each correspond to a different target spectral characteristic.

[0242] Table 2

[0243] For example, as shown in FIG20C(c), if the mobile phone receives a user click on the "High Temperature / Hidden Camera Identification" control (as an example of a user selecting an image usage scenario, i.e., as an example of a second user operation), the mobile phone determines that the image usage scenario is "High Temperature / Hidden Camera Identification." Based on the correspondence between this image usage scenario and the preset image usage scenarios and target spectral characteristics shown in Table 1, the mobile phone determines that the target spectral characteristic is "Excessive spectral response of the corresponding channel in the infrared band." As shown in FIG20C(c), if the mobile phone receives a user photo operation, it activates the multispectral image sensor to capture an image. After obtaining, for example, the aforementioned multispectral RAW image 101, the mobile phone further determines whether the correspondence conforms to the target spectral characteristics and performs corresponding processing as shown in Table 2, resulting in a color image 3 for display. Color image 3 is shown in FIG20C(d), where region A of color image 3 is highlighted, allowing the user to know that region A is a high temperature area or an area where a hidden camera is located. In this way, the user can intuitively identify the highlighted area on the color image 3 as a high-temperature dangerous area, or the user can intuitively identify the highlighted area on the color image 3 as a hidden camera.

[0244] For example, in a photo-taking scenario, the mobile phone can also display the display interface shown in FIG20D(a) based on user operation, which includes a "Settings" control. If the mobile phone receives a user click operation on the "Settings" control, the mobile phone displays the display interface shown in FIG20D(b) (as a second interface), which includes setting controls for target spectral characteristics. For example, the display interface includes two setting items: "The spectral response of the channel corresponding to the infrared band is greater than the corresponding spectral response threshold" and "The spectral response of the channel corresponding to the ultraviolet band is greater than the corresponding spectral response threshold." The user can select the target spectral characteristic through the selection control B displayed on the display interface. For example, by clicking the selection control B corresponding to "The spectral response of the channel corresponding to the infrared band is greater than the corresponding spectral response threshold," the user selects "The spectral response of the channel corresponding to the infrared band is greater than the corresponding spectral response threshold" as the target spectral characteristic (as another example of a second user operation). Furthermore, after selecting the target spectral characteristic, the user can also set the corresponding spectral response threshold. For example, the user can set the spectral response threshold by dragging the corresponding setting bar, for example, setting the spectral response threshold to 20600 lux (as another example of a second user operation). The phone determines the final target spectral signature based on the user-set spectral response threshold for the corresponding infrared channel. This target spectral signature is defined as a spectral response greater than the spectral response threshold of 20,600 lux. Then, as shown in Figures 20D(c) and 20D(d), upon receiving the user's request to return to the photo interface, the phone takes a photo and displays the corresponding image. If a camera is recognized, the camera's location is highlighted in the photo displayed by the phone to alert the user.

[0245] The selection and processing of other controls are similar to those of the "High Temperature / Hidden Camera Identification" control and will not be described here.

[0246] Of course, in other implementations of the present application, the mobile phone may also display other forms of display interfaces so that the user can set the target spectral characteristics, which can be set as needed.

[0247] For example, after a user opens the mobile phone's photo album app and opens a photo, as shown in FIG20E(a), the phone displays an image (e.g., the aforementioned color image 401, as another example of the first initial image). This display interface includes a "More" control. If the phone receives a user click on the "More" control, the phone displays the display interface shown in FIG20E(b) (as another example of the third interface), which includes the aforementioned settings controls such as the "Overexposure Processing" control, the "High Temperature / Hidden Camera Recognition" control, the "High UV Area Recognition" control, the "**Chemical Substance / Material (e.g., blood, leather, etc.) Recognition" control, and the "Artistic Filter" control. As shown in FIG20E(b), if the phone receives a user click on the "High Temperature / Hidden Camera Recognition" control (as another example of a second user operation), the phone further determines and processes the corresponding target spectral features shown in Table 2, obtaining a color image 3 for display. The color image 3 is shown in FIG20E(c), wherein area A in the color image 3 is highlighted, and the user can know that area A is a high-temperature area or there is a hidden camera.

[0248] The selection and processing of other controls are similar to those of the "High Temperature / Hidden Camera Identification" control and will not be described here.

[0249] In addition, electronic devices such as mobile phones can also perform further image processing based on the above processing in other image processing scenarios such as video besides the above-mentioned photography and photo processing to obtain images that better meet user needs.

[0250] In summary, the image processing method provided by the implementation of the present application can further process the obtained multispectral RAW image 101, color image 401, etc. according to different scenarios. That is, the mobile phone can display the aforementioned multiple controls corresponding to the target spectral characteristics according to user operations as identification information corresponding to different user needs for images or different image usage scenarios, so that the user can more conveniently and accurately select or input the user's needs for image characteristics or image usage scenarios. In this way, the mobile phone can conveniently and accurately determine the user's needs for image characteristics, thereby obtaining an image that better meets the user's needs, thereby improving the user experience.

[0251] Furthermore, in some other implementations of the present application, the mobile phone can also simultaneously display the original captured image and the processed image. For example, the mobile phone displays the image shown in Figure 20E(a) and the image shown in Figure 20E(c) on the main screen and the secondary screen of the mobile phone, respectively, or the mobile phone displays the image shown in Figure 20E(a) and the image shown in Figure 20E(c) on the main screen of the mobile phone in picture-in-picture mode.

[0252] Of course, the phone can also display controls related to image display modes, such as "split screen display" and "floating display," so that users can select the image display mode. In this way, the phone can easily and accurately determine the user's requirements for image display mode and display the image based on the user's selected display mode, which can better meet user needs and enhance the user experience.

[0253] Of course, the user's requirements for images may also include other requirements besides the above requirements for image features and image display methods.

[0254] In some scenarios, the process of taking pictures may cause unrealistic colors. Based on this, in one embodiment of the present application, as shown in FIG21A , the image processing system further includes a brightness adjustment module for performing brightness adjustment processing on the obtained color image (e.g., color image 401 or high-resolution color image 402) to obtain a color image after brightness adjustment processing.

[0255] As shown in FIG21A , in one embodiment of the present application, the brightness adjustment module includes a subchannel determination module, a brightness estimation module, and a brightness migration module. The subchannel determination module is used to select a single (or multiple) channel spectral response (i.e., a target subchannel or a second subchannel) based on the response characteristics of the multispectral RAW image 101 according to preset channel selection rules for image brightness estimation. The determined subchannel is then input into the brightness estimation module to obtain the estimated brightness as image brightness information (as another example of second image adjustment information corresponding to color restoration processing of the multispectral RAW image 101). Various brightness estimation methods can be used, with a common method using the mean of each pixel in the image as the brightness estimate. The brightness migration module then adjusts the brightness of the color image according to the brightness migration module to obtain a final color image. For example, the estimated result is multiplied (or divided) by the color image (as an example of a third target image, i.e., the estimated brightness is multiplied or divided by the brightness of the color image to obtain the adjusted image brightness), thereby obtaining an output color image (as another example of the fifth image, or as an example of the first target image).

[0256] As shown in Figure 21B, after brightness shifting, the response in overexposed areas of the color image approaches the maximum value of an 8-bit color image, resulting in a more balanced brightness distribution and improved visual quality. This results in a color image with improved color reproduction, effectively improving image quality and, in turn, the user experience.

[0257] In addition, in another implementation of the present application, the selected sub-channel response can also be first subjected to color restoration processing by an independent parameter-learnable image processing module to obtain a color-restored image, and then the image brightness estimation is performed based on the color-restored image. Alternatively, a color-restored image can be obtained through other methods such as 3-channel color restoration (such as AWB, CC), and then the image brightness estimation is performed based on the color-restored image.

[0258] In some other implementation schemes of the present application, after obtaining the target image (the target image is, for example, the aforementioned color image 401 and high-resolution color image 402), the image processing method also includes determining a display mode of the target image based on the target information, and displaying the target image according to the display mode.

[0259] The target information is at least one of information related to the image usage scenario corresponding to the image usage scenario, device information of the image display electronic device, and user's image processing requirement information for the image. Of course, the target information can also be other information, which can be selected and set as needed.

[0260] For example, the image usage scenario-related information may include, for example, the image usage scenario, and information related to the usage scenario regarding image resolution or clarity requirements, color accuracy requirements, and ambient light intensity information in the usage scenario. For example, the device information of the image display electronic device may include the number of screens of the aforementioned mobile phone, such as whether the mobile phone has only one display screen or includes two display screens, such as a primary screen and a secondary screen. For example, the user's image processing requirement information may include requirements for image usage scenarios, such as photo shooting scenes, photo shooting modes, and photo shooting requirements.

[0261] In some implementations of the present application, a mobile phone can obtain corresponding scene information, screen quantity information, user image processing requirement information for the image, and other target information from the API parameters corresponding to the image sensor that captures the multispectral RAW image 101 and the high-resolution color / grayscale RAW image in the image acquisition system.

[0262] Furthermore, the display method includes displaying the target image on a display screen such as a main screen, a secondary screen, etc. of an electronic device corresponding to the target image.

[0263] As shown in Figure 22A, the mobile phone can determine the display mode (for example, display on the main screen or the secondary screen) of the aforementioned color image 401 and the high-resolution color image 402 by automatic scene judgment or user selection based on the image usage scenario or the user's user demand information for the image.

[0264] For example, taking the example of a mobile phone determining the display mode of the aforementioned color image 401 and the high-resolution color image 402 obtained by taking a photo with the mobile phone, the mobile phone can determine the display mode of the color image 401 and the high-resolution color image 402 based on the image usage scenario and the number of screens of the mobile phone.

[0265] If the mobile phone includes two display screens, a main screen and a secondary screen, the mobile phone determines the image display mode according to the image usage scenario and the number of screens, which can be shown in Table 3 below.

[0266] Table 3

[0267] Among them, if the mobile phone determines that the image usage scenario has requirements for the spatial resolution or clarity of the image, such as the usage scenario is a daily photo or video shooting scenario, the mobile phone determines that the display method of the corresponding image under the usage scenario is: sending the high-resolution color image 402 to the main screen for display, and the color image 401 is not sent for display.

[0268] If the mobile phone determines that the image usage scenario has requirements for color accuracy, for example, the image usage scenario is a scenario of taking pictures of lipsticks of different colors, the mobile phone determines that the display method of the corresponding image in this scenario is: sending the color image 401 to the main screen for display after undergoing spatial resolution enhancement processing (such as interpolation, super-resolution reconstruction processing, etc.), or sending the color image 401 to the secondary screen for display, and the high-resolution color image 402 is not sent for display.

[0269] If the mobile phone determines that the image usage scenario is a special scenario, such as remote purchasing or live streaming, the mobile phone determines the display method for the corresponding image in this scenario: sending high-resolution color image 402 to the main screen for display, and simultaneously sending color image 401 to the secondary screen for display. Alternatively, high-resolution color image 402 and color image 401 are simultaneously sent to the main screen for display in a picture-in-picture format, wherein high-resolution color image 402 is above color image 401, or color image 401 is above high-resolution color image 402.

[0270] If the mobile phone determines that the ambient light of the image usage scenario is weak, such as night scenes, museums and other indoor dark scenes, the display method of the corresponding image in the scenario is determined as follows: the color image 401 is sent to the main screen for display after undergoing, for example, spatial resolution enhancement processing (such as interpolation, super-resolution reconstruction, etc.), or the color image 401 is sent to the secondary screen for display, and the high-resolution color image 402 is not displayed.

[0271] Furthermore, when the mobile phone is taking a photo, if it is determined that the ambient light of the image usage scene is weak, the mobile phone can also use infrared fill light to assist in imaging, so as to capture the corresponding image.

[0272] If the mobile phone has only one display screen, the mobile phone determines the image display mode according to the image usage scenario and the number of screens, as shown in Table 4 below.

[0273] Table 4

[0274] In some other implementation schemes of the present application, after obtaining the target image (such as the aforementioned color image 401 and the high-resolution color image 402), the image processing method also includes determining the display mode of the target image based on the user's selection operation of the image display mode (as an example of user needs), and displaying the target image according to the display mode.

[0275] Exemplarily, the image processing method can be applied in a mobile phone, for example, in a mobile phone photo taking scene or a mobile phone photo album scene.

[0276] For example, in a scenario where a user opens a camera app on a mobile phone to take a photo, as shown in FIG22B(a), the mobile phone displays a camera interface, which includes a "more" control. If the mobile phone receives a click operation on the "more" control from the user, the mobile phone displays a display interface as shown in FIG22B(b) (as another example of the fourth interface), which includes a "high spatial resolution / clarity" control, a "high color accuracy" control, a "low-light environment" control, etc. (as other examples of selection controls for image usage scenarios). If the mobile phone receives a click operation on a corresponding control from the user (as an example of a third user operation), the mobile phone determines the display mode of the corresponding image and displays the image according to the display mode.

[0277] For example, if the mobile phone includes two display screens, a main screen and a secondary screen, the mobile phone determines the display mode of the aforementioned color image 401 and the high-resolution color image 402 according to the user's selection operation of the image display mode, which can be shown in the following Table 5.

[0278] Table 5

[0279] For example, if the mobile phone receives a user click on the "High Spatial Resolution / Clearance" control, the mobile phone will send the captured high-resolution color image 402 to the main screen for display, and will not send the color image 401. The display methods corresponding to other controls are shown in Table 5 and are not repeated here.

[0280] In some implementations of the present application, after obtaining color image 401 and high-resolution color image 402, the mobile phone may also display color image 401 on the main screen of the mobile phone and high-resolution color image 402 on the secondary screen in a default manner, for example. Then, based on the user's drag operation on color image 401 and high-resolution color image 402, the mobile phone may display color image 401 on the secondary screen and high-resolution color image 402 on the main screen.

[0281] Alternatively, after obtaining the color image 401 and the high-resolution color image 402, the mobile phone can display reminder information (the reminder information, for example, includes the corresponding options of the controls shown in Figure 22B(b)) for the user to select, and process and display the image accordingly based on the user's selection.

[0282] Of course, in other implementations of the present application, the mobile phone can also determine the display method of the obtained image based on other methods according to the aforementioned target information to better meet user needs and enhance user experience.

[0283] In some other implementations of the present application, the image processing method also includes: the mobile phone determines whether it is necessary to capture only the multispectral RAW image 101 or to capture the multispectral RAW image 101 and the high-resolution color / grayscale RAW image 102 based on information related to the image usage scenario corresponding to the image usage scenario, screen information of the image display electronic device, user demand information of the user for the image, etc. (as some examples of target information), that is, determining whether to start the RGB sensor or the grayscale sensor, that is, determining whether the image input to the aforementioned image processing module (that is, the image processing system) is the multispectral RAW image 101 or the multispectral RAW image 101 and the high-resolution color / grayscale RAW image 102, and determining whether only the color image 401 or the high-resolution color image 402 is obtained after processing by the image processing module, or the color image 401 or the high-resolution color image 402 is obtained at the same time.

[0284] In some implementations of the present application, the target information can be determined based on the API parameters corresponding to the image sensor in the image acquisition system that captures the multispectral RAW image 101 and the high-resolution color / grayscale RAW image 102. The API parameters can be, for example, information indicating image resolution requirements, ambient light intensity information, screen number information of the display device, user operation information, etc.

[0285] Exemplarily, determining the image to be input into the aforementioned image processing module and determining the image obtained by processing the image processing module based on the target information can be: determining based on the target information whether to input the aforementioned multispectral RAW image 101 into the image processing model to obtain the color image 401, or to input the multispectral RAW image and the high-resolution color RAW image 102 into the image processing model at the same time to obtain the high-resolution image 402, or to input the multispectral RAW image and the high-resolution color RAW image 102 into the image processing model at the same time to obtain the color image 401 and the high-resolution image 402.

[0286] In some implementations of the present application, the image processing method can be applied to mobile phone photography scenarios.

[0287] In one implementation of the present application, the mobile phone can determine the image input to the image processing module and the image output by the image processing module based on the image resolution requirement information. Exemplarily, a preset image processing application (Application, APP, such as a camera application) or a mobile phone operating system (Operating System, OS) in the mobile phone calls the output image resolution indication information (used to indicate the image resolution of the final output image) output by the specified imaging module (such as the specified module in the camera application) in the corresponding API parameter in the image processing system in the mobile phone, determines the image resolution indication of the input image processing module, and selects whether to use the multispectral RAW image 101 as the input of the image processing module based on the image resolution indication. For example, if the image resolution indication is high, it is determined that the multispectral sensor for capturing the multispectral RAW image 101 and the RGB image sensor for capturing the high-resolution color RAW image 102 need to be started simultaneously to obtain the multispectral RAW image 101 and the high-resolution color RAW image 102, and the multispectral RAW image 101 and the high-resolution color RAW image 102 are input into the image processing module to obtain and output the high-resolution color image 402; if the image resolution indication is low, it is determined that only the multispectral sensor needs to be started, and the RGB image sensor does not need to be started, and only the captured multispectral RAW image 101 needs to be input into the image processing module to obtain and output the color image 401.

[0288] The image resolution corresponding to the image resolution indication may be determined based on an indication threshold. For example, if the input image resolution indication is greater than the indication threshold, the input image resolution indication is determined to be high. If the input image resolution indication is less than the indication threshold, the input image resolution indication is determined to be low. The indication threshold may be, for example, 1920×1080. Of course, the indication threshold may also be other values.

[0289] In one implementation of the present application, the mobile phone can determine the image input to the image processing module and the image output by the image processing module based on the ambient light intensity information. Exemplarily, after the image processing system in the mobile phone receives the API call request of the preset image processing application (such as a photo taking APP) or the mobile phone operating system, it identifies the ambient light intensity of the current scene and determines whether to start the RGB image sensor in combination with the ambient light intensity. If the ambient light intensity is weak (such as night scenes, museums and other indoor dark light scenes), it is determined not to start the RGB image sensor, only start the multispectral image sensor, and call the multispectral image sensor to shoot (for example, it can be taken by an infrared fill light) to obtain a multispectral RAW image 101, and the captured multispectral RAW image 101 is input into the image processing module to obtain and output a color image 401. If the ambient light intensity is strong (such as a strong light scene during the day), it is determined to start the RGB image sensor and the multispectral image sensor, and the RGB image sensor and the multispectral image sensor are called to shoot to obtain a high-resolution color RAW image 102 and a multispectral RAW image 101. The high-resolution color RAW image 102 and the multispectral RAW image 101 obtained by shooting are input into the image processing module to obtain and output a high-resolution color image 402 and a color image 401.

[0290] Furthermore, in this implementation, the mobile phone may also determine whether to further process the image (eg, color image 401 ) output by the image processing module according to the aforementioned target information.

[0291] For example, if the mobile phone determines that the ambient light intensity is weak and the color image 401 is obtained by shooting, it can further determine whether the obtained color image 401 needs to be further processed based on, for example, the aforementioned image resolution indication. For example, if the image resolution indication is low, it is determined that the color image 401 does not need to be further processed, and the output color image 401 can be directly used as the required image. If the input image resolution indication is high, it is determined that the color image 401 needs to be further processed, such as performing spatial resolution enhancement (such as interpolation, super-resolution reconstruction) (as another example of the first image adjustment processing) on ​​the obtained color image 401 to obtain a high-resolution color image 402 as the required image.

[0292] In other implementations of the present application, the mobile phone can also determine the image input to the image processing module, the image output by the image processing module, and the display mode of the image output by the image processing module based on the number of display screens of the mobile phone and the ambient light intensity information.

[0293] Exemplarily, the mobile phone calls the display device information and ambient light intensity information in the API parameters that indicate the multispectral imaging module outputs an image. If the display device information determines that the mobile phone has multiple screens and the ambient light intensity is strong, the mobile phone simultaneously activates the multispectral image sensor and the RGB image sensor to capture images, obtaining a high-resolution color RAW image 102 and a multispectral RAW image 101. The high-resolution color RAW image 102 and the multispectral RAW image 101 are then input into the image processing module, simultaneously generating color images 401 and 402. The high-resolution color image 402 is then displayed on the mobile phone's main screen, while the color image 401 is displayed on the mobile phone's secondary screen, or both the high-resolution color RAW image 102 and the multispectral RAW image 101 are simultaneously displayed on the mobile phone's main screen in picture-in-picture format. If the display device information determines that the mobile phone has multiple screens and the ambient light intensity is weak, the mobile phone only uses the multispectral sensor to capture images, obtaining the multispectral RAW image 101. The input multispectral RAW image 101 is then input into the image processing module, generating and outputting color image 401. Then, the color image 401 is sent to the main screen for display after undergoing spatial resolution enhancement (such as interpolation or super-resolution reconstruction), and 401 is sent to the secondary screen for display.

[0294] In other implementations of the present application, the mobile phone can also determine whether to activate, for example, the RGB sensor (as an example of determining whether to activate a second sensor) based on the user's selection or setting of an image usage scenario. For example, as shown in FIG. 22B(a) above, the mobile phone displays a camera interface, which includes a "More" control. If the mobile phone receives a user click on the "More" control, the mobile phone displays the display interface shown in FIG. 22B(b) (as an example of a first interface), which includes a "High Spatial Resolution / Clarity" control, a "High Color Accuracy" control, a "Low Light Environment" control, and the like (as other examples of selection controls for image usage scenarios). If the mobile phone receives a user click on a corresponding control (as an example of a first user operation), the mobile phone determines the image usage scenario selected by the user and determines whether to activate the RGB sensor based on the image usage scenario, the preset image usage scenario, and the correspondence between the preset image usage scenario and the information on whether to activate the RGB sensor. Furthermore, if the RGB sensor is determined to be activated, the RGB sensor is activated to capture an RGB image. Corresponding image adjustment processing is performed based on the captured RGB image and the multispectral image captured by the multispectral image sensor to obtain a target image for display. When it is determined that the RGB sensor is not started, corresponding image adjustment processing is performed according to the multispectral image collected by the multispectral image sensor to obtain a target image for display.

[0295] Of course, in other implementations of the present application, the mobile phone can also determine whether to start the RGB sensor or the grayscale sensor based on the aforementioned target information based on other methods, determine whether to input only the multispectral raw image 101 into the image processing module, or input the multispectral raw image 101 and the high-resolution color (or grayscale) raw image 102 into the image processing module together, determine whether to obtain only one of the color image 401 and the high-resolution color image 402 through the image processing module, or obtain the color image 401 and the high-resolution color image 402 at the same time, determine the further image adjustment processing method and display method of the mobile phone for the obtained color image 401 and the high-resolution color image 402, etc., so as to better meet user needs and enhance user experience.

[0296] In other implementations of the present application, the mobile phone may automatically determine the aforementioned image processing method based on at least one of the aforementioned information related to the image usage scenario and the number of screens on the mobile phone, among other device information. For example, upon receiving a user's camera operation, the mobile phone automatically obtains the aforementioned information related to the image usage scenario and the number of screens on the mobile phone, and automatically determines the aforementioned image processing method to perform the corresponding processing.

[0297] In other implementations of the present application, the mobile phone may also determine user demand information based on the aforementioned manual user operation, and determine the aforementioned image processing method based on the user demand information and perform corresponding processing. For example, the mobile phone receives a user click operation on the aforementioned control and a photo operation, determines the user demand information, determines the aforementioned image processing method based on the user demand information, and performs corresponding processing.

[0298] In addition, in the process of automatically judging or manually selecting the image processing method based on target information such as the image usage scenario in the above scenario, the calling relationship between the application and the corresponding algorithm module is involved. The algorithm module can provide an API interface to the corresponding application, and the application can obtain the corresponding API parameters based on the API interface.

[0299] In summary, the image processing method provided by the implementation of the present application can further determine whether to input only the multispectral raw image 101 into the image processing module, or to input both the multispectral raw image 101 and the high-resolution color / grayscale raw image 102 into the image processing module, determine whether to obtain only one of the color image 401 and the high-resolution color image 402 through the image processing module, or obtain both the color image 401 and the high-resolution color image 402 simultaneously, determine the further image adjustment processing method and display method of the obtained color image 401 and the high-resolution color image 402 by the mobile phone, and perform corresponding processing to obtain the processed image, based on the aforementioned target information. That is, the mobile phone can determine at least one of the images included in the image input to the image processing module, the images included in the image output by the image processing module, whether to perform further image adjustment processing on the image output by the image processing module, and the display method of the image output by the image processing module based on the aforementioned target information. In this way, images that better meet user needs can be obtained, effectively improving image quality and enhancing user experience.

[0300] In addition, the controls displayed on the aforementioned mobile phone are only some examples of controls for users to perform corresponding touch operations. The controls for users to perform corresponding touch operations can also be other text, graphic, and other controls, which can be set as needed. In addition, users can also interact with electronic devices such as mobile phones through other interactive methods other than touch operations, such as voice, gestures, etc., to achieve corresponding selection operations or interactions.

[0301] The above-mentioned image processing system performs color restoration processing, image preprocessing, image stylization processing, image spatial resolution enhancement, and other corresponding processing on image features or parameters on the multispectral RAW image 101 and the high-resolution color (or grayscale) RAW image 102, which can serve as some examples of image adjustment processing.

[0302] In summary, the image processing method provided by the implementation of the present application can be applied to, for example, a mobile phone. According to the user's selection operation of the control corresponding to the aforementioned target spectral feature information, such as the user's control for the user's needs for the image, and the user's photo operation, it is determined whether to obtain the multispectral image 101 as the initial image, or to obtain the multispectral image 101 and the high-resolution color (or grayscale) image 102 as the initial image, and determine the image adjustment method for the initial image, and determine whether to obtain only one of the color image 401 and the high-resolution color image 402, or to obtain both the color image 401 and the high-resolution color image 402 as the target image, and determine the further image adjustment processing method, display method, etc. of the obtained color image 401 and the high-resolution color image 402 by the mobile phone, and perform corresponding processing to obtain the processed image. That is, the mobile phone can determine at least one of the contents of the image included in the initial image, the image included in the target image, the image adjustment method for the initial image, the display method of the target image, etc. based on the aforementioned target information, and perform corresponding processing. In this way, an image that better meets the user's needs can be obtained, and the image quality and user experience are effectively improved.

[0303] Of course, this image processing method can also be applied to image processing scenarios such as mobile phone album image processing, and the aforementioned user operations can be specifically set according to the scenario.

[0304] In the image processing method provided by the implementation of the present application, the aforementioned first interface, second interface and other display interfaces may also be other forms of interfaces that facilitate users to perform corresponding operations, and user operations may also be other types of user operations, which can all be set as needed.

[0305] In summary, the image processing method provided by the implementation of this application can be applied to the field of multi-spectral imaging / image (MSI). Electronic devices can perform color restoration processing on multi-spectral RAW images or high-resolution color (or grayscale) images based on parameter-learnable image processing models and multi-spectral RAW images, which can improve color restoration accuracy and obtain color images with better color restoration, effectively improving image quality and thus improving user experience. In addition, electronic devices can perform the aforementioned other image adjustment processing on images to obtain images that better meet user needs, which also effectively improves user experience.

[0306] In addition, in some implementations, a color image can be obtained by performing channel-by-channel image preprocessing, spectral image interpolation, piecewise Gaussian approximation function processing on the multispectral RAW image, and then the final color image can be obtained by color correction of the color image. In this way, the multispectral RAW image is processed based on the traditional interpolation and function fitting algorithm. The interpolation and function fitting algorithm is fixed and cannot be applied to more scenes, and each step is related and independent, which can achieve full-channel end-to-end image optimization. The image processing method provided in the aforementioned embodiment of the present application performs color restoration processing on the multispectral RAW image or the high-resolution color (or grayscale) image based on the parameter-learnable image processing model and the multispectral RAW image. It can be applied to more scenes and obtain a color image with better color restoration, which effectively improves the image quality and thus improves the user experience.

[0307] In some implementations, image color can also be corrected by locally averaging the spectral response. However, this method only involves a simple averaging operation for spectral response processing, making it difficult to fully extract spectral information. However, the image processing method provided in the aforementioned embodiment of the present application, based on the aforementioned parameter-learnable image processing model and multispectral RAW images, can fully extract spectral features and perform color restoration processing on multispectral RAW images or high-resolution color (or grayscale) images. This can improve color restoration accuracy and obtain color images with better color restoration, effectively improving image quality and thereby enhancing user experience.

[0308] In some implementations, corresponding parameters can also be obtained based on classic statistical assumptions, and the classic white balance algorithm for color images can be extended to multispectral images, and the multispectral images can be white balanced according to the classic white balance algorithm. However, in this way, the classic white balance algorithm is obtained based on classic statistical assumptions, which is only effective for some scenes and has weak overall applicability. The image processing method provided in the aforementioned embodiment of the present application, based on the aforementioned parameter-based learnable image processing model and multispectral RAW images, can fully extract spectral features, perform color restoration processing on multispectral RAW images or high-resolution color (or grayscale) images, improve color restoration accuracy, obtain color images with better color restoration, effectively improve image quality, and thus improve user experience.

[0309] In summary, the embodiments of the present application provide a new image color processing process, which can improve the accuracy of color reproduction, obtain color images with better color reproduction, effectively improve image quality, and thus improve user experience.

[0310] In particular, the aforementioned light source estimation and color adaptation model based on multispectral RAW images provided in the embodiments of the present application can obtain more accurate image adjustment information based on the multispectral RAW images, and perform color restoration processing on the multispectral RAW images or high-resolution color (or grayscale) RAW images according to the image adjustment information, which can improve the accuracy of color restoration and obtain color images with better color restoration, effectively improving the image quality and thereby improving the user experience.

[0311] In one embodiment of the present application, the image processing system can be implemented, for example, by an image signal processor (ISP), which processes the image signal output by the image signal sensor. The ISP plays a central and dominant role in the camera system and is a key component of the camera. Its main functional features include demosaicing, automatic exposure, automatic white balance, lens shading removal, gamma correction, color space conversion, dynamic range correction, and image cropping.

[0312] An embodiment of the present application also provides an image processing model training method, which includes: inputting a training sample image into an initial image processing model, where the training sample image includes a multispectral original image collected by a multispectral image sensor; the initial image processing model obtains an output image based on the training sample image, where the output image is a color image; and training the initial image processing model based on the output image to obtain an image processing model.

[0313] In this embodiment, the obtained image processing model can be, for example, the image processing model including one or more neural networks shown in FIG8 , or can be the light source estimation and color adaptation model shown in FIG9 .

[0314] In addition, the training sample images may also include the aforementioned high-resolution color (or grayscale) images, which may be selected and set as needed.

[0315] In this implementation, the training sample image is a multispectral RAW image, the output image is a color image after color restoration processing, and the structure of the initial image processing model is the same as the aforementioned image processing module. In this way, the initial image processing model is trained according to the obtained output image, and the model parameters in the initial image processing model are adjusted until the training termination condition is reached, and the image processing model after model training can be obtained. The training termination condition may be, for example, the convergence of the corresponding target loss function or the reaching of a preset number of iterations. The convergence condition of the target loss function and the target loss function, or the number of iterations can be specifically set as needed, and this application does not make specific restrictions on this. Performing color restoration processing on the multispectral RAW image based on the image processing model after model training can improve the accuracy of color restoration and obtain a color image with better color restoration, effectively improving the image quality and thus improving the user experience.

[0316] An embodiment of the present application also provides an image processing device, which includes the aforementioned image processing system, for obtaining a corresponding initial image in response to the aforementioned user operation, performing corresponding image adjustment processing on the initial image, obtaining a corresponding target image, and displaying the corresponding target image. The specific image processing process is as described above and will not be repeated here.

[0317] Exemplarily, as shown in FIG23 , it includes: a first input module and an image processing module. Among them: the first input module is used to input a first image into the image processing module, the first image includes a first sub-image, and the first sub-image is a multispectral original image collected by a multispectral image sensor. The image processing module includes an image processing model pre-trained based on machine learning; the image processing module is used to perform a first color restoration process on the first image using the image processing model to obtain a second image. The first color restoration process includes adjusting the color of the first image based on the first image adjustment information obtained by the image processing model through the first sub-image. The second image is a color image.

[0318] The first input module and the image processing module can be implemented by software or hardware. For example, the implementation of the first input module will be described below using the first input module as an example. Similarly, the implementation of the image processing module can refer to the implementation of the first input module.

[0319] As an example of a software functional unit, the first input module may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the first input module may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0320] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0321] As an example of a hardware functional unit, the first input module may include at least one computing device, such as a server. Alternatively, the first input module may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0322] The multiple computing devices included in the first input module can be distributed in the same region or in different regions. The multiple computing devices included in the first input module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first input module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0323] In other embodiments, the first input module can be used to execute any step in the image processing method, and the image processing module can be used to execute any step in the image processing method. The steps that the first input module and the image processing module are responsible for implementing can be specified as needed. The full functions of the image processing device can be realized by respectively implementing different steps in the image processing method through the first input module and the image processing module.

[0324] In this embodiment, the process of each module realizing the corresponding function can be referred to the aforementioned content related to the image processing method, which will not be repeated here.

[0325] In this embodiment, the image processing apparatus may also be applied to computing devices such as computers and servers, or to a computing device cluster including at least one computing device, to implement image processing functions.

[0326] The present application also provides an image processing model training device, as shown in FIG24 , comprising: a second input module, an initial image processing model module, and a training module. The second input module is configured to input a training sample image into the initial image processing model module, the initial image processing model module comprising an initial image processing model, and the training sample image comprising a multispectral original image acquired by a multispectral image sensor; the initial image processing model module is configured to obtain an output image based on the training sample image, the output image being a color image; and the training module is configured to train the initial image processing model based on the output image to obtain an image processing model, which is applied to the aforementioned image processing method.

[0327] The second input module, the initial image processing model module, and the training module can all be implemented via software or hardware. For example, the second input module will be used as an example to describe its implementation. Similarly, the implementations of the initial image processing model module and the training module can refer to the implementations of the second input module.

[0328] As an example of a software functional unit, the second input module may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the second input module may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0329] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0330] As an example of a hardware functional unit, the second input module may include at least one computing device, such as a server. Alternatively, the second input module may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0331] The multiple computing devices included in the second input module can be distributed in the same region or in different regions. The multiple computing devices included in the second input module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the second input module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0332] In other embodiments, the second input module can be used to execute any step in the image processing model training method, the initial image processing model module can be used to execute any step in the image processing model training method, and the training module can be used to execute any step in the image processing model training method. The steps that the second input module, the initial image processing model module and the training module are responsible for implementing can be specified as needed. The full functions of the image processing model training device are realized by respectively implementing different steps in the image processing model training method through the second input module, the initial image processing model module and the training module.

[0333] In this embodiment, the process of each module realizing the corresponding function can be referred to the aforementioned content related to the image processing model training method, which will not be repeated here.

[0334] In this embodiment, the image processing model training device can also be applied to computing devices such as computers and servers, or to a computing device cluster including at least one computing device to realize the image processing model training function.

[0335] The image processing method provided in the embodiments of this application has a wide range of application scenarios. It can be applied to a variety of image processing applications, such as camera or mobile phone photography, autonomous driving, and terminal devices (such as sweeping robots and augmented reality (AR) / virtual reality (VR) glasses), and can effectively improve image processing effects. That is, the aforementioned electronic devices can be cameras or mobile phones, autonomous vehicles, terminal devices (such as sweeping robots and AR / VR glasses), and other electronic devices with image processing capabilities.

[0336] Furthermore, the image processing method provided in the embodiments of the present application can be deployed on computing nodes of related equipment, and through software modification and adaptation to hardware (such as image sensors, etc.), image color processing performance can be improved. Among them, the computing node can be a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a tensor processing unit (TPU), or an application-specific integrated circuit (ASIC).

[0337] That is, the aforementioned image processing system can be implemented by a CPU, GPU, NPU, TPU or ASIC, etc.

[0338] This application also provides a computing device 10. As shown in Figures 25A and 25B, computing device 10 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. Processor 104, memory 106, and communication interface 108 communicate with each other via bus 102. Computing device 10 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 10.

[0339] Bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, Figures 25A and 25B illustrate a single bus line, but this does not imply that there is only one bus or type of bus. Bus 102 may include a path for transmitting information between various components of computing device 10 (e.g., memory 106, processor 104, and communication interface 108).

[0340] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0341] The memory 106 may include volatile memory, such as random access memory (RAM). The processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0342] As shown in FIG25A , memory 106 stores executable program code. Processor 104 executes this executable program code to implement the functions of the aforementioned first input module and image processing module, i.e., the functions of the aforementioned image processing device, thereby implementing the image processing method. Specifically, memory 106 stores instructions for executing the image processing method.

[0343] Alternatively, as shown in FIG25B , memory 106 stores executable code, and processor 104 executes the executable code to implement the functions of the aforementioned second input module, initial image processing model module, and training module, respectively. This means that the functions of the aforementioned image processing model training device are implemented, thereby implementing the image processing model training method. Specifically, memory 106 stores instructions for executing the image processing model training method.

[0344] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 10 and other devices or a communication network.

[0345] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0346] 26A , the computing device cluster includes at least one computing device 10. The memory 106 in one or more computing devices 10 in the computing device cluster may store the same instructions for executing the image processing method.

[0347] In some possible implementations, the memory 106 of one or more computing devices 10 in the computing device cluster may also store partial instructions for executing the image processing method. In other words, the combination of one or more computing devices 10 can jointly execute the instructions for executing the image processing method.

[0348] The memories 106 in different computing devices 10 in the computing device cluster may store different instructions, each for executing a portion of the functions of the image processing apparatus. That is, the instructions stored in the memories 106 in different computing devices 10 may implement the functions of one or more of the first input module and the first image processing module.

[0349] As shown in Figure 26B, the computing device cluster includes at least one computing device 10. The memory 106 in one or more computing devices 10 in the computing device cluster may store the same instructions for executing the image processing model training method.

[0350] In some possible implementations, the memory 106 of one or more computing devices 10 in the computing device cluster may also store partial instructions for executing the image processing model training method. In other words, the combination of one or more computing devices 10 can jointly execute the instructions for executing the image processing model training method.

[0351] The memories 106 in different computing devices 10 in the computing device cluster may store different instructions, each for executing a portion of the functions of the image processing model training apparatus. That is, the instructions stored in the memories 106 in different computing devices 10 may implement the functions of one or more of the second input module, the initial image processing model module, and the training module.

[0352] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network, which may be a wide area network or a local area network.

[0353] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform an image processing method or an image processing model training method.

[0354] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the image processing method or instruct the computing device to execute the image processing model training method.

[0355] In addition, the above-mentioned image processing method and image processing model training method can be applied to a computing device, or to a computing device cluster including at least one computing device. The computing device or computing device cluster is used to deploy algorithms related to the image processing method involved in the embodiments of this application, or algorithms related to the image processing model training method, to achieve corresponding functions. The computing device can be an electronic device such as the above-mentioned vehicle, intelligent driving server, mobile phone, tablet computer, wearable device, or other types of devices, which can be selected and set as needed.

[0356] An embodiment of the present application provides an electronic device, comprising: a memory for storing a computer program, the computer program including program instructions; a processor for executing the program instructions so that the electronic device performs the aforementioned image processing method, or so that the electronic device performs the aforementioned image processing model training method.

[0357] Image color quality is a key factor affecting the photography performance of electronic devices such as cameras and mobile phones. Therefore, performing color restoration processing on the image can obtain an image with better color restoration performance and improve image quality. Performing other image adjustment processing on the image, such as the aforementioned highlighting, can further obtain an image that meets user needs, thereby effectively improving the user experience of the electronic device.

[0358] Electronic devices may include, for example, vehicle-mounted devices, mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MIDs), wearable devices (including, for example, smart watches, smart bracelets, pedometers, etc.), personal digital assistants, portable media players, navigation devices, video game devices, set-top boxes, virtual reality and / or augmented reality devices, Internet of Things devices, industrial control equipment, streaming media client devices, e-books, reading devices, POS machines, and other devices.

[0359] In the implementation manner of the present application, the terms "first", "second", etc. are only used to distinguish and describe, and cannot be understood as indicating or implying relative importance.

[0360] In the embodiments of the present application, some structural or method features may be shown in a specific arrangement and / or order in the accompanying drawings. However, it should be understood that such a specific arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0361] Although the present application has been illustrated and described with reference to certain embodiments of the present application, those skilled in the art should understand that the above is a further detailed description of the present application in conjunction with specific embodiments, and the specific implementation of the present application should not be considered to be limited to these descriptions. Those skilled in the art may make various changes in form and details, including making several simple deductions or substitutions, without departing from the spirit and scope of the present application.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a first image sensor, the first image sensor is a multispectral image sensor, and the method includes: Acquire a first initial image captured by the first image sensor; A first image adjustment process is performed on the first initial image to obtain a first target image, where the first image adjustment process includes a first color restoration process.

2. The image processing method according to claim 1, wherein: The electronic device further includes a second image sensor, where the second image sensor is an RGB image sensor or a grayscale image sensor. The method further includes: Acquire a second initial image captured by the second image sensor; A second image adjustment process is performed on the second initial image according to the first initial image to obtain a second target image, where the second image adjustment process includes a second color restoration process.

3. The image processing method according to claim 2, wherein: The method further comprises: The first target image and / or the second target image are displayed according to target information, where the target information includes information related to image usage scenarios and / or device information of the electronic device.

4. The image processing method according to claim 1, wherein: The electronic device further includes a second image sensor, where the second image sensor is an RGB image sensor or a grayscale image sensor. The method further includes: In response to a first user operation, determining whether to activate the second image sensor, In a case where it is determined that the second image sensor is not to be started, performing the first image adjustment process on the first initial image; When it is determined that the second image sensor is started, the second image adjustment processing of the second initial image based on the first initial image is performed, or the second image adjustment processing of the second initial image based on the first initial image and the first image adjustment processing of the first initial image are performed.

5. The image processing method according to claim 4, characterized in that The first user operation is a user selection operation of an image usage scenario, and the method further includes: Displaying a first interface, wherein the first interface includes a selection control for an image usage scenario; receiving a user's selection operation of an image usage scenario through the selection control; Then, in response to the first user operation, determining whether to start the second image sensor includes: In response to the first user operation, an image usage scenario selected by a user is determined, and whether to activate the second image sensor is determined according to the image usage scenario selected by the user.

6. The image processing method according to claim 4, wherein: The step of determining whether to activate the second image sensor in response to the first user operation includes: In response to the first user operation, acquiring target information, the target information including information related to the image usage scenario and / or device information of the electronic device; Determine whether to activate the second image sensor according to the target information.

7. The image processing method according to any one of claims 1 to 6, characterized in that: The performing a first image adjustment process on the first initial image to obtain a first target image includes: performing the first color restoration process on the image of the first area in the first initial image to obtain a fourth target image, and performing target processing on the image of the second area in the first initial image to obtain a fifth target image; performing image fusion processing on the fourth target image and the fifth target image to obtain the first target image; or, The performing a first image adjustment process on the first initial image to obtain a first target image includes: performing the first color restoration process on the entire first initial image to obtain a third target image, performing the first color restoration process on an image in a first region of the first initial image to obtain a fourth target image, and performing target processing on an image in a second region of the first initial image to obtain a fifth target image; Perform image fusion processing on the third target image, the fourth target image, and the fifth target image to obtain the first target image.

8. The image processing method according to claim 7, wherein: The first region is a region corresponding to a sub-channel in the first initial image that does not conform to the target spectral characteristics, and the second region is a region corresponding to a sub-channel in the first initial image that conforms to the target spectral characteristics.

9. The image processing method according to claim 8, characterized in that: The method further comprises: In response to a second user operation, the target spectral characteristic is determined.

10. The image processing method according to claim 9, wherein: The second user operation is a user setting operation of the target spectral feature, and the method further includes: displaying a second interface, wherein the second interface includes a setting control for the target spectral characteristic; receiving a setting operation of the target spectral feature by a user through the setting control; Then, in response to the second user operation, determining the target spectral feature includes: In response to the second user operation, the target spectral characteristic set by the user is determined.

11. The image processing method according to claim 9, wherein: The second user operation is a user selection operation of an image usage scenario, and the method further includes: Displaying a third interface, wherein the third interface includes a selection control for an image usage scenario; receiving a user's selection operation on the image usage scenario through the selection control; Then, in response to the second user operation, determining the target spectral feature includes: In response to the second user operation, the image usage scenario selected by the user is determined, and the target spectral feature is determined according to the image usage scenario selected by the user and a preset correspondence between the image usage scenario and the target spectral feature.

12. The image processing method according to claim 8, wherein: The target spectral feature is that the spectral response of the first sub-channel is greater than or equal to a preset first response threshold or less than a preset second response threshold.

13. The image processing method according to claim 7, wherein: The target processing is any of the following: Image masking of overexposed areas; Target area image brightness enhancement processing; Grayscale processing of target area image.

14. The image processing method according to any one of claims 1 to 6, characterized in that: The performing a first image adjustment process on the first initial image to obtain a first target image includes: performing the first color restoration process on the first initial image to obtain a third target image; and determining image brightness information according to spectral response information of the second sub-channel in the first initial image; Performing brightness migration processing on the third target image according to the image brightness information to obtain the first target image.

15. The image processing method according to any one of claims 1 to 6, characterized in that: The performing a first image adjustment process on the first initial image to obtain a first target image includes: determining a spectral characteristic of the first initial image; determining first image adjustment information according to the spectral characteristics; The first color restoration process is performed on the first initial image according to the first image adjustment information to obtain the first target image.

16. The image processing method according to claim 15, characterized in that: The first image adjustment information includes a color subspace projection matrix, a light source power spectrum original domain response, and a subspace color correction matrix. The first color restoration process is performed on the first initial image according to the first image adjustment information to obtain the first target image, including: Performing color subspace projection processing on the first initial image according to the color subspace projection matrix to obtain a subspace response image; Obtaining a white balance diagonal matrix according to the color subspace projection matrix and the original domain response of the light source power spectrum, and performing white balance processing on the subspace response image according to the white balance diagonal matrix to obtain a white balance image; According to the subspace color correction matrix, color correction processing is performed on the white-balanced image to obtain a chromatic adaptation image as the first target image, or color gamut transformation processing is performed on the chromatic adaptation image to obtain the first target image.

17. The image processing method according to any one of claims 1 to 6, characterized in that: The performing a second image adjustment process on the second initial image according to the first initial image to obtain a second target image includes: determining a spectral characteristic of the first initial image; determining second image adjustment information according to the spectral characteristics; The second color restoration process is performed on the second initial image according to the second image adjustment information to obtain the second target image.

18. The image processing method according to claim 17, wherein: If the second initial image is an RGB image, the second image adjustment information includes white balance adjustment parameters and a subspace color correction matrix, the white balance adjustment parameters include a white balance diagonal matrix, and performing the second color restoration process on the second initial image according to the second image adjustment information to obtain the second target image includes: performing white balance processing on the second initial image according to the white balance adjustment parameter to obtain a white balanced image; Color correction is performed on the white-balanced image according to the subspace color correction matrix to obtain a chromatic adaptation image as the second target image.

19. The image processing method according to claim 17, wherein: The second image adjustment information is a third image determined based on the first initial image, and performing the second color restoration process on the second initial image according to the second image adjustment information to obtain the second target image includes: According to the third image, a color migration process is performed on the second initial image to obtain the second target image.

20. The image processing method according to any one of claims 1 to 6, characterized in that: The performing a first image adjustment process on the first initial image to obtain a first target image includes: At least one neural network is used to perform the first color restoration process on the first initial image to obtain the first target image.

21. The image processing method according to claim 2, wherein: The first image adjustment process and / or the second image adjustment process further includes any one of the following processes: Image preprocessing; Image stylization; Image spatial resolution enhancement processing.

22. An electronic device, characterized in that: include: a memory for storing a computer program, wherein the computer program includes program instructions; A processor is used to execute the program instructions so that the electronic device performs the image processing method according to any one of claims 1 to 21.

23. A computing device cluster, characterized in that: It includes at least one computing device, each computing device includes a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the image processing method as described in any one of claims 1-21.

24. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the image processing method according to any one of claims 1 to 21.

25. A computer-readable storage medium, characterized in that The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the image processing method according to any one of claims 1 to 21.

Images