Image processing method and device, electronic equipment and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and storage medium, and belongs to the technical field of artificial intelligence. The image processing method comprises the following steps: acquiring a first sample image data set and a second sample image data set; training an initial difference detection model based on the first sample image data set and the second sample image data set to obtain a target difference detection model, wherein the target difference detection model is used for outputting a difference image of the first sample image data set and the second sample image data set; inputting a first to-be-processed image and a second to-be-processed image into the target difference detection model to obtain a target difference image; and performing image processing on the first to-be-processed image or the second to-be-processed image based on the target difference image to obtain a target image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to an image processing method and device, an electronic device and a storage medium. BACKGROUND

[0002] Due to differences in design schemes and manufacturing processes of different types of image sensors, the response characteristics of different image sensors to light are inconsistent, thereby causing differences in the signal acquisition process, which leads to large differences in color, brightness, definition and the like of original images output by different types of image sensors to an image signal processing (ISP) unit. The original differences between images cannot be eliminated when passing through an ISP pipeline, and finally result in differences in imaging effects of different types of image sensors, that is, a so-called multi-camera effect consistency abnormality problem.

[0003] In the related art, in order to solve the above problem, parameters of the ISP unit corresponding to different types of image sensors are usually adjusted in a targeted manner to eliminate the differences between images when the images pass through the ISP unit.

[0004] However, this method needs to manually adjust parameters, and it is difficult to guarantee the consistency of the multi-camera effect, and a large amount of human resources is consumed. SUMMARY

[0005] The purpose of the embodiments of the application is to provide an image processing method, device, electronic device and storage medium, which can guarantee the consistency of images collected by different camera modules.

[0006] In a first aspect, the embodiments of the application provide an image processing method, which comprises:

[0007] obtaining a first sample image data set and a second sample image data set, the first sample image data set being image data collected by a first camera module, and the second sample image data set being image data collected by a second camera module under the same shooting scene, wherein a field of view angle of the first camera module is different from a field of view angle of the second camera module, or a photosensitive performance of the first camera module is different from a photosensitive performance of the second camera module;

[0008] training an initial difference detection model based on the first sample image data set and the second sample image data set to obtain a target difference detection model, the target difference detection model being used to output a difference image of the first sample image data set and the second sample image data set;

[0009] input the first to-be-processed image and the second to-be-processed image into the target difference detection model to obtain a target difference image, the first to-be-processed image being an image collected by the first camera module, and the second to-be-processed image being an image collected by the second camera module under the same shooting scene;

[0010] perform image processing on the first to-be-processed image or the second to-be-processed image based on the target difference image to obtain a target image;

[0011] During training of the initial difference detection model, the first sample images in the first sample image dataset and the second sample images in the second sample image dataset are subjected to image channel splitting, first loss function values of different image channels are respectively calculated, and a total loss function value of the initial difference detection model is calculated based on the first loss function values of the different image channels.

[0012] In a second aspect, an embodiment of the present application provides an image processing device, which comprises:

[0013] The acquisition module is configured to acquire a first sample image dataset and a second sample image dataset, the first sample image dataset being image data collected by a first camera module, and the second sample image dataset being image data collected by a second camera module under the same shooting scene, wherein a field of view angle of the first camera module is different from a field of view angle of the second camera module, or a photosensitive performance of the first camera module is different from a photosensitive performance of the second camera module.

[0014] The training module is configured to train an initial difference detection model based on the first sample image dataset and the second sample image dataset to obtain a target difference detection model, the target difference detection model being configured to output a difference image of the first sample image dataset and the second sample image dataset.

[0015] The detection module is configured to input the first to-be-processed image and the second to-be-processed image into the target difference detection model to obtain a target difference image, the first to-be-processed image being an image collected by the first camera module, and the second to-be-processed image being an image collected by the second camera module under the same shooting scene.

[0016] The processing module is configured to perform image processing on the first to-be-processed image or the second to-be-processed image based on the target difference image to obtain a target image.

[0017] During training of the initial difference detection model, the first sample images in the first sample image dataset and the second sample images in the second sample image dataset are subjected to image channel splitting, first loss function values of different image channels are respectively calculated, and a total loss function value of the initial difference detection model is calculated based on the first loss function values of the different image channels.

[0018] In a third aspect, an electronic device is provided, which includes a processor and a memory. The memory stores programs or instructions executable on the processor. The programs or instructions, when executed by the processor, implement the steps of the method according to the first aspect.

[0019] In a fourth aspect, a readable storage medium is provided, which stores programs or instructions. The programs or instructions, when executed by a processor, implement the steps of the method according to the first aspect.

[0020] In a fifth aspect, a chip is provided, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to execute programs or instructions to implement the method according to the first aspect.

[0021] In a sixth aspect, a computer program product is provided, which is stored in a storage medium. The computer program product is executed by at least one processor to implement the method according to the first aspect.

[0022] In the embodiments of the present application, the initial difference detection model can be trained based on the first sample image data set and the second sample image data set collected by the camera modules with different field angles of view or different light sensing performances, to obtain a target difference detection model. Then, the first to-be-processed image and the second to-be-processed image collected by the camera modules with different field angles of view or different light sensing performances in the same shooting scene are input into the target difference detection model to obtain a target difference image. Then, the first to-be-processed image or the second to-be-processed image is processed based on the target difference image, so that the target image can be obtained. The shooting effect of the target image obtained in this way is consistent with that of the second to-be-processed image or the first to-be-processed image. In this way, the shooting effect of the images collected by the camera modules with different field angles of view or different light sensing performances can be made consistent through image processing, the difference between the camera modules is compensated, the consistency of the multi-camera effect is ensured, and a large amount of human resources is not consumed. Moreover, when the initial difference detection model is trained, the first sample image in the first sample image data set and the second sample image in the second sample image data set are subjected to image channel splitting, the first loss function value of different image channels is calculated respectively, and the total loss function value of the initial difference detection model is calculated based on the first loss function value of different image channels. In this way, the accuracy of the loss function value is improved, the precision of the model training is improved, and the consistency of the shooting effect of the target image and the second to-be-processed image or the first to-be-processed image is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment;

[0024] Figure 2is a schematic diagram of a first weight information generation process according to an exemplary embodiment;

[0025] Figure 3 is a flowchart of an image processing method according to an exemplary embodiment;

[0026] Figure 4 is a schematic diagram of a difference detection model training process according to an exemplary embodiment;

[0027] Figure 5 is a schematic diagram of a channel splitting process according to an exemplary embodiment;

[0028] Figure 6 is a structural block diagram of an image processing apparatus according to an exemplary embodiment;

[0029] Figure 7 is a structural block diagram of an electronic device according to an exemplary embodiment;

[0030] Figure 8 is a hardware structural schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0032] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.

[0033] As in the background, in the related art, in order to solve the problem of multi-camera effect consistency anomaly, the parameters of the ISP unit corresponding to different types of image sensors are usually debugged to eliminate the difference between images when the image passes through the ISP unit. Specifically, the ISP can include stages such as noise reduction (Noise Reduction, NR), optical black correction (Optical Black, OB), lens shading correction (Lens Shading Correction, LSC), automatic white balance (Auto White Balance, AWB), color correction (Color Correction, CC), local tone mapping (Local Tone Mapping, LTM) and global tone mapping (Global Tone Mapping, GTM). However, this method requires manual adjustment of parameters, which is difficult to ensure the consistency of multi-camera effects and also consumes a large amount of human resources.

[0034] In addition, in the related art, in order to solve the problem of multi-camera effect consistency anomaly, the mapping relationship between different image sensors can also be calculated by a calibration algorithm, which is usually a mapping matrix. Apply this mapping matrix to a certain module in the back end, such as the color correction module, use the color correction matrix (Color Correction Matrix, CCM) to eliminate the difference in color correction, but limited by the difference of other modules. The correction of a single module does not greatly improve the consistency of the multi-camera effect, and calibrating and correcting multiple modules also consumes a large amount of human resources.

[0035] To solve the problems in the prior art, the embodiment of the present application provides an image processing method, which can train an initial difference detection model based on a first sample image dataset and a second sample image dataset collected by camera modules with different field angles or different light sensing performances, obtain a target difference detection model, input a first to-be-processed image and a second to-be-processed image collected by different camera modules in the same shooting scene into the target difference detection model, obtain a target difference image, and then perform image processing on the first to-be-processed image or the second to-be-processed image based on the target difference image, so as to obtain a target image. The shooting effect of the target image obtained in this way is consistent with that of the second to-be-processed image or the first to-be-processed image. In this way, the shooting effect of images collected by different camera modules can be made consistent through image processing, the difference between different camera modules is compensated, the consistency of multi-camera effects is ensured, and a large amount of human resources does not need to be consumed. Moreover, when the initial difference detection model is trained, the first sample image in the first sample image dataset and the second sample image in the second sample image dataset are subjected to image channel splitting, the first loss function value of different image channels is calculated respectively, and the total loss function value of the initial difference detection model is calculated based on the first loss function value of different image channels. In this way, the accuracy of the loss function value is improved, the precision of model training is improved, and the consistency of the shooting effect of the target image and the second to-be-processed image or the first to-be-processed image is further improved.

[0036] The image processing method, device, electronic device and storage medium provided by the embodiment of the present application will be described in detail below in combination with the drawings, specific embodiments and application scenarios.

[0037] Figure 1 Fig. 1 is a flowchart of an image processing method according to an exemplary embodiment, and the execution subject of the image processing method can be an electronic device. It should be noted that the above execution subject does not constitute a limitation on the present application.

[0038] As shown in Fig. 1, the image processing method can include the following steps: Figure 1

[0039] Step 110: Obtain a first sample image dataset and a second sample image dataset.

[0040] Here, the first sample image dataset can be image data collected by a first camera module, and the second sample image dataset can be image data collected by a second camera module in the same shooting scene.

[0041] The field angle of the first camera module can be different from that of the second camera module (FOV), or the light sensing performance of the first camera module can be different from that of the second camera module.​

[0042] The photosensitive performance of the first camera module is different from the photosensitive performance of the second camera module, which can be caused by different hardware designs of the first camera module and the second camera module, for example, the materials of complementary metal oxide semiconductor (CMOS) or the sizes of photosensitive components can be different, or the sizes of pixel units can be different.

[0043] Exemplarily, the first sample image in the first sample image data set and the second sample image in the second sample image data set can be collected by different types of golden modules, and can be raw image files (RAW), and of course can be images in other formats, which are not limited here.

[0044] In step 120, the initial difference detection model is trained based on the first sample image data set and the second sample image data set to obtain a target difference detection model.

[0045] Here, the target difference detection model can be used to output a difference image of the first sample image data set and the second sample image data set.

[0046] Specifically, when training the initial difference detection model, the first sample image in the first sample image data set and the second sample image in the second sample image data set can be image channel split, the first loss function value of different image channels can be calculated respectively, and the total loss function value of the initial difference detection model can be calculated based on the first loss function value of different image channels.

[0047] Exemplarily, the initial difference detection model can be UNet, and the target difference detection model can be UNet.

[0048] Then, the model parameters of the initial difference detection model can be adjusted according to the total loss function value. With the training, the total loss function value will continuously decrease in the early stage, but at the end of the training, since the model has been fitted to the maximum limit, the total loss function value will usually fluctuate around a certain value. When the total loss function value cannot continuously decrease, the training can be stopped to obtain the target difference detection model.

[0049] Exemplarily, the first sample image and the second sample image can both be in RGGB Bayer format, and can be split into four channels of R channel, GR channel, GB channel and B channel.

[0050] In an optional implementation, the above step 110 can specifically include:

[0051] The initial difference detection model is used to perform difference detection on the first sample image in the first sample image dataset and the corresponding second sample image in the second sample image dataset to obtain the sample difference image;

[0052] The target sample image is obtained by fusing the sample difference image and the second sample image;

[0053] The target sample image is split into a first single-channel image corresponding to each of the four image channels, and the first sample image is split into a second single-channel image corresponding to each of the four image channels.

[0054] The loss function value is determined based on the first single-channel image and the second single-channel image corresponding to each of the four image channels, and the first loss function value corresponding to each of the four image channels is obtained.

[0055] The average of the first loss function values ​​corresponding to the four image channels is determined as the total loss function value;

[0056] The model parameters of the initial difference detection model are adjusted based on the total loss function value to train the target difference detection model.

[0057] Here, both the first single-channel image and the second single-channel image can be images containing only single-channel information.

[0058] For example, such as Figure 2 As shown, images E and F can be input into the initial UNet, where image E is the first sample image and image F is the second sample image. The initial UNet is used to perform difference detection on images E and F to obtain image G, which is the sample difference image. Then, images G and F are superimposed to obtain image H, which is the target sample image. Finally, the total loss function value is determined based on images E and H.

[0059] For example, taking the image channel splitting of the first sample image as an example, the image channel splitting process will be introduced, such as... Figure 3 As shown, image E can be in RGGB Bayer format. Image E can be split into R channels to obtain R channel image 310, GR channels to obtain GR channel image 320, GB channels to obtain GB channel image 330, and B channels to obtain B channel image 340. Then, linear interpolation is performed on R channel image 310, GR channel image 320, GB channel image 330, and B channel image 340 respectively to obtain R channel image 311, GR channel image 321, GB channel image 331, and B channel image 341, which are the second single-channel images corresponding to the multiple image channels respectively.

[0060] The specific process of splitting the target sample image into the first single-channel images respectively corresponding to the plurality of image channels is the same as that of splitting the first sample image into the second single-channel images respectively corresponding to the plurality of image channels, and will not be described herein again.

[0061] Then, the loss function values can be respectively determined according to the first single-channel image and the second single-channel image corresponding to each image channel in the plurality of image channels, to obtain the first loss function values respectively corresponding to the plurality of image channels, and then the average value of the first loss function values respectively corresponding to the plurality of image channels is determined as the total loss function value.

[0062] Exemplarily, the calculation formula of the total loss function value can be as follows:

[0063]

[0064] wherein, Loss is the total loss function value, Loss R Loss is the first loss function value corresponding to the R image channel, Loss GR Loss is the first loss function value corresponding to the GR image channel, Loss GB Loss is the first loss function value corresponding to the GB image channel, Loss B Loss is the first loss function value corresponding to the B image channel.

[0065] In this way, by separately determining the first loss function value corresponding to each image channel, and taking the average value of the first loss function values corresponding to the plurality of image channels as the total loss function for model training, whether the model is trained can be more accurately measured.

[0066] In an optional implementation, the above-mentioned determining the loss function value according to the first single-channel image and the second single-channel image corresponding to each image channel in the four image channels respectively, to obtain the first loss function values respectively corresponding to the four image channels, can specifically include:

[0067] The first single-channel image is down-sampled M times to obtain M third single-channel images of different image sizes, and the second single-channel image is down-sampled M times to obtain M fourth single-channel images of different image sizes, the first single-channel image and the second single-channel image are down-sampled by the same multiple, and M is an integer greater than 1;

[0068] The loss function values are respectively determined according to the fifth single-channel image and the sixth single-channel image under each image size in the M+1 image sizes, to obtain the second loss function values respectively corresponding to the M+1 image sizes, the fifth single-channel image is any one of the M third single-channel images and the first single-channel image, and the sixth single-channel image is any one of the M fourth single-channel images and the second single-channel image;

[0069] An average of second loss function values corresponding to the M+1 image sizes respectively is determined as the first loss function value.

[0070] Here, the multiple of the down-sampling of the first single-channel image and the second single-channel image can be the same. M can be an integer greater than 1. The fifth single-channel image can be any one of the M third single-channel images and the first single-channel image, and the sixth single-channel image can be any one of the M fourth single-channel images and the second single-channel image.

[0071] Since the information richness of directly performing loss measurement on an image of a certain image size is insufficient, the first single-channel image and the second single-channel image can be respectively subjected to M times of Gaussian down-sampling.

[0072] Exemplarily, the first single-channel image can be subjected to three times of Gaussian down-sampling to obtain down-sampled images of 1 / 2, 1 / 4 and 1 / 8 times of the original image, i.e., the third single-channel image. The second single-channel image can also be subjected to three times of Gaussian down-sampling to obtain down-sampled images of 1 / 2, 1 / 4 and 1 / 8 times of the original image, i.e., the fourth single-channel image. Then, the loss function values between the two original images, the loss function values between the two down-sampled images of 1 / 2 times, the loss function values between the two down-sampled images of 1 / 4 times and the loss function values between the two down-sampled images of 1 / 8 times are respectively calculated, and the average of the four loss function values is taken as the first loss function value corresponding to the image channel.

[0073] Taking the first loss function value corresponding to the R image channel as an example, the calculation formula of the first loss function value can be as follows:

[0074]

[0075] wherein, Loss R is the first loss function value corresponding to the R image channel, Loss1 is the second loss function value between the two original images, Loss 1 / 2 is the second loss function value between the two down-sampled images of 1 / 2 times, Loss 1 / 4 is the second loss function value between the two down-sampled images of 1 / 4 times, and Loss 1 / 8 is the second loss function value between the two down-sampled images of 1 / 8 times.

[0076] In this way, by respectively calculating the second loss function values of the single-channel images of multiple image sizes and taking the average of the second loss function values as the first loss function value of the image channel, the amount of information of the loss measurement can be enriched, thereby improving the accuracy of the model training.

[0077] In an optional implementation, the loss function value is determined according to the fifth single-channel image and the sixth single-channel image under each of the M+1 image sizes respectively, to obtain the second loss function value corresponding to each of the M+1 image sizes, which can specifically include:

[0078] The third loss function value corresponding to each of the at least two loss functions is determined according to the fifth single-channel image and the sixth single-channel image;

[0079] The second loss function value is obtained by weighted summation of the at least two third loss function values according to preset weights corresponding to the at least two loss functions respectively.

[0080] Here, for the image of each image size, because the image resolutions are different, the image feature information contained in the images also differs, and the characteristics focused on by different loss functions also differ, so at least two loss functions can be used to jointly determine the second loss function value corresponding to each image size. Specifically, the preset weight can be assigned to each loss function according to the characteristics of the loss function under different image sizes.

[0081] Exemplarily, the at least two loss functions can be at least two of Mean Squared Error (MSE), Structural Similarity Index Measurement (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS).

[0082] Exemplarily, it is known from analysis of the three loss functions of MSE, SSIM, and LPIPS that MSE is the mean squared error information, which is relatively sensitive to information in a high-resolution image and is relatively accurate in measurement, so a higher weight can be assigned to MSE at a high resolution, i.e., a larger image size, and the weight gradually decreases at a low resolution, that is, the preset weight corresponding to MSE can be positively correlated with the image size.

[0083] SSIM is a structural similarity index, which is also relatively accurate at a high resolution, i.e., a larger image size, so a higher weight can be assigned to SSIM at a larger image size, and the weight gradually decreases at a low resolution, that is, the preset weight corresponding to SSIM can also be positively correlated with the image size.

[0084] LPIPS is a visual perceptual index, which can be well measured at various resolutions, i.e., at various image sizes, so the preset weight of LPIPS can be gradually increased as the image size decreases to compensate for the weakness of the previous MSE and SSIM.

[0085] Exemplarily, taking the second loss function value corresponding to the original image size as an example, the calculation formula of the second loss function value can be as follows:

[0086] Loss1=α*Loss MSE +β*Loss SSIM +γ*Loss LPIPS

[0087] Wherein, Loss1 is the second loss function value between two original images, Loss MSE is the third loss function value corresponding to the MSE, Loss SSIM is the third loss function value corresponding to the SSIM, Loss LPIPS is the third loss function value corresponding to the LPIPS, and α, β and γ can all be preset weights.

[0088] Exemplarily, in the case that the image size is from large to small: 1, 1 / 2, 1 / 4, 1 / 8, the preset weights (α, β, γ) assigned in turn can be (0.2, 0.4, 0.4), (0.2, 0.3, 0.5), (0.1, 0.3, 0.6), (0.1, 0.2, 0.7).

[0089] In this way, at least two loss functions are used to more comprehensively measure the training accuracy of the model, so as to facilitate obtaining a target difference detection model with higher accuracy.

[0090] In an optional implementation, the above step 120 can specifically include:

[0091] Obtaining sample weight information;

[0092] Inputting the sample weight information, the first sample image in the first sample image data set and the corresponding second sample image in the second sample image data set into the initial difference detection model, extracting the difference features corresponding to the plurality of image regions in the first sample image and the second sample image by the initial difference detection model respectively, and weighting the difference features according to the weights corresponding to the plurality of image regions respectively to obtain the target sample image.

[0093] In an optional implementation, the initial difference detection model can include N feature extraction layers, the number of sample weight information can be N, and N can be an integer greater than 1, wherein different feature extraction layers can correspond to different image sizes, and different image sizes can correspond to different sample weight information.

[0094] The above obtaining sample weight information can specifically include:

[0095] For the N feature extraction layers, obtain the weights corresponding to the multiple image regions at the image size corresponding to each feature extraction layer, and obtain N sample weight information;

[0096] The above extracting the difference features corresponding to the multiple image regions in the first sample image and the second sample image by the initial difference detection model, and weighting the difference features according to the weights corresponding to the multiple image regions, to obtain the target sample image, includes:

[0097] Extracting the difference features of the multiple image regions extracted by the N feature extraction layers in the initial difference detection model under N image sizes;

[0098] According to the N sample weight information, the difference features of the multiple image regions extracted by the N feature extraction layers are weighted to obtain the target sample image.

[0099] In an optional implementation, the above obtaining the weights corresponding to the multiple image regions at the image size corresponding to each feature extraction layer, and obtaining the N sample weight information, can specifically include:

[0100] According to the similarity between the multiple image regions corresponding to the first sample image and the second sample image, the weights corresponding to the multiple image regions under the original image size are determined to obtain the first sample weight information;

[0101] The first sample weight information is down-sampled for N-1 times to determine the weights corresponding to the multiple image regions under N-1 image sizes to obtain N-1 second sample weight information;

[0102] The first sample weight information and the N-1 second sample weight information are determined as the N sample weight information.

[0103] The above process of obtaining sample weights and determining a target sample image based on sample weights is the same as the process of obtaining weight information and determining a target difference image based on weight information in the following, and the specific process is described below.

[0104] Step 130, input the first to-be-processed image and the second to-be-processed image into the target difference detection model to obtain a target difference image.

[0105] Here, the first to-be-processed image can be an image collected by the first camera module, and the second to-be-processed image can be an image collected by the second camera module under the same shooting scene.

[0106] Specifically, the target difference detection model can be used to extract the difference features between the first to-be-processed image and the second to-be-processed image to obtain the target difference image.

[0107] In an optional implementation, the step 130 can specifically include:

[0108] obtaining a second to-be-processed image and a third to-be-processed image;

[0109] cropping a region corresponding to the second to-be-processed image from the third to-be-processed image according to the second to-be-processed image to obtain a first to-be-processed image.

[0110] Here, the second to-be-processed image and the third to-be-processed image can be collected by different camera modules. The FOV of the third to-be-processed image can be greater than that of the second to-be-processed image. The third to-be-processed image can include a region corresponding to the second to-be-processed image. The first to-be-processed image can be the region corresponding to the second to-be-processed image in the third to-be-processed image.

[0111] Specifically, a mature matching algorithm can be used to determine the region corresponding to the second to-be-processed image from the third to-be-processed image, and the region corresponding to the second to-be-processed image is cropped out, and then the cropped image is restored to the resolution consistent with the second to-be-processed image through sampling to obtain the first to-be-processed image.

[0112] The matching algorithm can be a Scale-invariant feature transform (SIFT) based matching algorithm.

[0113] For example, image A, i.e., the second to-be-processed image, and image B, i.e., the third to-be-processed image, can be collected by two different types of Golden modules, and both the image A and the image B can be Pure RAW. Due to the difference in FOV of the two camera modules, the field of view information of the image B is different from that of the image A, the FOV of the image B is greater than that of the image A, and the image B includes a region corresponding to the image A. Then, the image B and the image A can be matched by using the SIFT based matching algorithm to determine the region in the image B corresponding to the image A, and the region corresponding to the image A is cropped out, and the resolution is restored to be consistent with that of the image A through sampling to obtain image C, i.e., the first to-be-processed image.

[0114] In this way, since the scenes in the first to-be-processed image and the second to-be-processed image are the same, the difference between them, i.e., the target difference image, can be more accurately determined. Since the first to-be-processed image is part of the third to-be-processed image, the target difference image can accurately represent the difference in effect between the second to-be-processed image and the third to-be-processed image.

[0115] In an optional implementation, the step 130 can specifically include:

[0116] obtaining weight information;

[0117] inputting the weight information, the first to-be-processed image and the second to-be-processed image into the target difference detection model, extracting, by the target difference detection model, difference features corresponding to the plurality of image regions in the first to-be-processed image and the second to-be-processed image respectively, and performing weighted processing on the difference features according to the weights corresponding to the plurality of image regions respectively to obtain a target difference image.

[0118] Here, the difference sizes between different image regions in the first to-be-processed image and the second to-be-processed image are different, so different weights can be assigned to different image regions, and therefore the weight information can include weights corresponding to the plurality of image regions respectively. The weight information can be set manually according to experience or determined according to the first to-be-processed image and the second to-be-processed image, which is not limited here. Exemplarily, the weight information can be a weight table.

[0119] Exemplarily, the weight table can include weights corresponding to the plurality of image regions respectively. After inputting the weight table, the image C and the image A into the UNet, the UNet can extract difference features corresponding to the plurality of image regions in the image C and the image A respectively, and then perform matrix point multiplication on the difference features by using the weight table to obtain the target difference image.

[0120] In this way, by using the weights corresponding to the plurality of image regions respectively, the difference features corresponding to the plurality of image regions in the first to-be-processed image and the second to-be-processed image respectively are weighted processed, different weights can be assigned to different image regions, and the accuracy of the target difference image is improved.

[0121] In an optional implementation, the target difference detection model can include N feature extraction layers, the number of weight information can be N, and N can be an integer greater than 1, wherein different feature extraction layers can correspond to different image sizes, and different image sizes can correspond to different weight information.

[0122] The obtaining of the weight information can specifically include:

[0123] For the N feature extraction layers, the weights corresponding to the plurality of image regions under the image size corresponding to each feature extraction layer are obtained to obtain N weight information.

[0124] The extracting, by the target difference detection model, of the difference features corresponding to the plurality of image regions in the first to-be-processed image and the second to-be-processed image respectively, and the performing of the weighted processing on the difference features according to the weights corresponding to the plurality of image regions respectively to obtain the target difference image, includes:

[0125] extracting, by the N feature extraction layers in the target difference detection model, difference features respectively corresponding to the plurality of image regions in the N image sizes of the first to-be-processed image and the second to-be-processed image;

[0126] performing weighted processing on the difference features of the plurality of image regions respectively extracted by the N feature extraction layers according to the N weight information, to obtain a target difference image.

[0127] Here, the feature extraction layer can at least include a concatenation layer, and the image size corresponding to each feature extraction layer can refer to the size of the difference features output by the concatenation layer in the feature extraction layer.

[0128] Since the feature extraction layer can further include a down-sampling layer, the down-sampling layer can be used to down-sample the input features, therefore, different feature extraction layers can correspond to different image sizes, and the N feature extraction layers correspond to N image sizes.

[0129] The difference features of each image size need to be associated with the weight information of the same size, therefore, different image sizes correspond to different weight information, and N image sizes correspond to N weight information.

[0130] Since the feature extraction layer can further include a convolution layer, the convolution layer can be used to extract the difference features respectively corresponding to the plurality of image regions in the first to-be-processed image and the second to-be-processed image, therefore, the feature extraction layer can extract the difference features respectively corresponding to the plurality of image regions in the first to-be-processed image and the second to-be-processed image.

[0131] Specifically, the weight information in the N image sizes can be obtained, and the N weight information is respectively input to the N feature extraction layers in the target difference detection model. The difference features respectively corresponding to the plurality of image regions in the N image sizes of the first to-be-processed image and the second to-be-processed image are extracted through the N feature extraction layers, and then the difference features of the plurality of image regions respectively extracted by the N feature extraction layers are weighted processed according to the N weight information, to obtain a target difference image.

[0132] Exemplarily, the target difference detection model can include four feature extraction layers. The weight information in the four image sizes can be obtained, and the four weight information is respectively input to the four feature extraction layers. The difference features respectively corresponding to the plurality of image regions in the four image sizes of the image C and the image A are extracted through the four feature extraction layers, and then the difference features of the plurality of image regions respectively extracted by the four feature extraction layers are weighted processed according to the four weight information, to obtain a target difference image.

[0133] In this way, based on the weight information corresponding to multiple image sizes, prior guidance can be provided to the feature extraction layers corresponding to multiple image sizes, further improving the accuracy of difference features and obtaining more accurate target difference images.

[0134] In one optional implementation, for the N feature extraction layers, the weights corresponding to multiple image regions under the image size corresponding to each feature extraction layer are obtained, resulting in N weight information, which may specifically include:

[0135] Based on the similarity between multiple corresponding image regions in the first and second images to be processed, the weights corresponding to the multiple image regions under the original image size are determined to obtain the first weight information.

[0136] The first weight information is downsampled N-1 times to determine the weights corresponding to multiple image regions under N-1 image sizes, thus obtaining N-1 second weight information.

[0137] The first weight information and N-1 second weight information are determined as N weight information.

[0138] Here, since the target difference detection model is designed to extract difference features, it needs to pay more attention to the differences between images. Therefore, regions with lower similarity need to be assigned more weight, and regions with higher similarity need to be assigned less weight. In other words, the similarity of each image region and its corresponding weight can be negatively correlated.

[0139] Specifically, SSIM calculation can be performed on multiple corresponding image regions in the first and second images to be processed using a sliding window. For any image region, the larger the SSIM value, the higher the similarity; the smaller the SSIM value, the lower the similarity. Based on the SSIM values ​​corresponding to different image regions, the weights corresponding to different image regions can be determined. For any image region, the larger the SSIM value, the smaller the weight; the smaller the SSIM value, the larger the weight. In this way, the first weight information can be obtained.

[0140] Then, the first weight information can be downsampled N-1 times to obtain N-1 second weight information, which can be the weight information corresponding to N-1 image sizes respectively.

[0141] In this way, we obtain the first weight information and N-1 second weight information, which is N weight information.

[0142] For example, such as Figure 4 As shown, multiple corresponding image regions in image C and image A can be selected using a sliding window, such as c1 and a1, c2 and a2, c... n-1 and a n-1 cn and a n , respectively, and determine the weights corresponding to the plurality of image regions in the original image size according to the SSIM values, for example, the weights corresponding to c1 and a1 are w1, the weights corresponding to c2 and a2 are w2, the weights corresponding to c3 and a3 are w3, and the weights corresponding to c4 and a4 are w4. n-1 and a n-1 , respectively, and determine the weights corresponding to the plurality of image regions in the original image size according to the SSIM values, for example, the weights corresponding to c1 and a1 are w1, the weights corresponding to c2 and a2 are w2, the weights corresponding to c3 and a3 are w3, and the weights corresponding to c4 and a4 are w4. n-1 , respectively, and determine the weights corresponding to the plurality of image regions in the original image size according to the SSIM values, for example, the weights corresponding to c1 and a1 are w1, the weights corresponding to c2 and a2 are w2, the weights corresponding to c3 and a3 are w3, and the weights corresponding to c4 and a4 are w4. n , respectively, and determine the weights corresponding to the plurality of image regions in the original image size according to the SSIM values, for example, the weights corresponding to c1 and a1 are w1, the weights corresponding to c2 and a2 are w2, the weights corresponding to c3 and a3 are w3, and the weights corresponding to c4 and a4 are w4. n , respectively, and determine the weights corresponding to the plurality of image regions in the original image size according to the SSIM values, for example, the weights corresponding to c1 and a1 are w1, the weights corresponding to c2 and a2 are w2, the weights corresponding to c3 and a3 are w3, and the weights corresponding to c4 and a4 are w4. n , respectively, and determine the weights corresponding to the plurality of image regions in the original image size according to the SSIM values, for example, the weights corresponding to c1 and a1 are w1, the weights corresponding to c2 and a2 are w2, the weights corresponding to c3 and a3 are w3, and the weights corresponding to c4 and a4 are w4.

[0143] In this way, the weights corresponding to the plurality of image regions in the original image size are determined according to the similarity between the corresponding plurality of image regions in the first to-be-processed image and the second to-be-processed image, the first weight information is obtained, the weights corresponding to the plurality of image regions are more accurately determined, and then the second weight information corresponding to the plurality of image sizes is quickly obtained through multiple down-sampling of the first weight information.

[0144] In some examples, the UNet includes a convolutional layer, a down-sampling layer, a splicing layer, and an up-sampling layer. Taking the UNet as an example, the specific process of the above step 130 can be as shown in FIG. 13B. Figure 5

[0145] In step 1301, the image C and the image A are input into the convolutional layer, and the image A and the image C are subjected to difference extraction by the convolutional layer to obtain difference features 501.

[0146] In step 1302, the difference features 501 are input into the down-sampling layer, and the difference features 501 are subjected to down-sampling by the down-sampling layer to obtain difference features 502.

[0147] In step 1303, the difference features 502 are input into the convolutional layer, and the difference features 502 are subjected to difference extraction by the convolutional layer to obtain difference features 503.

[0148] In step 1304, the difference features 503 are input into the down-sampling layer, and the difference features 503 are subjected to down-sampling by the down-sampling layer to obtain difference features 504.

[0149] In step 1305, the difference features 504 are input into the convolutional layer, and the difference features 504 are subjected to difference extraction by the convolutional layer to obtain difference features 505.

[0150] In step 1306, the difference features 505 are input into the down-sampling layer, and the difference features 505 are subjected to down-sampling by the down-sampling layer to obtain difference features 506.

[0151] ​Step 1307, input the difference feature 506 to the convolution layer, and perform difference extraction on the difference feature 506 by the convolution layer to obtain a difference feature 507.

[0152] Step 1308, input the difference feature 507 to the down-sampling layer, and perform down-sampling on the difference feature 507 by the down-sampling layer to obtain a difference feature 508.

[0153] Step 1309, input the difference feature 508 to the convolution layer, and perform difference extraction on the difference feature 508 by the convolution layer to obtain a difference feature 509.

[0154] Step 1310, input the difference feature 509 and the difference feature 507 to the splicing layer, and perform splicing on the difference feature 509 and the difference feature 507 by the splicing layer to obtain a difference feature 510.

[0155] Step 1311, perform SSIM measurement on the image C and the image A to obtain a weight table 1.

[0156] Step 1312, perform three times of down-sampling on the weight table 1 to obtain a weight table 2, a weight table 3 and a weight table 4.

[0157] It should be noted that the order of the steps 1301-1310 and the steps 1311-1312 is not limited in the embodiments of the present application.

[0158] Step 1313, perform matrix point multiplication on the difference feature 510 and the weight table 4 to obtain a difference feature 511.

[0159] Step 1314, input the difference feature 511 to the convolution layer, and perform difference extraction on the difference feature 511 by the convolution layer to obtain a difference feature 512.

[0160] Step 1315, input the difference feature 512 to the up-sampling layer, perform up-sampling on the difference feature 512 by the up-sampling layer, and input the up-sampled difference feature and the difference feature 505 to the splicing layer, and perform splicing on the up-sampled difference feature and the difference feature 505 by the splicing layer to obtain a difference feature 513.

[0161] Step 1316, perform matrix point multiplication on the difference feature 513 and the weight table 3 to obtain a difference feature 514.

[0162] Step 1317, input the difference feature 514 to the convolution layer, and perform difference extraction on the difference feature 514 by the convolution layer to obtain a difference feature 515.

[0163] Step 1318, input the difference feature 515 into the upsampling layer, perform upsampling on the difference feature 515 by the upsampling layer, and input the upsampled difference feature and the difference feature 502 into the splicing layer, splice the upsampled difference feature and the difference feature 502 by the splicing layer, and obtain the difference feature 516.

[0164] Step 1319, perform matrix point multiplication on the difference feature 516 and the weight table 2, and obtain the difference feature 517.

[0165] Step 1317, input the difference feature 517 into the convolution layer, perform difference extraction on the difference feature 517 by the convolution layer, and obtain the difference feature 518.

[0166] Step 1318, input the difference feature 518 into the upsampling layer, perform upsampling on the difference feature 518 by the upsampling layer, and input the upsampled difference feature and the difference feature 501 into the splicing layer, splice the upsampled difference feature and the difference feature 501 by the splicing layer, and obtain the difference feature 519.

[0167] Step 1319, perform matrix point multiplication on the difference feature 519 and the weight table 1, and obtain the difference feature 520.

[0168] Step 1317, input the difference feature 520 into the convolution layer, perform difference extraction on the difference feature 520 by the convolution layer, and obtain the target difference image 521.

[0169] In some examples, the sizes of the image C and the image A may not conform to the original input of the UNet, so before inputting the image C and the image A into the UNet, that is, before step 1301, the image C and the image A can be first down-sampled or up-sampled to make the sizes of the image C and the image A conform to the original input of the UNet.

[0170] For example, the sizes of the image C and the image A are both 4080x3060, and the original input of the UNet is 572x572, so a down-sampling layer can be added in the input front end to down-sample the image C and the image A, so that the sizes of the image C and the image A are consistent with the original input of the UNet.

[0171] In addition, since the target difference image needs to be fused with the image A subsequently, the size of the target difference image needs to be the same as the original size of the image A. In some examples, the original output of the UNet may not be the same as the original size of the image A, so the size of the target difference image 521 output by the UNet may be different from the original size of the image A. After the UNet outputs the target difference image, that is, after step 1301, the target difference image 521 output by the UNet can be down-sampled or up-sampled to make the size of the target difference image the same as the original size of the image A.

[0172] For example, the size of the target difference image 521 output by the UNet is 388x388, and the original size of the image A is 408x3060, so an upsampling layer can be added at the output end to upsample the target difference image 521, so that the size of the target difference image is the same as the original size of the image A.

[0173] In step 140, the first to-be-processed image or the second to-be-processed image is processed based on the target difference image to obtain a target image.

[0174] Here, the first to-be-processed image can be processed based on the target difference image to obtain a target image consistent with the effect of the second to-be-processed image, or the second to-be-processed image can be processed based on the target difference image to obtain a target image consistent with the effect of the first to-be-processed image.

[0175] Specifically, the target difference image and the first to-be-processed image or the second to-be-processed image can be superimposed to obtain a target image, for example, the pixel values of the target difference image and the first to-be-processed image or the second to-be-processed image at corresponding positions can be added to obtain a target image.

[0176] Thus, the initial difference detection model can be trained based on the first sample image dataset and the second sample image dataset collected by the camera modules with different field angles of view or different light sensitivity, and the target difference detection model can be obtained. Then, the first to-be-processed image and the second to-be-processed image collected by the different camera modules in the same shooting scene are input into the target difference detection model to obtain a target difference image, and then the first to-be-processed image or the second to-be-processed image is processed based on the target difference image to obtain a target image. The target image obtained in this way is consistent with the shooting effect of the second to-be-processed image or the first to-be-processed image. In this way, the shooting effect of the images collected by different camera modules can be made consistent through image processing, which compensates for the differences between different camera modules and ensures the consistency of the multi-camera effect, without consuming a large amount of human resources. Moreover, when training the initial difference detection model, the first sample image in the first sample image dataset and the second sample image in the second sample image dataset are subjected to image channel splitting, the first loss function values of different image channels are calculated respectively, and the total loss function value of the initial difference detection model is calculated based on the first loss function values of different image channels. In this way, the accuracy of the loss function value can be improved to improve the precision of model training, thereby further improving the consistency of the shooting effect of the target image and the second to-be-processed image or the first to-be-processed image.

[0177] The target difference detection model used in the image processing method provided in the embodiments of the present application can effectively determine the similarity of two inputs, thereby accurately measuring the effect difference between different camera modules and generating a difference image. The image processing method provided in the embodiments of the present application can accurately separate the original difference between different camera modules by compensating for the difference image, and can eliminate the difference while retaining effective image information. The original RAW data of different camera modules can be input into the target difference detection model, and the target difference image predicted by the target difference detection model is then obtained. Then, the RAW image collected by one of the camera modules is processed to obtain a RAW image that has little difference from the RAW image collected by the other camera module. In this way, the RAW images collected by different camera modules can be directly input into the ISP for processing, thereby ensuring the consistency of the ISP input. Therefore, the parameter reuse between different camera modules can be realized, and the debugging workload is greatly reduced. Meanwhile, the method has high precision and strong robustness, and is suitable for various types of images. Moreover, the difference between different camera modules is effectively solved, the stability and consistency of the camera module are improved, and the method can be widely applied to various types of camera modules, thereby bringing a new technical breakthrough to the photography equipment industry.

[0178] It should be noted that the application scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0179] The image processing method provided in the embodiments of the present application can be executed by an image processing device. In the embodiments of the present application, the image processing device is taken as an example to illustrate the image processing device provided in the embodiments of the present application.

[0180] Based on the same inventive concept, the present application also provides an image processing device. The image processing device will be described in detail below in combination with Figure 6 The image processing device provided in the embodiments of the present application will be described in detail.

[0181] Figure 6 is a structural block diagram of an image processing device according to an exemplary embodiment.

[0182] As shown in Figure 6 , the image processing device 600 can include:

[0183] The acquisition module 601 is configured to acquire a first sample image data set and a second sample image data set. The first sample image data set is image data collected by a first camera module, and the second sample image data set is image data collected by a second camera module under the same shooting scene. The field of view angle of the first camera module is different from the field of view angle of the second camera module, or the photosensitive performance of the first camera module is different from the photosensitive performance of the second camera module.

[0184] The training module 602 is configured to train an initial difference detection model based on the first sample image data set and the second sample image data set to obtain a target difference detection model. The target difference detection model is configured to output a difference image of the first sample image data set and the second sample image data set.

[0185] The detection module 603 is configured to input a first to-be-processed image and a second to-be-processed image into the target difference detection model to obtain a target difference image. The first to-be-processed image is an image collected by the first camera module, and the second to-be-processed image is an image collected by the second camera module under the same shooting scene.

[0186] The processing module 604 is configured to perform image processing on the first to-be-processed image or the second to-be-processed image based on the target difference image to obtain a target image.

[0187] During the training of the initial difference detection model, the first sample image in the first sample image data set and the second sample image in the second sample image data set are subjected to image channel splitting, the first loss function values of different image channels are calculated respectively, and the total loss function value of the initial difference detection model is calculated based on the first loss function values of different image channels.

[0188] The image processing apparatus 600 is described in detail as follows:

[0189] In one embodiment, the training module 602 can include:

[0190] The detection sub-module is configured to perform difference detection on the first sample image in the first sample image data set and the corresponding second sample image in the second sample image data set by using the initial difference detection model to obtain a sample difference image.

[0191] The fusion sub-module is configured to fuse the sample difference image and the second sample image to obtain a target sample image.

[0192] The splitting sub-module is configured to split the target sample image into a first single-channel image corresponding to four image channels respectively, and split the first sample image into a second single-channel image corresponding to four image channels respectively.

[0193] The first determining sub-module is configured to determine a loss function value according to a first single-channel image and a second single-channel image corresponding to each of the four image channels, respectively, to obtain a first loss function value corresponding to each of the four image channels.

[0194] The second determining sub-module is configured to determine an average value of the first loss function values corresponding to the four image channels as a total loss function value.

[0195] The training sub-module is configured to adjust model parameters of the initial difference detection model according to the total loss function value, to obtain a target difference detection model.

[0196] In one of the embodiments, the first determining sub-module can include:

[0197] The downsampling unit is configured to perform M times of downsampling on the first single-channel image to obtain M third single-channel images of different image sizes, and perform M times of downsampling on the second single-channel image to obtain M fourth single-channel images of different image sizes, wherein the first single-channel image and the second single-channel image are downsampled by the same number of times, and M is an integer greater than 1.

[0198] The first determining unit is configured to determine a loss function value according to a fifth single-channel image and a sixth single-channel image under each of M+1 image sizes, respectively, to obtain a second loss function value corresponding to each of the M+1 image sizes, wherein the fifth single-channel image is any one of the M third single-channel images and the first single-channel image, and the sixth single-channel image is any one of the M fourth single-channel images and the second single-channel image.

[0199] The second determining unit is configured to determine an average value of the second loss function values corresponding to the M+1 image sizes as the first loss function value.

[0200] In one of the embodiments, the first determining unit can include:

[0201] The first determining sub-unit is configured to determine a third loss function value corresponding to at least two kinds of loss functions according to the fifth single-channel image and the sixth single-channel image.

[0202] The calculating sub-unit is configured to perform weighted summation on the at least two third loss function values according to preset weights corresponding to the at least two kinds of loss functions, to obtain the second loss function value.

[0203] In one of the embodiments, the detection module 603 can include:

[0204] The obtaining sub-module is configured to obtain weight information, wherein the weight information includes weights corresponding to a plurality of image regions, respectively.

[0205] The extraction submodule is configured to input the weight information, the first to-be-processed image, and the second to-be-processed image into the target difference detection model, extract, by the target difference detection model, difference features corresponding to the plurality of image regions in the first to-be-processed image and the second to-be-processed image respectively, and perform weighted processing on the difference features according to the weights corresponding to the plurality of image regions respectively, to obtain the target difference image.

[0206] In one of the embodiments, the target difference detection model comprises N feature extraction layers, and the number of the weight information is N, where N is an integer greater than 1. Different feature extraction layers correspond to different image sizes, and different image sizes correspond to different weight information.

[0207] The acquisition submodule can comprise:

[0208] The acquisition unit is configured to, for the N feature extraction layers, acquire the weights corresponding to the plurality of image regions at the image size corresponding to each feature extraction layer, to obtain the N weight information.

[0209] The extraction submodule can comprise:

[0210] The extraction unit is configured to extract, by the N feature extraction layers in the target difference detection model, the difference features corresponding to the plurality of image regions at the N image sizes for the first to-be-processed image and the second to-be-processed image.

[0211] The processing unit is configured to perform weighted processing on the difference features of the plurality of image regions extracted by the N feature extraction layers respectively according to the N weight information, to obtain the target difference image.

[0212] In one of the embodiments, the acquisition unit can comprise:

[0213] The second determination subunit is configured to determine the weights corresponding to the plurality of image regions at the original image size according to the similarity between the plurality of image regions in the first to-be-processed image and the second to-be-processed image, to obtain the first weight information.

[0214] The third determination subunit is configured to perform N-1 times of down-sampling on the first weight information, to determine the weights corresponding to the plurality of image regions at the N-1 image sizes, to obtain the N-1 second weight information.

[0215] The fourth determination subunit is configured to determine the first weight information and the N-1 second weight information as the N weight information.

[0216] Thus, the initial difference detection model can be trained based on the first sample image dataset and the second sample image dataset collected by the camera modules with different field angles or different light sensing performances, to obtain a target difference detection model. Then, the first to-be-processed image and the second to-be-processed image collected by the different camera modules in the same shooting scene are input into the target difference detection model, to obtain a target difference image. Then, the first to-be-processed image or the second to-be-processed image is processed based on the target difference image, to obtain a target image. The shooting effect of the target image obtained in this way is consistent with that of the second to-be-processed image or the first to-be-processed image. In this way, the shooting effect of the images collected by the different camera modules can be made consistent through image processing, the difference between the different camera modules is compensated, the consistency of the multi-camera effect is ensured, and a large amount of human resources does not need to be consumed. Moreover, when the initial difference detection model is trained, the first sample image in the first sample image dataset and the second sample image in the second sample image dataset are subjected to image channel splitting, the first loss function value of different image channels is calculated respectively, and the total loss function value of the initial difference detection model is calculated based on the first loss function value of the different image channels. In this way, the accuracy of the loss function value is improved, the precision of the model training is improved, and the consistency of the shooting effect of the target image and the second to-be-processed image or the first to-be-processed image is further improved.

[0217] The image processing apparatus in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices than terminals. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the like. The electronic device can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like. The embodiments of the present application are not limited in this regard.

[0218] The image processing apparatus in the embodiments of the present application can be an apparatus with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, which are not limited in the embodiments of the present application.

[0219] The image processing apparatus provided in the embodiments of the present application can realize Figures 1 to 5 The method embodiments realize various processes and achieve the same technical effects. To avoid repetition, the various steps of the image processing method embodiments and the same technical effects are not described herein.

[0220] Optionally, as shown in Figure 7 The embodiments of the present application also provide an electronic device 700, which includes a processor 701 and a memory 702, and the memory 702 stores programs or instructions executable on the processor 701. When the programs or instructions are executed by the processor 701, the various steps of the above image processing method embodiments are realized, and the same technical effects are achieved. To avoid repetition, the various steps of the image processing method embodiments and the same technical effects are not described herein.

[0221] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0222] Figure 8 To realize the hardware structure of an electronic device in the embodiments of the present application.

[0223] The electronic device 800 includes but is not limited to a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810, and the like.

[0224] Those skilled in the art can understand that the electronic device 800 can also include a power supply (such as a battery) for powering various components. The power supply can be logically connected to the processor 810 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the embodiments of the present application does not constitute a limitation on the electronic device. The electronic device can include more or fewer components than those shown, or combine certain components, or have different component arrangements, which are not described herein.

[0225] The processor 810 is configured to acquire a first sample image data set and a second sample image data set. The first sample image data set is image data collected by a first camera module, and the second sample image data set is image data collected by a second camera module under the same shooting scene. The field of view angle of the first camera module is different from the field of view angle of the second camera module, or the photosensitive performance of the first camera module is different from the photosensitive performance of the second camera module.

[0226] training the initial difference detection model based on the first sample image data set and the second sample image data set to obtain a target difference detection model, the target difference detection model being configured to output a difference image of the first sample image data set and the second sample image data set;

[0227] inputting the first to-be-processed image and the second to-be-processed image into the target difference detection model to obtain a target difference image, the first to-be-processed image being an image captured by the first camera module, and the second to-be-processed image being an image captured by the second camera module under the same shooting scene;

[0228] performing image processing on the first to-be-processed image or the second to-be-processed image based on the target difference image to obtain a target image;

[0229] In the training of the initial difference detection model, the first sample image in the first sample image data set and the second sample image in the second sample image data set are subjected to image channel splitting, the first loss function values of different image channels are calculated respectively, and the total loss function value of the initial difference detection model is calculated based on the first loss function values of different image channels.

[0230] Thus, the initial difference detection model can be trained based on the first sample image data set and the second sample image data set captured by the camera modules with different field angles of view or different light sensitivities, to obtain a target difference detection model. Then, the first to-be-processed image and the second to-be-processed image captured by the different camera modules under the same shooting scene are input into the target difference detection model to obtain a target difference image. Then, the first to-be-processed image or the second to-be-processed image is subjected to image processing based on the target difference image, to obtain a target image. The target image obtained in this way has the same shooting effect as the second to-be-processed image or the first to-be-processed image. In this way, the shooting effect of the images captured by the different camera modules can be made consistent through image processing, the difference between the different camera modules is compensated, the consistency of the multi-camera effect is ensured, and a large amount of human resources is not consumed. Moreover, in the training of the initial difference detection model, the first sample image in the first sample image data set and the second sample image in the second sample image data set are subjected to image channel splitting, the first loss function values of different image channels are calculated respectively, and the total loss function value of the initial difference detection model is calculated based on the first loss function values of different image channels. In this way, the accuracy of the loss function value is improved, the precision of the model training is improved, and the consistency of the shooting effect of the target image and the second to-be-processed image or the first to-be-processed image is further improved.

[0231] In some embodiments, the processor 810 is further configured to perform difference detection on a first sample image in the first sample image data set and a corresponding second sample image in the second sample image data set by using an initial difference detection model to obtain a sample difference image;

[0232] fuse the sample difference image and the second sample image to obtain a target sample image;

[0233] split the target sample image into a first single-channel image corresponding to each of the four image channels, and split the first sample image into a second single-channel image corresponding to each of the four image channels;

[0234] determine a loss function value according to the first single-channel image and the second single-channel image corresponding to each of the four image channels to obtain a first loss function value corresponding to each of the four image channels;

[0235] determine an average of the first loss function values corresponding to the four image channels as a total loss function value;

[0236] adjust the model parameters of the initial difference detection model according to the total loss function value to obtain a target difference detection model.

[0237] In this way, by separately determining the first loss function value corresponding to each image channel and taking the average of the first loss function values corresponding to multiple image channels as the total loss function for model training, the model training can be more accurately measured.

[0238] In some embodiments, the processor 810 is further configured to perform M times down-sampling on the first single-channel image to obtain M third single-channel images of different image sizes, perform M times down-sampling on the second single-channel image to obtain M fourth single-channel images of different image sizes, the first single-channel image and the second single-channel image are down-sampled by the same number of times, and M is an integer greater than 1;

[0239] determine a loss function value according to a fifth single-channel image and a sixth single-channel image under each of M+1 image sizes to obtain a second loss function value corresponding to each of the M+1 image sizes, the fifth single-channel image being any one of the M third single-channel images and the first single-channel image, and the sixth single-channel image being any one of the M fourth single-channel images and the second single-channel image;

[0240] determine an average of the second loss function values corresponding to the M+1 image sizes as the first loss function value.

[0241] Therefore, by calculating the second loss function value for each single-channel image of multiple image sizes respectively and taking the average value as the first loss function value of the image channel, the information amount of the loss measurement can be enriched, thereby improving the accuracy of model training.

[0242] In some embodiments, the processor 810 is further configured to determine, according to the fifth single-channel image and the sixth single-channel image, third loss function values respectively corresponding to at least two loss functions.

[0243] According to preset weights respectively corresponding to the at least two loss functions, the at least two third loss function values are weighted and summed to obtain a second loss function value.

[0244] Therefore, by using at least two loss functions, the training accuracy of the model can be more comprehensively measured, thereby facilitating obtaining a target difference detection model with higher accuracy.

[0245] In some embodiments, the processor 810 is further configured to obtain weight information, the weight information including weights respectively corresponding to multiple image regions.

[0246] The weight information, the first to-be-processed image, and the second to-be-processed image are input into the target difference detection model, and the target difference detection model extracts difference features respectively corresponding to the multiple image regions in the first to-be-processed image and the second to-be-processed image, and performs weighted processing on the difference features according to the weights respectively corresponding to the multiple image regions to obtain a target difference image.

[0247] Therefore, by performing weighted processing on the difference features respectively corresponding to the multiple image regions in the first to-be-processed image and the second to-be-processed image according to the weights respectively corresponding to the multiple image regions, different weights can be allocated to different image regions, thereby improving the accuracy of the target difference image.

[0248] In some embodiments, the target difference detection model includes N feature extraction layers, the number of weight information is N, and N is an integer greater than 1, wherein different feature extraction layers correspond to different image sizes, and different image sizes correspond to different weight information; the processor 810 is further configured to, for the N feature extraction layers, obtain the weights respectively corresponding to the multiple image regions under the image size corresponding to each feature extraction layer to obtain N weight information.

[0249] The N feature extraction layers in the target difference detection model extract the difference features respectively corresponding to the multiple image regions of the first to-be-processed image and the second to-be-processed image under the N image sizes.

[0250] The N weight information is used to perform weighted processing on the difference features of the multiple image regions extracted by the N feature extraction layers to obtain a target difference image.

[0251] In this way, the feature extraction layers corresponding to the plurality of image sizes can be guided in advance based on the weight information corresponding to the plurality of image sizes, the accuracy of the difference features is further improved, and a more accurate target difference image is obtained.

[0252] In some embodiments, the processor 810 is further configured to determine, according to the similarity between the corresponding plurality of image regions in the first to-be-processed image and the second to-be-processed image, the weights corresponding to the plurality of image regions at the original image size respectively, to obtain first weight information.

[0253] The first weight information is down-sampled for N-1 times to determine the weights corresponding to the plurality of image regions at N-1 image sizes respectively, to obtain N-1 second weight information.

[0254] The first weight information and the N-1 second weight information are determined as N weight information.

[0255] In this way, the weights corresponding to the plurality of image regions at the original image size can be determined according to the similarity between the corresponding plurality of image regions in the first to-be-processed image and the second to-be-processed image, to obtain the first weight information, the weights corresponding to the plurality of image regions at the original image size can be determined more accurately, and then the second weight information corresponding to the plurality of image sizes can be quickly obtained by down-sampling the first weight information for multiple times.

[0256] It should be understood that in the embodiments of the present application, the input unit 804 can include a graphics processing unit (GPU) 8041 and a microphone 8042. The graphics processing unit 8041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 806 can include a display panel 8061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 can include a touch detection device and a touch controller. The other input devices 8072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, and the like, which will not be described here.

[0257] The memory 809 can be used to store software programs and various data. The memory 809 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 809 can include a volatile memory or a non-volatile memory, or the memory 809 can include both a volatile memory and a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 809 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0258] The processor 810 can include one or more processing units; optionally, the processor 810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 810.

[0259] The embodiments of the present application also provide a readable storage medium, the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize various processes of the above-mentioned image processing method embodiments, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0260] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable memory, a random access memory, a magnetic disk or an optical disk, etc.

[0261] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions to realize the processes of the above image processing method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0262] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0263] The embodiment of the present application provides a computer program product, which is stored in a storage medium, and the program product is executed by at least one processor to realize the processes of the above image processing method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0264] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in the opposite order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0265] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product in essence or in the form of a part that contributes to the prior art, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0266] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a first sample image data set and a second sample image data set, the first sample image data set being image data collected by a first camera module, and the second sample image data set being image data collected by a second camera module under the same shooting scene, wherein a field of view angle of the first camera module is different from a field of view angle of the second camera module, or a photosensitive performance of the first camera module is different from a photosensitive performance of the second camera module; training an initial difference detection model based on the first sample image data set and the second sample image data set to obtain a target difference detection model, the target difference detection model being used for outputting a difference image of the first sample image data set and the second sample image data set; inputting a first to-be-processed image and a second to-be-processed image into the target difference detection model to obtain a target difference image, the first to-be-processed image being an image collected by the first camera module, and the second to-be-processed image being an image collected by the second camera module under the same shooting scene; performing image processing on the first to-be-processed image or the second to-be-processed image based on the target difference image to obtain a target image; wherein, when training the initial difference detection model, a first sample image in the first sample image data set and a second sample image in the second sample image data set are subjected to image channel splitting, a first loss function value of different image channels is calculated respectively, and a total loss function value of the initial difference detection model is calculated based on the first loss function value of the different image channels.

2. The method of claim 1, wherein, The training of the initial difference detection model based on the first sample image data set and the second sample image data set to obtain the target difference detection model comprises: performing difference detection on a first sample image in the first sample image data set and a corresponding second sample image in the second sample image data set by using an initial difference detection model to obtain a sample difference image; fusing the sample difference image and the second sample image to obtain a target sample image; splitting the target sample image into a first single-channel image corresponding to four image channels respectively, and splitting the first sample image into a second single-channel image corresponding to the four image channels respectively; determining a loss function value according to the first single-channel image and the second single-channel image corresponding to each of the four image channels respectively to obtain a first loss function value corresponding to the four image channels respectively; determining an average value of the first loss function value corresponding to the four image channels respectively as the total loss function value; adjusting model parameters of the initial difference detection model according to the total loss function value to obtain the target difference detection model.

3. The method of claim 2, wherein, The determination of the loss function value according to the first single-channel image and the second single-channel image corresponding to each of the four image channels respectively to obtain the first loss function value corresponding to the four image channels respectively comprises: M times down-sampling is performed on the first single-channel image to obtain M third single-channel images with different image sizes, and M times down-sampling is performed on the second single-channel image to obtain M fourth single-channel images with different image sizes, the first single-channel image and the second single-channel image are down-sampled by the same number of times, and M is an integer greater than 1; determining a loss function value according to the fifth single-channel image and the sixth single-channel image under each of the M+1 image sizes, respectively, to obtain a second loss function value corresponding to the M+1 image sizes, respectively, the fifth single-channel image being any one of the M third single-channel images and the first single-channel image, and the sixth single-channel image being any one of the M fourth single-channel images and the second single-channel image; determining the average of the second loss function values corresponding to the M+1 image sizes, respectively, as the first loss function value.

4. The method of claim 3, wherein, The determining a loss function value according to the fifth single-channel image and the sixth single-channel image under each of the M+1 image sizes, respectively, to obtain a second loss function value corresponding to the M+1 image sizes, respectively, includes: determining a third loss function value corresponding to at least two loss functions according to the fifth single-channel image and the sixth single-channel image, respectively; performing weighted summation on the at least two third loss function values according to preset weights corresponding to the at least two loss functions, respectively, to obtain the second loss function value.

5. The method of claim 1, wherein, The inputting the first to-be-processed image and the second to-be-processed image into the target difference detection model to obtain a target difference image includes: obtaining weight information, the weight information including weights corresponding to a plurality of image regions, respectively; inputting the weight information, the first to-be-processed image, and the second to-be-processed image into the target difference detection model, extracting, by the target difference detection model, difference features corresponding to the plurality of image regions in the first to-be-processed image and the second to-be-processed image, respectively, and performing weighted processing on the difference features according to the weights corresponding to the plurality of image regions, respectively, to obtain the target difference image.

6. The method of claim 5, wherein, The target difference detection model includes N feature extraction layers, and the number of the weight information is N, N being an integer greater than 1, wherein different feature extraction layers correspond to different image sizes, and different image sizes correspond to different weight information. The obtaining the weight information includes: for the N feature extraction layers, obtaining weights corresponding to a plurality of image regions under an image size corresponding to each feature extraction layer, to obtain N pieces of weight information; extracting, by the N feature extraction layers in the target difference detection model, difference features corresponding to a plurality of image regions in the first to-be-processed image and the second to-be-processed image under N image sizes, respectively; and ​ The difference features of the plurality of image regions extracted by the N feature extraction layers are weighted to obtain the target difference image.

7. The method of claim 6, wherein, The N weight information is obtained by acquiring the weight corresponding to each image region under the image size corresponding to each feature extraction layer. The weight corresponding to each image region under the original image size is determined according to the similarity between the corresponding image regions in the first image and the second image, and the first weight information is obtained. The first weight information is down-sampled for N-1 times to determine the weight corresponding to each image region under N-1 image sizes, and N-1 second weight information is obtained. The first weight information and the N-1 second weight information are determined as the N weight information.

8. An image processing apparatus characterized by comprising: The device comprises: The acquisition module is configured to acquire a first sample image dataset and a second sample image dataset, the first sample image dataset being image data collected by a first camera module, and the second sample image dataset being image data collected by a second camera module under the same shooting scene, wherein the field of view of the first camera module is different from the field of view of the second camera module, or the photosensitive performance of the first camera module is different from the photosensitive performance of the second camera module. The training module is configured to train an initial difference detection model based on the first sample image dataset and the second sample image dataset to obtain a target difference detection model, the target difference detection model being configured to output a difference image of the first sample image dataset and the second sample image dataset. The detection module is configured to input a first image to be processed and a second image to be processed into the target difference detection model to obtain a target difference image, the first image to be processed being an image collected by the first camera module, and the second image to be processed being an image collected by the second camera module under the same shooting scene. The processing module is configured to perform image processing on the first image to be processed or the second image to be processed based on the target difference image to obtain a target image. During the training of the initial difference detection model, the first sample image in the first sample image dataset and the second sample image in the second sample image dataset are subjected to image channel splitting, the first loss function value of different image channels is calculated respectively, and the total loss function value of the initial difference detection model is calculated based on the first loss function value of the different image channels.

9. An electronic device, comprising: The device comprises a processor and a memory, the memory storing programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the image processing method according to any one of claims 1-7.

10. A readable storage medium, characterized by, The programs or instructions are stored on the readable storage medium, and the programs or instructions are executed by the processor to implement the steps of the image processing method according to any one of claims 1-7.

Images