A high-speed imaging image signal processing method and related apparatus

Abstract

The application provides a high-speed imaging image signal processing method and related device, including: acquiring a plurality of image sets, the image set including an RGB image and a RAW image captured by a high-speed camera under the same shooting scene; for any RAW image, generating an intermediate reference image of the RAW image, aligning the RGB image to the intermediate reference image to obtain an aligned reference image; using the intermediate reference image and the aligned reference image of the same RAW image, generating a binary mask of the RAW image; using the RAW image and the aligned reference image, completing training and testing of an image signal processing model, the training including: inputting the RAW image into the image signal processing model to obtain an RGB image output by the model, using the binary mask, the aligned reference image and the RGB image output by the model of the same RAW image, calculating a first loss value, and adjusting the image signal processing model using the first loss value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a high-speed imaging image signal processing method and related device. BACKGROUND

[0002] High-speed cameras can use high-speed imaging technology to capture RAW images (original images or unprocessed images). At present, high-speed cameras have played a key role in scientific research in multiple disciplines, such as biomechanical motion analysis, bio-inspired robot structure dynamics optimization, and real-time observation of neural activity and neutrophil migration trajectories. However, the RAW images captured by high-speed cameras are often accompanied by significant noise and severe color distortion, which seriously hinders the accurate observation of fast dynamic processes.

[0003] In order to improve the image quality, a data-driven image signal processing (ISP) model can process the RAW image, and output an RGB image, wherein the user perception quality of the RGB image is better than that of the RAW image, and the user perception quality reflects the subjective evaluation of the user's visual experience of the image.

[0004] At present, the data set of the data-driven ISP model is constructed using low-speed imaging technology. The degradation of the RAW image captured by the low-speed imaging technology is significantly different from that of the RAW image captured by the high-speed camera. The ISP model trained based on such a data set cannot effectively generalize to the noise and color distortion specific to the RAW image captured by the high-speed camera. Therefore, the accuracy of the ISP model in processing the RAW image captured by the high-speed camera decreases, and thus the user perception quality of the RGB image output by the ISP model is reduced. SUMMARY

[0005] In view of the above problems, the present application provides a high-speed imaging image signal processing method and related device to improve the accuracy of the ISP model in processing the RAW image captured by the high-speed camera, thereby improving the user perception quality of the RGB image output by the ISP model. The specific scheme is as follows:

[0006] The first aspect of the present application provides a high-speed imaging image signal processing method, comprising:

[0007] Obtaining a plurality of image sets corresponding to different shooting scenes, wherein the image set comprises an RGB image and a RAW image captured by a high-speed camera under the same shooting scene;

[0008] For any RAW image in the image set, generating an intermediate reference image of the RAW image, aligning the RGB image to the intermediate reference image to obtain an aligned reference image, the difference between the intermediate reference image and the RAW image is that the color of the intermediate reference image is the same as the color of the RGB image;

[0009] Using the intermediate reference image and the aligned reference image of the same RAW image, generating a binary mask of the RAW image;

[0010] Using the RAW image and the aligned reference image of the RAW image in the image set, completing the training and testing of the image signal processing model, the training includes: inputting the RAW image into the image signal processing model to obtain the RGB image output by the image signal processing model, using the binary mask, the aligned reference image and the RGB image output by the image signal processing model of the same RAW image, calculating a first loss value, using the first loss value to adjust the model parameters of the image signal processing model, so that the image signal processing model learns the feature representation of the RAW image to the RGB image, and learns the model parameters matched with the feature representation, the binary mask is used to filter out the area with misalignment problem between images when calculating the first loss value.

[0011] In a possible implementation, the using the intermediate reference image and the aligned reference image of the same RAW image to generate a binary mask of the RAW image includes:

[0012] Dividing the intermediate reference image and the aligned reference image according to a preset division manner, so as to divide the intermediate reference image into a plurality of first image blocks and divide the aligned reference image into a plurality of second image blocks;

[0013] Extracting the phase spectrum of each first image block and the phase spectrum of each second image block;

[0014] For the first image block and the second image block with the same position, using the phase spectrum of the first image block and the phase spectrum of the second image block to calculate the alignment score between the first image block and the second image block;

[0015] In a case where the alignment score is less than or equal to a threshold value, determining that element values of the first image block and the second image block in the binary mask are a first value, in a case where the alignment score is greater than the threshold value, determining that element values of the first image block and the second image block in the binary mask are a second value, the first image block and the second image block being a misalignment problem area in a case where the alignment score is less than or equal to a threshold value, the first value being used to filter out the misalignment problem area when calculating the first loss value.

[0016] In a possible implementation, the calculating the first loss value includes:

[0017] The first loss value is calculated by using a formula , is the RGB image of the RAW image output by the image signal processing model, is the aligned reference image of the RAW image, is the binary mask, and represents element-wise multiplication, is the L1 norm.

[0018] In a possible implementation, the training further includes: extracting a first amplitude spectrum of the aligned reference image of the RAW image, and extracting a second amplitude spectrum of the RGB image of the RAW image output by the image signal processing model.

[0019] A Wassertain distance between the first amplitude spectrum and the second amplitude spectrum is calculated by using a slice Wassertain distance function, and the Wassertain distance is taken as a second loss value.

[0020] The adjusting the model parameters of the image signal processing model by using the first loss value includes: calculating a loss value of the image signal processing model by using the first loss value and the second loss value, and adjusting the model parameters of the image signal processing model by using the loss value.

[0021] In a possible implementation, when the RAW image is input into the image signal processing model, a gain corresponding to the RAW image is input into the image signal processing model, the gain being related to a frame rate adopted by the RAW image, so as to determine a degradation degree of the RAW image by using the gain.

[0022] In a possible implementation, the method further includes:

[0023] capturing a RAW image to be processed by using the high-speed camera; ​

[0024] inputting the to-be-processed RAW image and a gain adopted by the high-speed camera when the to-be-processed RAW image is captured into the image signal processing model;

[0025] determining, by the image signal processing model, a degradation degree of the to-be-processed RAW image through the gain;

[0026] processing, by the image signal processing model, the to-be-processed RAW image guided by the degradation degree of the to-be-processed RAW image, to obtain an RGB image output by the image signal processing model.

[0027] In a possible implementation, the image set includes one RGB image and multiple RAW images captured by the high-speed camera at different frame rates under the same shooting scene, and the method further includes:

[0028] calculating, for any RAW image, an alignment score of an intermediate reference image and an aligned reference image of the RAW image;

[0029] summing up the alignment scores of all RAW images in the same image set as an alignment score of the image set;

[0030] sorting the alignment scores of all image sets in a descending order of the alignment scores;

[0031] generating a test data set of the image signal processing model by using image sets in a preset proportion in front of the sorting, and generating a training data set of the image signal processing model by using image sets after the preset proportion, wherein any data set in the test data set and the training data set includes multiple RAW images captured at different frame rates and aligned reference images of the RAW images, the training data set is used for training the image signal processing model, and the test data set is used for testing the trained image signal processing model.

[0032] In a possible implementation, the image set includes one RGB image and multiple RAW images captured by the high-speed camera at different frame rates under the same shooting scene;

[0033] The image signal processing model can learn a feature representation of a RAW image to an RGB image under a degradation degree indicated by different frame rates;

[0034] The generating, for any RAW image in the image set, of an intermediate reference image of the RAW image and the aligning of the RGB image to the intermediate reference image to obtain an aligned reference image includes:

[0035] The trained image mapper is used to process each RAW image in the image set to obtain an intermediate reference image of each RAW image output by the image mapper, the image mapper being trained using a specific RAW image in the image set and an RGB image in the image set, so that the image mapper learns the color difference characteristics between the specific RAW image and the RGB image and learns to restore an image with the same color as the RGB image using the color difference characteristics, the specific RAW image being a RAW image taken at a specific frame rate in the image set.

[0036] The spatial relationship between the intermediate reference image of the specific RAW image and the RGB image is estimated, and the RGB image is aligned to the intermediate reference image using the spatial relationship to obtain an aligned reference image of each RAW image in the image set to which the specific RAW image belongs.

[0037] The second aspect of the application provides a high-speed imaging image signal processing device, comprising:

[0038] An acquisition unit is configured to acquire a plurality of image sets corresponding to different shooting scenes, wherein each image set comprises an RGB image and a RAW image taken by a high-speed camera under the same shooting scene.

[0039] A first generation unit is configured to generate an intermediate reference image of a RAW image in any image set, align the RGB image to the intermediate reference image, and obtain an aligned reference image, wherein the intermediate reference image is different from the RAW image in that the color of the intermediate reference image is the same as that of the RGB image.

[0040] A second generation unit is configured to generate a binary mask of a RAW image using the intermediate reference image and the aligned reference image of the RAW image.

[0041] A training unit is configured to complete training and testing of an image signal processing model using a RAW image in the image set and the aligned reference image of the RAW image, wherein the training comprises: inputting the RAW image into the image signal processing model to obtain an RGB image output by the image signal processing model, calculating a first loss value using the binary mask, the aligned reference image, and the RGB image output by the image signal processing model of the same RAW image, and adjusting model parameters of the image signal processing model using the first loss value, so that the image signal processing model learns the feature representation of the RAW image to the RGB image and learns the model parameters matching the feature representation, and the binary mask is used to filter out areas with misalignment problems between images when calculating the first loss value.

[0042] The third aspect of the present application provides an electronic device comprising at least one processor and a memory connected to the processor, wherein:

[0043] The memory is configured to store a computer program;

[0044] The processor is configured to execute the computer program, so that the electronic device can implement the high-speed imaging image signal processing method of the first aspect or any implementation manner of the first aspect.

[0045] The fourth aspect of the present application provides a computer program product comprising computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement the high-speed imaging image signal processing method of the first aspect or any implementation manner of the first aspect.

[0046] The fifth aspect of the present application provides a computer storage medium, the storage medium carrying one or more computer programs, which, when executed by an electronic device, can cause the electronic device to implement the high-speed imaging image signal processing method of the first aspect or any implementation manner of the first aspect.

[0047] By the above technical solution, the high-speed imaging image signal processing method and related device provided by the present application use the RAW image captured by the high-speed camera and the aligned reference image of the RAW image as the data set, wherein the aligned reference image is obtained by aligning the intermediate reference image of the RAW image from the RGB image, and the aligned reference image can be the RGB image of the RAW image. Training and testing the ISP model using the data set composed of the RAW image and the aligned reference image of the RAW image can enable the ISP model to learn the feature representation of the RAW image captured by the high-speed camera to the RGB image, and learn the model parameters matched with the feature representation, so that the ISP model can use the matched model parameters to process the RAW image captured by the high-speed camera when processing the RAW image using the ISP model, thereby improving the accuracy and improving the user perception quality of the output RGB image.

[0048] And introducing a binary mask when calculating the first loss value of the ISP model, the binary mask can filter out the area with misalignment problem between the aligned reference image and the RGB image output by the ISP model when calculating the first loss value, so as to reduce the influence of pixel misalignment between images on the performance of the ISP model, thereby improving the accuracy when processing the RAW image using the trained ISP model and improving the user perception quality of the RGB image output by the ISP model. Misalignment problem can refer to pixel misalignment problem. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above-described and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent as various embodiments of the present disclosure are described in greater detail. It should be understood that the drawings are not to scale, as elements of the figures are not necessarily drawn to scale.

[0050] Figure 1 A flowchart of a high-speed imaging image signal processing method provided by the present application;

[0051] Figure 2 A flowchart of generating a binary mask in a high-speed imaging image signal processing method provided by the present application;

[0052] Figure 3 Another flowchart of a high-speed imaging image signal processing method provided by the present application;

[0053] Figure 4 A schematic diagram of a high-speed imaging image signal processing method provided by the present application;

[0054] Figure 5 A visualization result diagram of ISP models trained on different data sets;

[0055] Figure 6 A visual comparison diagram of ISP models trained directly on RHID data sets and models trained using MisISP;

[0056] Figure 7 A visual comparison diagram of ISP models trained directly on ZRR data sets and models trained using MisISP;

[0057] Figure 8 A visual comparison diagram of real applications of capturing moving objects at a frame rate of 10000 FPS;

[0058] Figure 9 A schematic diagram of the impact of different signal gain inputs on results;

[0059] Figure 10 A structural schematic diagram of a high-speed imaging image signal processing device provided by the present application;

[0060] Figure 11 A structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0061] The embodiments of the present application will be described below with reference to the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0062] The embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art can know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0063] The terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is only a way of distinguishing the objects with the same attributes in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or equipment containing a series of units do not necessarily limit to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

[0064] At present, the data set of the data-driven ISP model is constructed based on low-speed imaging technology. Specifically, the data set includes a plurality of image pairs, one image pair corresponds to one shooting scene, and one image pair includes an RGB image shot in the same shooting scene and a RAW image shot by using low-speed imaging technology. The RGB image can be shot by a single-lens reflex camera, wherein the image quality of the RAW image is lower than that of the RGB image. After the data set is constructed, all image pairs in the data set are used to complete the training and testing of the ISP model, so that the trained ISP model learns the feature representation of the RAW image shot by using low-speed imaging technology to the RGB image, and learns the model parameters matched with the feature representation. Thus, the trained ISP model can process the input RAW image and output the RGB image of the RAW image.

[0065] In addition to being able to be shot by using low-speed imaging technology, the RAW image can also be shot by using high-speed imaging technology by a high-speed camera. However, there is a significant difference between the degradation degree of the RAW image shot by using low-speed imaging technology and the degradation degree of the RAW image shot by the high-speed camera, which makes the above-mentioned ISP model difficult to effectively generalize to the noise and color distortion specific to the RAW image shot by the high-speed camera. When the above-mentioned ISP model is used to process the RAW image shot by the high-speed camera, the accuracy thereof is reduced, and thus the user perception quality of the RGB image output by the ISP model is reduced.

[0066] To solve the above technical problems, the embodiment of the present application provides a high-speed imaging image signal processing method and related device, using the RAW image photographed by the high-speed camera and the aligned reference image of the RAW image as the data set, the aligned reference image can be an RGB image, using these data sets to train and test the ISP model, so that the ISP model can learn the feature representation of the RAW image photographed by the high-speed camera to the RGB image, and learn the model parameters matched with the feature representation, so that when the ISP model is used to process the RAW image photographed by the high-speed camera, the ISP model can use the matched model parameters to process the RAW image, improve the accuracy and thus improve the user perception quality of the output RGB image.

[0067] And introduce a binary mask when calculating the first loss value of the ISP model, which can filter out the area between the aligned reference image and the RGB image output by the ISP model. The misalignment problem exists to reduce the influence of pixel misalignment between images on the performance of the ISP model, so that when the trained ISP model is used to process the RAW image, the accuracy can be improved and the user perception quality of the RGB image output by the ISP model can be improved. Misalignment problem can refer to pixel misalignment problem.

[0068] The high-speed imaging image signal processing method and related device provided by the embodiment of the present application will be described below with reference to the accompanying drawings. Please refer to Figure 1 which shows an optional flow of a high-speed imaging image signal processing method provided by the embodiment of the present application, which can include the following steps:

[0069] S101, obtain a plurality of image sets corresponding to different shooting scenes, the image set includes an RGB image and a RAW image photographed by a high-speed camera under the same shooting scene. Wherein, the RGB image can be photographed by a single-lens reflex camera.

[0070] In this embodiment, the RGB image and the RAW image in one image set are for the same shooting scene, and the shooting scenes corresponding to different image sets can be different. In order to reduce the difference between the RAW image and the RGB image, the high-speed camera and the single-lens reflex camera can simultaneously photograph the same shooting scene, and the field of view of the high-speed camera and the single-lens reflex camera when photographing can be kept consistent.

[0071] In a possible implementation, a coaxial imaging system is built for a high-speed camera and a single-lens reflex camera, the coaxial imaging system comprising: the high-speed camera, the single-lens reflex camera, a beam splitter, and a synchronous trigger. The high-speed camera can be a Revealer S1310C, and the high-speed camera is configured to capture RAW images by using high-speed imaging technology. The single-lens reflex camera can be a Canon EOS 90D, and the single-lens reflex camera is configured to capture RGB images, and the image quality of the RGB images is better than that of the RAW images. The beam splitter is configured to ensure that the fields of view of the high-speed camera and the single-lens reflex camera are consistent, for example, the high-speed camera and the single-lens reflex camera are fixed on an adjustable lifting platform and a translation platform, so as to accurately align the optical axes of the high-speed camera and the single-lens reflex camera with the intersection point of the beam splitter, so that the fields of view of the high-speed camera and the single-lens reflex camera are the same, so as to capture different quality images of the same field of view of the shooting object under the shooting scene. The synchronous trigger is configured to trigger the high-speed camera and the single-lens reflex camera to capture images at the same time. In addition, in order to minimize the influence of ambient light, the entire coaxial imaging system can be surrounded by a light shield plate, and only illumination from a predetermined direction is allowed.

[0072] In the process of acquiring the image set, the single-lens reflex camera can work in an aperture priority mode, for example, the fixed aperture is f / 11, and the minimum ISO (sensitivity) can be set to 100, so as to ensure that the single-lens reflex camera captures clear and low-noise RGB images, and the resolution of the RGB images can be 1024x1024. The high-speed camera outputs unprocessed 12-bit RAW images, and the resolution can be 1024x1024, and the aperture size randomly changes to increase the diversity of the RAW images.

[0073] In some examples, the high-speed camera can capture images at the same frame rate under different shooting scenes, so that each image set can include one RAW image and one RGB image, the RAW images in different image sets correspond to the same frame rate, and the corresponding ISP model can learn the feature representation of the RAW image to the RGB image under the degradation degree indicated by the frame rate and learn the model parameters matched with the feature representation.

[0074] In some examples, the high-speed camera can capture images at different frame rates under different shooting scenes, so that each image set can include one RAW image and one RGB image, but the RAW images in different image sets can correspond to different frame rates, so that the ISP model can learn the feature representation of the RAW image to the RGB image under the degradation degree indicated by the different frame rates and learn the model parameters matched with the different feature representations, so that the trained ISP model can process RAW images under different frame rates.

[0075] In some examples, the high-speed camera can capture images with different frame rates in the same shooting scene, so that each image set can include multiple RAW images and one RGB image, the multiple RAW images correspond to different frame rates, so that the ISP model can learn the feature representation of the RAW image to the RGB image under the degradation indicated by the different frame rates, and learn the model parameters matched with different feature representations, so that the trained ISP model can process RAW images under different frame rates. For example, the commonly used frame rates of high-speed cameras are 100, 1000, 2000, 3500, 5000 and 10000 frames per second (fps), so for each shooting scene, 6 RAW images are captured at frame rates of 100, 1000, 2000, 3500, 5000 and 10000 frames per second (fps). The exposure time of the high-speed camera under each frame rate can be set to the maximum value of the frame rate, and in order to reduce the error under different frame rates, the gain of the high-speed camera is adjusted synchronously with the frame rate.

[0076] It should be noted here that after obtaining the images captured by the high-speed camera and the single-lens reflex camera, the user manually deletes images with poor image quality, such as motion that may cause the captured images to be blurred in a motion scene. The user can manually delete these images. Of course, the present embodiment can also use other ways to delete images, such as determining the sharpness of the image based on at least one of a gradient algorithm, a frequency domain analysis and a statistical algorithm, and using the sharpness of the image to determine whether to delete the image. The present embodiment does not limit the way of deleting images.

[0077] S102, for any RAW image in an image set, generating an intermediate reference image of the RAW image, aligning the RGB image to the intermediate reference image to obtain an aligned reference image.

[0078] Although the RAW image and the RGB image in the same image set are the same shooting scene, and the field of view between the high-speed camera and the single-lens reflex camera is consistent at the time of shooting, there are still feature differences between the RAW image and the RGB image, so for any RAW image in the image set, the intermediate reference image of the RAW image and the aligned reference image are generated, wherein the difference between the intermediate reference image and the RAW image is that the color of the intermediate reference image is the same as that of the RGB image, so that the intermediate reference image reduces the color difference with the RGB image. Aligning the RGB image to the intermediate reference image can align the position of a pixel point in the RGB image to the position of the pixel point in the intermediate reference image, so that the same pixel point is close in position in the intermediate reference image and the aligned reference image. The aligned reference image is still an RGB image, so the pixel misregistration degree between the aligned reference image and the intermediate reference image is significantly less than the pixel misregistration degree between the RGB image and the intermediate reference image, thereby reducing the pixel misregistration degree between the aligned reference image and the intermediate reference image. The aligned reference image and the intermediate reference image can be regarded as an image pair that is not completely aligned.

[0079] In the present embodiment, the positions of the same pixel points in the RAW image and the intermediate reference image of the RAW image are the same, and the aligned reference image only changes the position of the pixel point in the image relative to the RGB image. Therefore, the aligned reference image is the same as the RGB image except for the position of the pixel point, and the ISP model is trained using the aligned reference image of the RAW image, which can reduce the influence of the pixel misregistration between the RAW image and the aligned reference image on the performance of the ISP model.

[0080] The intermediate reference image can be generated by a trained image mapper, wherein the trained image mapper learns the color difference features between the RAW image and the RGB image, and learns to restore an image with the same color as the RGB image using the color difference features, so that the image output by the image mapper only changes the color of the image relative to the RAW image. Therefore, the RAW image is input into the trained image mapper to obtain the RGB image output by the image mapper, which is denoted as the intermediate reference image.

[0081] One feasible way to align the RGB image to the intermediate reference image to obtain the aligned reference image is to apply a pre-trained optical flow network to estimate the spatial relationship between the intermediate reference image and the RGB image in the image set, and then perform a warping operation to generate the aligned reference image, which is as follows:

[0082] .

[0083] wherein, the aligned reference image, for an RGB image in the image set, for an intermediate reference image, for a pre-trained optical flow network, for a warping operation. The spatial relationship between the intermediate reference image and the RAW image is the same (the positions of the pixels in the image are the same), and aligning the intermediate reference image is actually aligning the RAW image, thereby reducing the misalignment between the pixels of the aligned reference image and the RAW image, and further reducing the degree of misalignment between the pixels of the aligned reference image and the RAW image.

[0084] S103, using the intermediate reference image and the aligned reference image of the same RAW image, to generate a binary mask of the RAW image. In this embodiment, the role of the binary mask is to filter out the areas with misalignment problems between images. The area with misalignment problems refers to the positional relationship between the pixels in the area. The positional relationship between the pixels in different images is problematic because of the sensor difference between the high-speed camera and the single-lens reflex camera, and the object movement and occlusion during shooting. These areas with misalignment problems will affect the performance of the ISP model, so this embodiment generates a binary mask for each RAW image according to the areas with misalignment problems between the intermediate reference image and the aligned reference image of the RAW image.

[0085] Because the intermediate reference image of the RAW image only changes the color relative to the RAW image, the binary mask of the RAW image is also applicable to the RAW image and the aligned reference image of the RAW image. In summary, by aligning the intermediate reference image, the alignment error between the RAW image and the aligned reference image of the RAW image caused by high noise and color difference is alleviated, a higher-precision matching of the RAW image and the aligned reference image is achieved, and the ISP model can learn the feature representation of the RAW image to a higher-precision RGB image.

[0086] In a possible implementation, the process of generating the binary mask of the RAW image can include the following steps as shown in Figure 2 .

[0087] S201, dividing the intermediate reference image and the aligned reference image according to a preset division manner, so as to divide the intermediate reference image into a plurality of first image blocks and divide the aligned reference image into a plurality of second image blocks.

[0088] In some examples, the preset partition manner can be to partition the image into a plurality of image blocks of the same size, whereby the intermediate reference image and the aligned reference image can be equally divided into a plurality of image blocks, but the size of the intermediate reference image and the aligned reference image needs to be adjusted to the same size before partition. In some examples, the preset partition manner can specify the size of each image block, and the image is partitioned according to the specified size, which is relatively complex in the implementation of personalized partition.

[0089] S202, extract the phase spectrum of each first image block and the phase spectrum of each second image block. The phase spectrum of each image block is extracted by using Fourier transform.

[0090] S203, for the first image block and the second image block with the same position, calculate the alignment score between the first image block and the second image block by using the phase spectrum of the first image block and the phase spectrum of the second image block. In this embodiment, the alignment score between the first image block and the second image block can be calculated by using a similarity function or a distance function, and one possible implementation is as follows:

[0091] .

[0092] wherein, is the alignment score, is a cosine similarity function; represents extracting the phase spectrum of the image block, specifically using discrete Fourier transform to extract the phase spectrum of the image block; represents extracting the feature from the pre-trained VGG network model; represents the size of the image block, and represents equally dividing the image, such as Discrete Fourier transform is used to extract the phase spectrum and VGG network model is used to extract the feature, which is an optional way and is not limited in this embodiment.

[0093] S204, in the case that the alignment score is less than or equal to a threshold value, determining that the element value of the first image block and the second image block in the binary mask is a first value, in the case that the alignment score is greater than the threshold value, determining that the element value of the first image block and the second image block in the binary mask is a second value, and in the case that the alignment score is less than or equal to the threshold value, the first image block and the second image block are the area with misalignment problem, and the first value is used to filter out the area with misalignment problem when calculating the first loss value.

[0094] For example, wherein is a threshold value, for example . is the element value of the i th element in the binary mask .

[0095] Although the pixel misalignment between the aligned reference image and the RAW image is reduced, the two images can be affected by non-rigid motion and occlusion, etc., so that there are still areas with misalignment problems between the two images. Therefore, the embodiment further identifies the areas with misalignment problems between the images by using the alignment score, so as to filter out these areas in the loss calculation and reduce the influence on the ISP model performance.

[0096] S104, using the RAW image in the image set and the aligned reference image of the RAW image, completing the training and testing of the ISP model. The training includes: inputting the RAW image into the ISP model to obtain the RGB image output by the ISP model, using the binary mask of the same RAW image, the aligned reference image and the RGB image output by the ISP model, calculating the first loss value, and adjusting the model parameters of the ISP model using the first loss value, so that the ISP model learns the feature representation of the RAW image to the RGB image and learns the model parameters matched with the feature representation.

[0097] Wherein, the aligned reference image is the reference image of the RAW image in the ISP model training process, and the binary mask is used to filter out the areas with misalignment problems between the images when calculating the first loss value. In the embodiment, the spatial relationship of the content in the RGB image output by the ISP model is the same as that in the RAW image, so the binary mask of a RAW image can be applied to the ISP model output RGB image and the aligned reference image. The binary mask is used to remove the areas with misalignment problems between the two images, and only the areas without misalignment problems are used to calculate the first loss value.

[0098] As a possible implementation, the first loss value is calculated by using the formula

[0099] The first loss value is calculated by using the formula is the RGB image of the RAW image output by the ISP model, is the aligned reference image of the RAW image, is the binary mask, and represents element-wise multiplication, is the L1 norm, so that only the areas without misalignment problems are considered in the process of calculating the first loss value.

[0100] ​In summary, the high-speed imaging signal processing method provided in this embodiment can use RAW images captured by a high-speed camera and aligned reference images of those RAW images as datasets. The aligned reference images can be RGB images. Using these datasets, the ISP model is trained and tested, enabling it to learn the feature representations from the RAW images captured by the high-speed camera to RGB images, and to learn model parameters matching these feature representations. Thus, when processing RAW images captured by a high-speed camera, the ISP model can use the matching model parameters, improving accuracy and consequently enhancing the user-perceived quality of the output RGB images. The aligned reference images of the RAW images can be obtained by aligning RGB images captured by a DSLR camera to an intermediate reference image. This means that the image quality of the aligned reference images of the RAW images is close to or the same as that of the RGB images captured by the DSLR camera. Therefore, training using the aligned reference images of the RAW images allows the ISP model to learn the feature representations from the RAW images to high-quality RGB images, resulting in RGB images recovered by the ISP model with image quality close to or the same as those captured by the DSLR camera.

[0101] Furthermore, a binary mask is introduced when calculating the first loss value of the ISP model. This binary mask can filter out areas where there is misalignment between the aligned reference image and the RGB image output by the ISP model when calculating the first loss value, thereby reducing the impact of pixel misalignment between images on the performance of the ISP model. As a result, when using the trained ISP model to process RAW images, the accuracy can be improved and the user-perceived quality of the RGB image output by the ISP model can be improved.

[0102] In addition, the above Figure 1 The high-speed imaging image signal processing method shown can train an ISP model using a first loss value calculated from the phase spectra of two images. The method can also incorporate the amplitude spectra of the two images during ISP model training to consider the different sensitivities of the phase and amplitude spectra to misalignment and image degradation (such as noise and blur). Correspondingly, the training of the ISP model in the high-speed imaging image signal processing method provided in this embodiment can further include: extracting the first amplitude spectrum of an aligned reference image of the RAW image; extracting the second amplitude spectrum of the RGB image of the RAW image output by the ISP model; calculating the Walstein distance between the first and second amplitude spectra using the slice Walstein distance function, and using the Walstein distance as the second loss value; correspondingly, adjusting the model parameters of the ISP model using the first loss value includes: calculating the loss value of the ISP model using the first and second loss values, and adjusting the model parameters of the ISP model using the loss value.

[0103] For example, using formulas calculating a second loss value , denotes a sliced Wassertein distance function, denotes a magnitude spectrum of the extracted image. After obtaining the second loss value, the ISP model is trained by using calculating a loss value of the ISP model, is a balance value used to balance the proportion of the two loss values, which is a constant and is not limited in the embodiment. By introducing the second loss value, the noise influence can be effectively suppressed and the clarity of the image can be enhanced, so as to further improve the user perception quality of the RGB image output by the ISP model.

[0104] In some examples, the high-speed imaging image signal processing method provided by the embodiment can further input the gain corresponding to the RAW image into the ISP model when the RAW image is input into the ISP model, the gain being related to the frame rate of the RAW image, so as to determine the degradation degree of the RAW image through the gain, thereby enabling the ISP model to use more parameters to restore the RGB image and improve the user perception quality of the RGB image. Especially in the scene of different frame rates, the ISP model can determine the degradation degree through the gain, so that the ISP model does not need to learn the degradation degree through a large number of additional parameters, thereby enabling the ISP model to focus on image restoration. Correspondingly, the gain corresponding to the RAW image also needs to be input when the trained ISP model is used to process the RAW image, and the process includes steps 1 to 4:

[0105] Step 1: capturing a RAW image to be processed by a high-speed camera.

[0106] Step 2: inputting the RAW image to be processed and the gain of the high-speed camera when capturing the RAW image to be processed into the ISP model;

[0107] Step 3: determining the degradation degree of the RAW image to be processed by the ISP model through the gain;

[0108] Step 4: processing the RAW image to be processed by the ISP model guided by the degradation degree of the RAW image to be processed, to obtain the RGB image output by the ISP model.

[0109] Please refer to Figure 3 which shows another optional flow of the high-speed imaging image signal processing method provided by the embodiment, which can include the following steps:

[0110] S301, acquiring a plurality of image sets corresponding to different shooting scenes, the image set including one RGB image under the same shooting scene and a plurality of RAW images captured by a high-speed camera at different frame rates. The RGB image can be captured by a single-lens reflex camera.

[0111] S302, for any image set, generating the intermediate reference image of the RAW image, aligning the RGB image to the intermediate reference image to obtain the aligned reference image.

[0112] S303, using the intermediate reference image and the aligned reference image of the same RAW image, generating the binary mask of the RAW image.

[0113] S304, for any RAW image, calculating the alignment score of the intermediate reference image and the aligned reference image of the RAW image. In some examples, the alignment score can refer to the above-mentioned Figure 2 The alignment score of any image block in the intermediate reference image and the aligned reference image is calculated, and the sum of the alignment scores of all image blocks is taken as the alignment score of the intermediate reference image and the aligned reference image. Of course, the intermediate reference image and the aligned reference image can also calculate the alignment score in other ways, which will not be introduced here.

[0114] S305, taking the sum of the alignment scores of all RAW images in the same image set as the alignment score of the image set.

[0115] S306, sorting the alignment scores of all image sets in descending order of alignment score.

[0116] S307, using the image set sorted in the front of the preset proportion to generate the test data set of the ISP model, using the image set sorted after the preset proportion to generate the training data set of the ISP model, any data set in the test data set and the training data set includes multiple RAW images and aligned reference images of the RAW images taken at different frame rates, the training data set is used to train the ISP model, and the test data set is used to test the trained ISP model. For example, using the image set sorted in the front of 15% to generate the test data set, and using the image set sorted after 15% to generate the training data set.

[0117] In this embodiment, steps S301 to S303 are the same as steps S101 to S103 described above, which will not be described here. In addition, in the case of calculating the alignment score in other ways in step S304, steps S304 to S307 can be processed in parallel with steps S302 and S303.

[0118] S308, using the RAW images and the aligned reference images in the training data set to train the ISP model, and using the RAW images and the aligned reference images in the test data set to test the ISP model.

[0119] The training includes: inputting a RAW image into the ISP model to obtain an RGB image output by the ISP model, calculating a first loss value by using a binary mask of the RAW image, an aligned reference image and the RGB image output by the ISP model, and adjusting the model parameters of the ISP model by using the first loss value, so that the ISP model learns the feature representation of the RAW image to the RGB image and learns the model parameters matched with the feature representation. For details of the specific training process, please refer to the above Figure 1 As shown in the figure, the test process is the same as the existing test process, which will not be described here.

[0120] In some examples, the image set includes one RGB image and multiple RAW images taken by a high-speed camera at different frame rates under the same shooting scene, and a feasible way to generate the intermediate reference image of the RAW image and align the RGB image to the intermediate reference image is as follows:

[0121] The trained image mapper is used to process each RAW image in the image set to obtain the intermediate reference image of each RAW image output by the image mapper. The image mapper is trained by using a specific RAW image in the image set and the RGB image in the image set, so that the image mapper learns the color difference feature between the specific RAW image and the RGB image, and learns to restore an image with the same color as the RGB image by using the color difference feature. The specific RAW image is a RAW image taken at a specific frame rate in the image set; and the spatial relationship between the intermediate reference image of the specific RAW image and the RGB image is estimated, and the aligned reference image of each RAW image in the image set to which the specific RAW image belongs is generated by using the spatial relationship. In some examples, the image mapper can be trained before the image set is obtained. For example, multiple data sets of the image mapper are obtained, any data set includes a RAW image and an RGB image corresponding to the same shooting scene, the RAW image is taken by a high-speed camera at a specific frame rate, the RGB image is taken by a single-lens reflex camera, and the image mapper is trained by using the multiple data sets, so that the image mapper learns the color difference feature between the specific RAW image and the RGB image, and learns to restore an image with the same color as the RGB image by using the color difference feature. The specific RAW image is a RAW image taken at a specific frame rate in the image set.

[0122] The specific frame rate can be the minimum frame rate of the high-speed camera, and the image quality of the RAW image taken at the minimum frame rate is optimal. The image mapper includes an encoder and a decoder . The image mapper learns the color difference feature between and , and outputs a one-dimensional vector , is denoted as:

[0123] ;

[0124] RAW images taken by a high-speed camera, in training the image mapper can be specific to a particular RAW image. To reduce computational cost, down-sampled images of and are used as inputs to , as and are down-sampled by a factor of 2. Subsequently, the decoder maps to a color-difference-reduced intermediate reference image through a series of differentiable, prior-guided transformation functions (including brightness adjustment, white balance, color correction, gamma transformation, and tone mapping, etc.), as follows:

[0125] ;

[0126] where is a set of fixed, untrainable initialization parameters, has the same dimension as , and the residual learning formulation stabilizes the training and accelerates the convergence. Here, is a scalar numerical gain for brightness correction, is a white balance multiplier for R, G, B channels, is a color correction matrix, is a gamma correction exponent. The tone mapping function is a sigmoid curve defined as:

[0127] ;

[0128] where is a trainable parameter that controls the shape of the sigmoid curve; denotes the max function, while is a small positive number to prevent instability caused by values close to zero, e.g. can take 1e-8. Thus, the parameter set of the image mapper is denoted as . Thus, the image mapping process of the image mapper is:

[0129] ;

[0130] where are the learnable parameters of the image mapper. Since and The spatial relationship between them is dislocated, and direct pixel-by-pixel supervision is not feasible, so in order to train the image mapper, the image mapper can use a global histogram matching loss and a total variation loss to calculate the loss value of the image mapper, as follows:

[0131] wherein is the loss value of the image mapper, is an empirical value, such as a balance weight that can be set to 0.005. The global histogram matching loss encourages global color consistency, which is defined as:

[0132]

[0133] wherein represents a differentiable histogram calculation, and the cumulative distribution function is calculated. is an L1 norm used to calculate the difference between elements. Because the decoder only adjusts the color of the image, the global histogram loss ensures that the mapped image is perceptually aligned with the original image while effectively reducing color and brightness differences.

[0134] The total variation loss used to suppress noise is defined as: wherein and are horizontal and vertical gradient operators, respectively. The loss value calculated using the above formula is used to train the image mapper, so that the image mapper learns the color difference features between the specific RAW image and the RGB image, and learns to restore an image with the same color as the RGB image using the color difference features. It should be noted that the loss value used by the image mapper is only an example, and the present embodiment is not limited thereto.

[0135] After the image mapper is trained, the trained image mapper is used to process each RAW image in the image set to obtain an intermediate reference image output by the image mapper for each RAW image. The intermediate reference image for each RAW image only changes the color of the image relative to the RAW image.

[0136] Because the capture interval of the high-speed camera is extremely short, the motion between the RAW images in the same image set can be ignored, so the spatial relationship in the same image set can be reused. Based on this, the present embodiment can estimate the spatial relationship between the intermediate reference image of a specific RAW image and the RGB image, and the estimated spatial relationship is reused to align the intermediate reference images of other RAW images to generate aligned reference images of the other RAW images. The generation process can be using the formula​​ The alignment strategy, not described in detail here, is an anchor-based alignment strategy by which the influence of noise is reduced.

[0137] The following describes the high-speed imaging image signal processing method provided by the present application, Figure 4 The optional structure of the image mapper and the ISP model is shown in Figure 4 The image mapper in Figure 4 can be trained using RAW images captured by a high-speed camera at the lowest frame rate and RGB images captured by a single-lens reflex camera. The RAW image captured by the high-speed camera is defined as , and the RGB image captured by the single-lens reflex camera is defined as The goal of the ISP model is to learn the mapping from to , which can be represented as: where are the learnable parameters (i.e., the model parameters to be trained) of the ISP model, and is the gain used by the high-speed camera.

[0138] The training process of the ISP model is shown in Figure 4 , and the training process shown in Figure 4 can be named as MisISP training architecture. In Figure 4 , the RAW image captured by the high-speed camera is input into the encoder of the image mapper after being demosaiced and downsampled, and the RGB image captured by the single-lens reflex camera is input into the encoder of the image mapper after being downsampled, and a one-dimensional vector is output by the encoder. The one-dimensional vector and the demosaiced RAW image are input into the decoder of the image mapper to obtain the intermediate reference image output by the decoder. The RGB image captured by the single-lens reflex camera is aligned to the intermediate reference image to obtain the aligned reference image . After the intermediate reference image and the aligned reference image of the same RAW image are divided into multiple image blocks, the alignment score of each image block is calculated to generate the binary mask of the RAW image using the alignment score.

[0139] The RAW image captured by the high-speed camera and the gain used when capturing the RAW image are input into the ISP model, and the gain is used to determine the degree of degradation of the RAW image. The ISP model outputs the RGB image of the RAW image.Then, the first loss value is calculated by using the RGB image of the RAW image, the binary mask of the RAW image and the aligned reference image of the RAW image output by the ISP model, the second loss value is calculated by using the RGB image of the RAW image and the aligned reference image of the RAW image, and the loss value of the ISP model is obtained by using the first loss value and the second loss value. The ISP model is trained by using the loss value. For details, please refer to the description above, which will not be described here.

[0140] To evaluate the high-speed imaging image signal processing method provided by the embodiments of the present application, the applicant has conducted rich experiments. All experiments are implemented using the PyTorch framework and trained on two NVIDIA GeForce RTX 3090 graphics processing units (GPUs). For the training of the ISP model, the Adam optimizer is used, and the initial learning rate is , and the learning rate is gradually reduced by using the cosine annealing strategy. The training process of the ISP model is iterated for 300,000 times, and each time a batch of 16 RAW images is used, which are randomly cropped into 128×128 image blocks as input. For the image mapper, since only the least degraded RAW images are used, the number of training iterations is reduced to 100,000, and random cropping is omitted to better maintain overall color consistency. In addition, in order to evaluate the image quality, three widely used indicators are adopted, including the peak signal-to-noise ratio (PSNR), the multi-scale structural similarity index measure (MS-SSIM) and the learned perceptual image patch similarity (LPIPS).

[0141] The experiment is mainly carried out on the RHID dataset, which includes multiple RAW images captured by a high-speed camera at different frame rates and RGB images captured by a single-lens reflex camera. The high-speed camera and the single-lens reflex camera are located in a coaxial imaging system. The high-speed camera can capture 6 RAW images of the same shooting scene at frame rates of 100, 1000, 2000, 3500, 5000 and 10000 frames per second (fps). The RHID dataset can cover various contents captured at different times of the day and weather conditions, including buildings, roads, vegetation, water bodies, vehicles and text, etc. As in the experiment, a total of 12,375 scenes of data were captured under different lighting and environmental conditions. The RHID dataset includes 240,480 pairs of images for training and 42,432 pairs of images for testing. In addition, the existing ISP datasets S7, ZRR and MAI are also used for ISP model training to illustrate the necessity of using high-speed imaging datasets. The experimental process and experimental results are as follows:

[0142] Figure 5 The above Figure 4 The processing effect of the ISP model obtained by the MisISP training architecture shown in Figure 5 The RAW in is a RAW image captured by a high-speed camera under different shooting scenes, Figure 5 The Reference in is an RGB image captured by a single-lens reflex camera, Figure 5 Ours in records the RGB image output by the ISP model after processing the RAW image captured by the high-speed camera. As shown in Figure 5 It can be seen that the image quality of the RGB image output by the ISP model is close to that of the RGB image captured by the single-lens reflex camera, effectively improving the user perception quality.

[0143] Necessity of RHID dataset: three existing ISP models are used, including LiteISPNet, MetaISP and ConvIR. The S7 dataset, ZRR dataset, MAI dataset and RHID dataset are used to train the three existing ISP models, respectively. The performance of the three ISP models on the S7 dataset, ZRR dataset, MAI dataset and RHID dataset is evaluated. In order to compare fairly, all ISP models are trained using the same learning strategy and tested on RAW images captured by a high-speed camera. The visual results are shown in Figure 6 Figure 6 ​RAW is a RAW image captured by a high-speed camera, Bulid-ISP is an image output by an ISP built in the high-speed camera, Reference is an RGB image captured by a single-lens reflex camera, and the other images are processing results of the RAW image by three ISP models respectively trained by using the four data sets. From Figure 6 It can be seen that although the S7 data set, the ZRR data set and the MAI data set contain exposure-related degradation to different degrees, it is difficult to generalize to the complex noise pattern and color distortion inherent in the RAW image captured by using the high-speed imaging technology, and the RGB image output by the model trained by using these data sets has obvious noise and color distortion, as shown in the part of the red box in Figure 6 It can be seen that although the S7 data set, the ZRR data set and the MAI data set contain exposure-related degradation to different degrees, it is difficult to generalize to the complex noise pattern and color distortion inherent in the RAW image captured by using the high-speed imaging technology, and the RGB image output by the model trained by using these data sets has obvious noise and color distortion, as shown in the part of the red box in

[0144] Table 1 shows the comparison results of the ISP models trained by using different data sets, and it can be seen from the comparison results shown in Table 1 that among the ISP models trained by using the S7 data set, the ZRR data set and the MAI data set, the ISP model trained by using the S7 data set has the best performance; however, compared with the model trained by using the RHID data set, the model trained by using the S7 data set still shows a PSNR drop of 4.55 dB on average.

[0145] Table 1 shows the comparison results of the ISP models trained by using different data sets, and it can be seen from the comparison results shown in Table 1 that among the ISP models trained by using the S7 data set, the ZRR data set and the MAI data set, the ISP model trained by using the S7 data set has the best performance; however, compared with the model trained by using the RHID data set, the model trained by using the S7 data set still shows a PSNR drop of 4.55 dB on average.

[0146]

[0147] Effectiveness of MisISP training framework: To demonstrate the effectiveness of MisISP training framework, MisISP training framework is applied to several representative state-of-the-art ISP models and image restoration models, including LiteISPNet, SYENet, MetaISP, FourierISP, ConvIR and TaylorFormer, and two training methods are adopted for these models, one is to train directly on RHID dataset (labeled as Baseline), and the other is to train using RHID dataset and MisISP framework (labeled as Ours). The performance comparison of the models is shown in Table 2, and it can be known from Table 2 that compared with the ISP built-in high-speed camera, all models trained on RHID dataset have achieved significant improvement, and the PSNR is increased by at least 3.76dB. When further combined with MisISP framework, the PSNR of these models is increased by at least 2.83dB, and the performance improvement is mainly because MisISP training framework can alleviate the pixel misalignment of image pairs (i.e. the area with misalignment problem can be filtered out during training), effectively reduce blur and improve reconstruction quality. Figure 7 A visual comparison is provided, which compares mainly in Figure 7 the part shown in the middle red box, from Figure 7 it can be known that the ISP built-in high-speed camera is difficult to suppress noise, and the image output by the high-speed camera appears underexposure and color distortion. Compared with the ISP built-in high-speed camera, the image output by the model trained on the RHID dataset although the texture is severely missing, but the brightness and color fidelity of the image are improved and the noise is removed. When enhanced using MisISP training framework, the texture of the image output by all models is clearer, and the image is close to the RGB image (Reference) taken by the single-lens reflex camera, effectively proving the effectiveness of MisISP training structure. Figure 7

[0148] Table 2 Comparison results of ISP models trained directly on RHID dataset and models trained using MisISP framework

[0149]

[0150] ​Generalization of MisISP training framework: Although MisISP training framework is proposed based on RHID dataset, MisISP training framework can be well generalized to other existing datasets for training ISP models. For example, existing ZRR dataset, although pre-aligned and manually selected, still has obvious pixel misalignment between paired images. To further verify the effectiveness of MisISP training structure, experiments are conducted on ZRR dataset. According to the alignment score, the top 1000 most aligned image pairs are selected from ZRR dataset as the test set, and the remaining image pairs are used for training. Experiments can be done on LiteISPNet, SYENet, MetaISP, FourierISP, ConvIR and TaylorFormer models. Two training methods are used for these models, one is to train directly on ZRR dataset (labeled as Baseline), and the other is to train using ZRR dataset and MisISP framework (labeled as Ours). The performance comparison of the models is shown in Table 3, and the visual comparison is shown in Figure 8 Table 2 and Figure 8 It can be seen from Table 2 and

[0151] Table 3 Comparison of ISP models trained directly on ZRR dataset and models trained using MisISP framework

[0152]

[0153] In practical applications on moving objects: Motion blur is easy to occur when using single-lens reflex cameras to shoot moving objects. Therefore, in order to obtain high-quality RGB images as reference images, scenes with dynamic objects are avoided as much as possible when collecting RHID dataset. Although the high-speed motion content contained in RHID dataset is limited, the ISP model trained using the proposed MisISP training framework and RHID dataset is still effective for actual high-speed application scenarios. For this purpose, a high-speed camera is used to capture several scenes containing fast-moving objects at a frame rate of 10,000 fps and process them using the built-in ISR of the high-speed camera. As shown in Figure 9 Fig. 6, the images captured by the single-lens reflex camera Figure 9In contrast, the ISP model trained with the MisISP training framework on the RHID dataset successfully reconstructs clear images of moving objects even at 10,000 fps, and the output images have the same quality as the images captured by the DSLR, and the clarity of moving objects can be better than that of the DSLR.

[0154] Ablation experiments: The PMIA, and individual contributions to the MisISP training framework were evaluated in Table 4. The baseline model is LiteISPNet trained on the RHID dataset using L1 norm to calculate the loss value, i.e., the ISP model before introducing PMIA, and The performance of the ISP model is significantly improved after introducing PMIA, indicating that PMIA effectively alleviates the pixel misalignment (also known as pixel bias) and improves image fidelity. The introduction of MisISP further suppresses the adverse effects of misaligned pixels (i.e., pixels in regions with misalignment issues), thereby improving image quality. At the same time, reduces the over-smoothing of texture details, thereby significantly improving the user-perceived quality of the output images (measured by LPIPS). These results show that PMIA, and in the MisISP training framework substantially improve the performance of the ISP model.

[0155] Table 4 Ablation experiments of MisISP

[0156]

[0157] The impact of different implementation details in the MisISP training framework on performance was evaluated in Table 5.

[0158] Effectiveness of image mapper: The image mapper reduces the difference between the RAW image and the reference image, thereby effectively improving the performance of alignment. In the implementation process, the alignment results with and without the image mapper were compared, and the quantitative results obtained by deleting the image mapper are reported in Table 5 (see the row marked “w / o ”).

[0159] Effectiveness of the alignment strategy: In the MisISP training architecture, an anchor-based alignment strategy (AAS) is used for alignment. This involves selecting the spatial relationship between the intermediate reference image of the RAW image with the lowest frame rate and the RGB image captured by the DSLR camera as the alignment reference. As shown in Table 5 (see the row labeled "w / o AAS"), when the intermediate reference image of the RAW image captured at each frame rate is aligned with the reference image (here, the reference image is a high-quality RGB image), while keeping all other training conditions unchanged, a significant decrease in the comparison metrics (PSNR and MS-SSIM) can be observed. This decrease is mainly due to the increased noise level in the RAW images obtained under high gain and short exposure conditions. Furthermore, the image mapper is designed to adjust brightness and color rather than remove noise, thus the alignment accuracy decreases under these noisy conditions.

[0160] Comparison of Optical Flow Consistency: In the field of optical flow estimation, optical flow consistency (OFC) is a commonly used metric to measure the accuracy of flow prediction. OFC can also be used to generate binary masks in regions with misalignment problems. However, OFC often fails in weakly textured regions. Table 5 (see the table marked "...") → The line showing "" indicates that OFC was used to generate the binary mask instead of The performance of the ISP model further confirms this limitation. In contrast, the proposed... Evaluating bias based on the similarity between phase spectra avoids this error and provides more reliable supervision.

[0161] Gain effectiveness: The gain associated with each RAW image during training. Used as additional input to the ISP model. As shown in Table 5 (see label "w / o") (The line containing ""), delete This will lead to a decrease in all evaluation metrics. During the reasoning process, The value can be provided automatically by the high-speed camera or adjusted manually by the user. Figure 9 Shows different manual settings The impact of the value on the output. A larger value... This value produces a strong noise reduction effect, but may also suppress fine image details. In practical applications, high-speed cameras can provide a reasonable default value, which users can then fine-tune according to their personal preferences.

[0162] Table 5. Performance impact of different implementation details

[0163]

[0164] The above introduces a high-speed imaging image signal processing method provided by the embodiment of the application. The following introduces a device for executing the high-speed imaging image signal processing method.

[0165] Please refer to Figure 10 , Figure 10 FIG. 1 is a structural schematic diagram of a high-speed imaging image signal processing device provided by the embodiment of the application. The specific description of each unit in the high-speed imaging image signal processing device can be referred to the above method embodiment. As shown in FIG. 1, the high-speed imaging image signal processing device can include an acquisition unit 10, a first generation unit 20, a second generation unit 30, and a training unit 40. Figure 10

[0166] The acquisition unit 10 is configured to acquire a plurality of image sets corresponding to different shooting scenes, wherein each image set includes an RGB image and a RAW image captured by a high-speed camera under the same shooting scene.

[0167] The first generation unit 20 is configured to generate an intermediate reference image of the RAW image in any image set, align the RGB image to the intermediate reference image to obtain an aligned reference image, and the difference between the intermediate reference image and the RAW image is that the color of the intermediate reference image is the same as that of the RGB image.

[0168] The second generation unit 30 is configured to generate a binary mask of the RAW image by using the intermediate reference image and the aligned reference image of the RAW image.

[0169] The training unit 40 is configured to complete the training and testing of the image signal processing model by using the RAW image and the aligned reference image of the RAW image in the image set, and the training includes: inputting the RAW image into the image signal processing model to obtain an RGB image output by the image signal processing model, calculating a first loss value by using the binary mask, the aligned reference image, and the RGB image output by the image signal processing model of the same RAW image, adjusting the model parameters of the image signal processing model by using the first loss value, so that the image signal processing model learns the feature representation of the RAW image to the RGB image and learns the model parameters matched with the feature representation, and the binary mask is used to filter out the area with misalignment problem between images when calculating the first loss value.

[0170] In a possible implementation, the second generation unit 30 generates the binary mask of the RAW image by using the intermediate reference image and the aligned reference image of the RAW image, and the method includes:

[0171] ​The intermediate reference image and the aligned reference image are divided according to a preset division mode, so as to divide the intermediate reference image into a plurality of first image blocks and divide the aligned reference image into a plurality of second image blocks; a phase spectrum of each first image block and a phase spectrum of each second image block are extracted; for the first image block and the second image block at the same position, an alignment score between the first image block and the second image block is calculated by using the phase spectrum of the first image block and the phase spectrum of the second image block; in a case where the alignment score is less than or equal to a threshold value, it is determined that an element value of the first image block and the second image block in a binary mask is a first value, in a case where the alignment score is greater than the threshold value, it is determined that the element value of the first image block and the second image block in the binary mask is a second value, and in the case where the alignment score is less than or equal to the threshold value, the first image block and the second image block are a region with a misalignment problem, and the first value is used to filter out the region with the misalignment problem when the first loss value is calculated.

[0172] In a possible implementation, the training unit 40 calculates the first loss value by using the binary mask of the same RAW image, the aligned reference image of the RAW image, and the RGB image output by the image signal processing model.

[0173] The first loss value is calculated by using the formula , is the RGB image output by the image signal processing model for the RAW image, is the aligned reference image of the RAW image, is the binary mask, and represents element-wise multiplication, is the L1 norm.

[0174] In a possible implementation, the training unit 40 is further configured to extract a first amplitude spectrum of the aligned reference image of the RAW image, extract a second amplitude spectrum of the RGB image output by the image signal processing model for the RAW image, calculate a Wassertain distance between the first amplitude spectrum and the second amplitude spectrum by using a slice Wassertain distance function, and take the Wassertain distance as a second loss value. The adjusting of the model parameters of the image signal processing model by using the first loss value includes: calculating a loss value of the image signal processing model by using the first loss value and the second loss value, and adjusting the model parameters of the image signal processing model by using the loss value.

[0175] In a possible implementation, when the training unit 40 inputs the RAW image into the image signal processing model, the training unit 40 inputs a gain corresponding to the RAW image into the image signal processing model, the gain is related to a frame rate adopted by the RAW image, and the degradation degree of the RAW image is determined by the gain.

[0176] ​In a possible implementation, the obtaining unit 10 in the high-speed imaging image signal processing apparatus is further configured to capture a RAW image to be processed by using the high-speed camera; the high-speed imaging image signal processing apparatus further includes a processing unit configured to input the RAW image to be processed and a gain used by the high-speed camera when capturing the RAW image to be processed into the image signal processing model; the image signal processing model is configured to determine a degradation degree of the RAW image to be processed according to the gain; and the image signal processing model is configured to process the RAW image to be processed according to the degradation degree of the RAW image to be processed, to obtain an RGB image output by the image signal processing model.

[0177] In a possible implementation, the image set includes one RGB image and multiple RAW images captured by using the high-speed camera at different frame rates in the same shooting scene; and the high-speed imaging image signal processing apparatus further includes a calculation unit, a sorting unit and a data set generation unit. The calculation unit is configured to calculate, for any RAW image, an alignment score of an intermediate reference image and an aligned reference image of the RAW image, and sum up the alignment scores of all RAW images in the same image set as an alignment score of the image set; the sorting unit is configured to sort the alignment scores of all image sets in descending order of the alignment scores; and the data set generation unit is configured to generate a test data set of the image signal processing model by using image sets in a preset proportion in front of the sorting, and generate a training data set of the image signal processing model by using image sets after the preset proportion in the sorting, wherein any data set in the test data set and the training data set includes multiple RAW images captured at different frame rates and the aligned reference images of the RAW images, the training data set is used for training the image signal processing model, and the test data set is used for testing the trained image signal processing model.

[0178] In a possible implementation, the image set includes one RGB image and multiple RAW images captured by using the high-speed camera at different frame rates in the same shooting scene; and the image signal processing model can learn a feature representation of the RAW image to the RGB image under a degradation degree indicated by the different frame rates; and the first generation unit 10 generates, for any RAW image in any image set, an intermediate reference image of the RAW image, and aligns the RGB image to the intermediate reference image to obtain an aligned reference image.

[0179] The trained image mapper is used to process each RAW image in the image set to obtain an intermediate reference image of each RAW image output by the image mapper, the image mapper being trained by using a specific RAW image in the image set and an RGB image in the image set, so that the image mapper learns the color difference feature between the specific RAW image and the RGB image and learns to restore an image with the same color as the RGB image by using the color difference feature, the specific RAW image being a RAW image captured at a specific frame rate in the image set; a spatial relationship between the intermediate reference image of the specific RAW image and the RGB image is estimated, and the RGB image is aligned to the intermediate reference image by using the spatial relationship, to obtain an aligned reference image of each RAW image in the image set to which the specific RAW image belongs.

[0180] The embodiments of the present application also provide an electronic device. Referring to Figure 11 FIG. 1 shows a structural schematic diagram of an electronic device suitable for implementing the electronic device in the embodiments of the present application. The electronic device in the embodiments of the present application can include but is not limited to, for example, a mobile phone, a notebook computer, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a camera, etc. Figure 11 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0181] As shown in Figure 11 The electronic device can include a processing device (such as a graphic processor, etc.) 401, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded from a storage device 408 to a random access memory (RAM) 403. In the state that the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0182] Generally, the following devices can be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a memory card, a hard disk, etc.; and a communication device 409. The communication device 409 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 11The electronic device with various apparatuses is shown, but it should be understood that all the shown apparatuses are not required, but instead can be implemented with more or less apparatuses. The processing apparatus 401 can be regarded as a processor in the electronic device, the ROM 402, the RAM 403 and the storage apparatus 408 can be regarded as memories in the electronic device, which are used to store computer programs; the processor is used to execute the computer programs, so that the electronic device can implement any of the high-speed imaging image signal processing methods provided in the embodiments of the present application.

[0183] The embodiments of the present application further provide a computer program product, which includes computer readable instructions, and when the computer readable instructions run on an electronic device, the electronic device can implement any of the high-speed imaging image signal processing methods provided in the embodiments of the present application.

[0184] The embodiments of the present application further provide a computer readable storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement any of the high-speed imaging image signal processing methods provided in the embodiments of the present application.

[0185] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments. In addition, in the apparatus embodiments provided in the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0186] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special integrated circuit, special CPU, special memory, special component, etc. Generally, any function completed by computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, software program implementation is a better embodiment. Based on such understanding, the technical solution of the application or the part of the application which makes contribution to the prior art can be embodied in the form of software product, which is stored in readable storage medium, such as computer floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a plurality of instructions for making a computer device (which can be personal computer, training device or network device, etc.) execute the method described in various embodiments of the application.

[0187] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be achieved in the form of a computer program product, entirely or partially.

[0188] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiments of the application is generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

Claims

1. A high speed imaging image signal processing method, characterized by, The method comprises the following steps: acquiring a plurality of image sets corresponding to different shooting scenes, wherein each image set comprises an RGB image and a RAW image captured by a high-speed camera under the same shooting scene; generating an intermediate reference image of the RAW image in any image set, aligning the RGB image to the intermediate reference image to obtain an aligned reference image, wherein the difference between the intermediate reference image and the RAW image is that the color of the intermediate reference image is the same as that of the RGB image; generating a binary mask of the RAW image by using the intermediate reference image and the aligned reference image of the RAW image; training and testing an image signal processing model by using the RAW image and the aligned reference image of the RAW image in the image set, wherein the training comprises: inputting the RAW image into the image signal processing model to obtain an RGB image output by the image signal processing model, calculating a first loss value by using the binary mask, the aligned reference image, and the RGB image output by the image signal processing model, and adjusting the model parameters of the image signal processing model by using the first loss value, so that the image signal processing model learns the feature representation of the RAW image to the RGB image and learns the model parameters matched with the feature representation, wherein the binary mask is used to filter out the area with misalignment problems between images when the first loss value is calculated.

2. The method of claim 1, wherein, The method further comprises the following steps: dividing the intermediate reference image and the aligned reference image according to a preset division manner to divide the intermediate reference image into a plurality of first image blocks and divide the aligned reference image into a plurality of second image blocks; extracting the phase spectrum of each first image block and the phase spectrum of each second image block; calculating the alignment score between the first image block and the second image block by using the phase spectrum of the first image block and the phase spectrum of the second image block when the first image block and the second image block are located at the same position; determining the element value of the first image block and the second image block in the binary mask as a first value when the alignment score is less than or equal to a threshold value, and determining the element value of the first image block and the second image block in the binary mask as a second value when the alignment score is greater than the threshold value, wherein the first image block and the second image block are the area with misalignment problems when the alignment score is less than or equal to the threshold value, and the first value is used to filter out the area with misalignment problems when the first loss value is calculated.

3. The method according to claim 1 or 2, characterized in that, The method further comprises the following steps: The first loss value is calculated using the formula is the RGB image of the RAW image output by the image signal processing model, is the aligned reference image of the RAW image, is the binary mask, and ⊙ denotes element-wise multiplication, is the L1 norm.​​ 4. The method of claim 3, wherein, extracting the first amplitude spectrum of the aligned reference image of the RAW image and extracting the second amplitude spectrum of the RGB image output by the image signal processing model; calculating a Wulsten distance between the first amplitude spectrum and the second amplitude spectrum by using a sliced Wulsten distance function, and taking the Wulsten distance as a second loss value; the adjusting the model parameters of the image signal processing model by using the first loss value comprises: calculating a loss value of the image signal processing model by using the first loss value and the second loss value, and adjusting the model parameters of the image signal processing model by using the loss value.

5. The method of claim 3, wherein, when the RAW image is input into the image signal processing model, a gain corresponding to the RAW image is input into the image signal processing model, the gain is related to a frame rate adopted by the RAW image, and a degradation degree of the RAW image is determined by the gain.

6. The method of claim 5, wherein, The method further comprises: taking a RAW image to be processed by the high-speed camera; inputting the RAW image to be processed and a gain adopted by the high-speed camera when taking the RAW image to be processed into the image signal processing model; determining a degradation degree of the RAW image to be processed by the image signal processing model through the gain; processing the RAW image to be processed by the image signal processing model guided by the degradation degree of the RAW image to be processed to obtain an RGB image output by the image signal processing model.

7. The method of claim 1, wherein, The image set comprises one RGB image and multiple RAW images taken by the high-speed camera with different frame rates under the same shooting scene, and the method further comprises: for any RAW image, calculating an alignment score of an intermediate reference image and an aligned reference image of the RAW image; taking a sum of the alignment scores of all RAW images in the same image set as an alignment score of the image set; sorting the alignment scores of all image sets in a descending order of the alignment scores; generating a test data set of the image signal processing model by using image sets in a front preset proportion in the sorting, and generating a training data set of the image signal processing model by using image sets after the preset proportion in the sorting, any data set in the test data set and the training data set comprising multiple RAW images taken with different frame rates and aligned reference images of the RAW images, the training data set being used for training the image signal processing model, and the test data set being used for testing the trained image signal processing model.

8. The method of claim 1, wherein, The image set comprises one RGB image and multiple RAW images taken by the high-speed camera with different frame rates under the same shooting scene; The image signal processing model can learn a feature representation of the RAW image to the RGB image under a degradation degree indicated by different frame rates; for any RAW image in the image set, generating an intermediate reference image of the RAW image, and aligning the RGB image to the intermediate reference image to obtain an aligned reference image comprises: The trained image mapper is used to process each RAW image in the image set to obtain an intermediate reference image of each RAW image output by the image mapper, the image mapper being trained using a specific RAW image in the image set and an RGB image in the image set, so that the image mapper learns the color difference characteristics between the specific RAW image and the RGB image and learns to restore an image with the same color as the RGB image using the color difference characteristics, the specific RAW image being a RAW image taken at a specific frame rate in the image set; a spatial relationship between the intermediate reference image of the specific RAW image and the RGB image is estimated, and the RGB image is aligned to the intermediate reference image using the spatial relationship to obtain an aligned reference image of each RAW image in the image set to which the specific RAW image belongs.

9. A high speed imaging image signal processing device, characterized by Comprise: an acquisition unit configured to acquire a plurality of image sets corresponding to different shooting scenes, the image set comprising an RGB image and a RAW image taken by a high-speed camera under the same shooting scene; a first generation unit configured to generate an intermediate reference image of a RAW image in any image set, align the RGB image to the intermediate reference image, and obtain an aligned reference image, the intermediate reference image being different from the RAW image in that the color of the intermediate reference image is the same as that of the RGB image; a second generation unit configured to generate a binary mask of a RAW image using the intermediate reference image and the aligned reference image of the RAW image; a training unit configured to complete training and testing of an image signal processing model using a RAW image and the aligned reference image of the RAW image in the image set, the training comprising: inputting the RAW image into the image signal processing model to obtain an RGB image output by the image signal processing model, calculating a first loss value using the binary mask, the aligned reference image, and the RGB image output by the image signal processing model of the same RAW image, and adjusting model parameters of the image signal processing model using the first loss value, so that the image signal processing model learns the feature representation of the RAW image to the RGB image and learns the model parameters matching the feature representation, the binary mask being used to filter out areas with misalignment problems between images when calculating the first loss value.

10. An electronic device, comprising: Comprise at least one processor and a memory connected to the processor, wherein: the memory is configured to store a computer program; the processor is configured to execute the computer program to enable the electronic device to implement the high-speed imaging image signal processing method according to any one of claims 1 to 8.

Images