High-speed imaging image signal processing method and related device

Abstract

The invention provides a high-speed imaging image signal processing method and a related device, and the method comprises the steps: obtaining a plurality of image sets which comprise an RGB image in the same shooting scene and an RAW image shot by a high-speed camera; for any RAW image, generating an intermediate reference image of the RAW image, and aligning the RGB image to the intermediate reference image to obtain an aligned reference image; generating a binary mask of the RAW image by using the middle reference image and the aligned reference image of the same RAW image; training and testing of an image signal processing model are completed through the RAW image and the aligned reference image, training comprises the steps that the RAW image is input into the image signal processing model, an RGB image output by the model is obtained, a first loss value is calculated through a binary mask of the same RAW image, the aligned reference image and the RGB image output by the model, and a second loss value is calculated through the binary mask of the same RAW image, the aligned reference image and the RGB image output by the model; and adjusting the image signal processing model by using the first loss value.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a high-speed imaging image signal processing method and related apparatus. Background Technology

[0002] High-speed cameras can capture RAW images (raw or unprocessed images) using high-speed imaging technology. Currently, high-speed cameras play a crucial role in scientific research across multiple disciplines, such as biomechanical motion analysis, bio-inspired robot structural dynamics optimization, and real-time observation of neural activity and neutrophil migration trajectories. However, RAW images captured by high-speed cameras are often accompanied by significant noise and severe color distortion, seriously hindering the accurate observation of rapid dynamic processes.

[0003] To improve image quality, data-driven image signal processing (ISP) models can process RAW images and output RGB images. The perceived quality of RGB images is better than that of RAW images. Perceived quality reflects the user's subjective evaluation of the visual experience of the image.

[0004] Currently, the datasets for data-driven ISP models are constructed using low-speed imaging technology. The degradation level of RAW images captured using low-speed imaging technology differs significantly from that of RAW images captured by high-speed cameras. ISP models trained on such datasets struggle to effectively generalize to the noise and color distortion unique to RAW images captured by high-speed cameras. Consequently, the accuracy of ISP models in processing RAW images captured by high-speed cameras decreases, thereby reducing the perceived quality of the RGB images output by the ISP models. Summary of the Invention

[0005] In view of the above problems, this application provides a high-speed imaging image signal processing method and related apparatus to improve the accuracy of the ISP model when processing RAW images captured by a high-speed camera, thereby improving the user-perceived quality of the RGB images output by the ISP model. The specific solution is as follows:

[0006] The first aspect of this application provides a high-speed imaging image signal processing method, including:

[0007] Acquire multiple image sets corresponding to different shooting scenarios, wherein the image set includes an RGB image and a RAW image captured by a high-speed camera in the same shooting scenario;

[0008] For any RAW image in the image set, generate an intermediate reference image for the RAW image, and align the RGB image to the intermediate reference image to obtain an aligned reference image. The difference between the intermediate reference image and the RAW image is that the colors of the intermediate reference image are the same as the colors of the RGB image.

[0009] A binary mask for a RAW image is generated using an intermediate reference image and an aligned reference image from the same RAW image.

[0010] The image signal processing model is trained and tested using RAW images from the image set and aligned reference images of those RAW images. The training includes: inputting the RAW images into the image signal processing model to obtain RGB images output by the image signal processing model; calculating a first loss value using a binary mask of the same RAW image, the aligned reference image, and the RGB images output by the image signal processing model; and adjusting the model parameters of the image signal processing model using the first loss value so that the image signal processing model learns the feature representation from RAW images to RGB images and learns model parameters that match the feature representation. The binary mask is used to filter out regions with misalignment problems between images when calculating the first loss value.

[0011] In one possible implementation, generating the binary mask of the RAW image using an intermediate reference image and an aligned reference image from the same RAW image includes:

[0012] The intermediate reference image and the aligned reference image are divided according to a preset division method, so that the intermediate reference image is divided into multiple first image blocks and the aligned reference image is divided into multiple second image blocks;

[0013] Extract the phase spectrum of each first image block and the phase spectrum of each second image block;

[0014] For the first image block and the second image block that are in the same position, the alignment score between the first image block and the second image block is calculated using the phase spectrum of the first image block and the phase spectrum of the second image block.

[0015] If the alignment score is less than or equal to a threshold, the element values ​​of the first image block and the second image block in the binary mask are determined to be a first value. If the alignment score is greater than the threshold, the element values ​​of the first image block and the second image block in the binary mask are determined to be a second value. If the alignment score is less than or equal to the threshold, the first image block and the second image block are regions with misalignment problems. The first value is used to filter out regions with misalignment problems when calculating the first loss value.

[0016] In one possible implementation, calculating the first loss value using a binary mask of the same RAW image, an aligned reference image, and the RGB image output by the image signal processing model includes:

[0017] Using formula Calculate the first loss value , The RGB image of the RAW image output by the image signal processing model. This is a reference image for aligning the RAW image. The binary mask is represented by ⊙, where ⊙ denotes element-wise multiplication. It is an L1 norm.

[0018] In one possible implementation, the training further includes: extracting a first amplitude spectrum of an aligned reference image of the RAW image, and extracting a second amplitude spectrum of the RGB image of the RAW image output by the image signal processing model;

[0019] The Walstein distance between the first amplitude spectrum and the second amplitude spectrum is calculated using the slice Walstein distance function, and the Walstein distance is used as the second loss value.

[0020] The step of adjusting the model parameters of the image signal processing model using the first loss value includes: calculating the loss value of the image signal processing model using the first loss value and the second loss value, and adjusting the model parameters of the image signal processing model using the loss value.

[0021] In one possible implementation, when the RAW image is input into the image signal processing model, the gain corresponding to the RAW image is input into the image signal processing model, the gain being related to the frame rate used by the RAW image, so as to determine the degree of degradation of the RAW image through the gain.

[0022] In one possible implementation, the method further includes:

[0023] A RAW image to be processed is captured using the high-speed camera;

[0024] The RAW image to be processed and the gain used by the high-speed camera when capturing the RAW image to be processed are input into the image signal processing model;

[0025] The degree of degradation of the RAW image to be processed is determined by the image signal processing model through the gain.

[0026] The image signal processing model is guided by the degree of degradation of the RAW image to be processed to process the RAW image to obtain the RGB image output by the image signal processing model.

[0027] In one possible implementation, the image set includes a single RGB image from the same shooting scene and multiple RAW images captured by a high-speed camera at different frame rates. The method further includes:

[0028] For any RAW image, calculate the alignment score between the intermediate reference image and the aligned reference image of the RAW image;

[0029] The sum of the alignment scores of all RAW images in the same image set is taken as the alignment score of the image set.

[0030] Sort all image sets by alignment score from largest to smallest;

[0031] A test dataset for the image signal processing model is generated using a set of images ordered at the top of a preset ratio, and a training dataset for the image signal processing model is generated using a set of images ordered at the bottom of the preset ratio. Either the test dataset or the training dataset includes multiple RAW images taken at different frame rates and reference images aligned with the RAW images. The training dataset is used to train the image signal processing model, and the test dataset is used to test the trained image signal processing model.

[0032] In one possible implementation, the image set includes a single RGB image of the same shooting scene and multiple RAW images captured by a high-speed camera at different frame rates;

[0033] The image signal processing model can learn the feature representation of RAW images to RGB images under different frame rate degradation levels;

[0034] The step of generating an intermediate reference image for any RAW image in the image set, and aligning the RGB image to the intermediate reference image to obtain the aligned reference image includes:

[0035] The trained image mapper is used to process each RAW image in the image set to obtain an intermediate reference image for each RAW image output by the image mapper. The image mapper is trained using a specific RAW image and an RGB image in the image set to learn the color difference features between the specific RAW image and the RGB image, and to learn how to use the color difference features to reconstruct an image with the same colors as the RGB image. The specific RAW image is a RAW image in the image set captured at a specific frame rate.

[0036] Estimate the spatial relationship between the intermediate reference image of the specific RAW image and the RGB image, and use the spatial relationship to align the RGB image to the intermediate reference image to obtain an alignment reference image for each RAW image in the image set to which the specific RAW image belongs.

[0037] A second aspect of this application provides a high-speed imaging image signal processing apparatus, comprising:

[0038] The acquisition unit is used to acquire multiple image sets corresponding to different shooting scenarios. The image sets include an RGB image and a RAW image captured by a high-speed camera in the same shooting scenario.

[0039] The first generation unit is used to generate an intermediate reference image for any RAW image in any image set, and to align the RGB image to the intermediate reference image to obtain an aligned reference image. The difference between the intermediate reference image and the RAW image is that the colors of the intermediate reference image are the same as the colors of the RGB image.

[0040] The second generation unit is used to generate a binary mask for the RAW image using an intermediate reference image and an aligned reference image of the same RAW image.

[0041] The training unit is used to train and test the image signal processing model using RAW images from the image set and aligned reference images of those RAW images. The training includes: inputting the RAW images into the image signal processing model to obtain RGB images output by the image signal processing model; calculating a first loss value using a binary mask of the same RAW image, the aligned reference image, and the RGB images output by the image signal processing model; and adjusting the model parameters of the image signal processing model using the first loss value so that the image signal processing model learns the feature representation from RAW images to RGB images and learns model parameters that match the feature representation. The binary mask is used to filter out regions with misalignment problems between images when calculating the first loss value.

[0042] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0043] The memory is used to store computer programs;

[0044] The processor is used to execute the computer program so that the electronic device can implement the high-speed imaging image signal processing method of the first aspect or any implementation thereof.

[0045] The fourth aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the high-speed imaging image signal processing method of the first aspect or any implementation thereof.

[0046] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform a high-speed imaging image signal processing method as described in the first aspect or any implementation thereof.

[0047] By employing the above technical solution, this application provides a high-speed imaging image signal processing method and related apparatus. The dataset consists of RAW images captured by a high-speed camera and aligned reference images of those RAW images. The aligned reference images are obtained by aligning RGB images to an intermediate reference image of the RAW image; these aligned reference images can be the RGB images of the RAW images. By training and testing an ISP model using the dataset composed of the RAW images and their aligned reference images, the ISP model can learn the feature representations from the RAW images captured by the high-speed camera to the RGB images, and learn model parameters that match these feature representations. Thus, when processing RAW images captured by a high-speed camera, the ISP model can use the matched model parameters to process the RAW images, improving accuracy and consequently enhancing the user-perceived quality of the output RGB images.

[0048] Furthermore, a binary mask is introduced when calculating the first loss value of the ISP model. This binary mask can filter out regions where there is a misalignment problem between the aligned reference image and the RGB image output by the ISP model when calculating the first loss value, so as to reduce the impact of pixel misalignment between images on the performance of the ISP model. Thus, when using the trained ISP model to process RAW images, the accuracy can be improved and the user-perceived quality of the RGB image output by the ISP model can be improved. The misalignment problem can refer to the pixel misalignment problem. Attached Figure Description

[0049] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0050] Figure 1 A flowchart of a high-speed imaging image signal processing method provided in this application;

[0051] Figure 2 A flowchart illustrating the generation of a binary mask in the high-speed imaging image signal processing method provided in this application;

[0052] Figure 3 Another flowchart of a high-speed imaging image signal processing method provided in this application;

[0053] Figure 4 A schematic diagram of the high-speed imaging image signal processing method provided in this application;

[0054] Figure 5 Visualizations of ISP models trained on different datasets;

[0055] Figure 6 A visual comparison of the ISP model trained directly on the RHID dataset and the model trained using MisISP;

[0056] Figure 7 A visual comparison of the ISP model trained directly on the ZRR dataset and the model trained using MisISP;

[0057] Figure 8 A visual comparison chart for real-world applications of capturing moving objects at 10000 FPS.

[0058] Figure 9 A schematic diagram illustrating the effect of different signal gain inputs on the results;

[0059] Figure 10 A schematic diagram of a high-speed imaging signal processing device provided in this application;

[0060] Figure 11 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0061] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0062] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0063] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0064] Currently, the dataset for data-driven ISP models is constructed based on low-speed imaging technology. Specifically, the dataset includes multiple image pairs, each corresponding to a shooting scene. Each image pair includes an RGB image taken in the same scene and a RAW image taken using low-speed imaging technology. The RGB image can be taken with a DSLR camera, while the RAW image has lower image quality than the RGB image. After constructing the dataset, all image pairs are used to train and test the ISP model. This allows the trained ISP model to learn the feature representations of RAW images taken using low-speed imaging technology to convert them into RGB images, and to learn model parameters that match these feature representations. Thus, the trained ISP model can process input RAW images and output their RGB counterparts.

[0065] RAW images can be captured using both low-speed imaging techniques and high-speed cameras. However, the degradation level of RAW images captured using low-speed imaging techniques differs significantly from that of RAW images captured by high-speed cameras. This makes it difficult for the aforementioned ISP model to effectively generalize to the noise and color distortion unique to RAW images captured by high-speed cameras. Consequently, the accuracy of the aforementioned ISP model decreases when processing RAW images captured by high-speed cameras, thereby reducing the perceived quality of the RGB images output by the ISP model.

[0066] To address the aforementioned technical problems, embodiments of this application provide a high-speed imaging image signal processing method and related apparatus. The method uses RAW images captured by a high-speed camera and aligned reference images of these RAW images as datasets. The aligned reference images can be RGB images. These datasets are used to train and test an ISP model, enabling the ISP model to learn the feature representations from the RAW images captured by the high-speed camera to RGB images, and to learn model parameters that match these feature representations. Thus, when processing RAW images captured by a high-speed camera using the ISP model, the ISP model can employ the matched model parameters to process the RAW images, improving accuracy and thereby enhancing the user-perceived quality of the output RGB images.

[0067] Furthermore, a binary mask is introduced when calculating the first loss value of the ISP model. This binary mask can filter out regions where there is a misalignment problem between the aligned reference image and the RGB image output by the ISP model when calculating the first loss value, so as to reduce the impact of pixel misalignment between images on the performance of the ISP model. Thus, when using the trained ISP model to process RAW images, the accuracy can be improved and the user-perceived quality of the RGB image output by the ISP model can be improved. The misalignment problem can refer to the pixel misalignment problem.

[0068] The high-speed imaging image signal processing method and related apparatus provided in the embodiments of this application will be described below with reference to the accompanying drawings. Please refer to... Figure 1 This illustrates an optional flow of a high-speed imaging image signal processing method provided in an embodiment of this application, which may include the following steps:

[0069] S101. Acquire multiple image sets corresponding to different shooting scenarios. The image set includes an RGB image of the same shooting scenario and a RAW image taken by a high-speed camera. The RGB image can be taken by an SLR camera.

[0070] In this embodiment, the RGB images and RAW images in an image set are for the same shooting scene, while different image sets may correspond to different shooting scenes. To reduce the difference between RAW images and RGB images, a high-speed camera and a DSLR camera can simultaneously shoot the same shooting scene, and the field of view of the high-speed camera and the DSLR camera can remain consistent during shooting.

[0071] In one possible implementation, a coaxial imaging system is built for a high-speed camera and a DSLR camera. This system includes a high-speed camera, a DSLR camera, a beam splitter, and a synchronization trigger. The high-speed camera could be a Revealer S1310C, used to capture RAW images using high-speed imaging technology. The DSLR camera could be a Canon EOS 90D, used to capture RGB images, which have superior image quality compared to RAW images. The beam splitter ensures that the fields of view of the high-speed camera and the DSLR camera are identical. For example, the high-speed camera and the DSLR camera are fixed on an adjustable lifting and translation platform to precisely align the optical axes of the high-speed camera and the DSLR camera with the intersection point of the beam splitter, achieving the same field of view for both cameras. This allows for capturing images of different qualities of the same subject within the same field of view during a shooting scene. The synchronization trigger triggers the high-speed camera and the DSLR camera to simultaneously capture images. Furthermore, to minimize the influence of ambient light, the entire coaxial imaging system can be surrounded by a light-blocking plate, allowing illumination only from a predetermined direction.

[0072] During image acquisition, DSLR cameras can operate in aperture-priority mode, such as with a fixed aperture of f / 11 and a minimum ISO of 100, to ensure that the camera captures clear and low-noise RGB images with a resolution of 1024×1024. High-speed cameras output unprocessed 12-bit RAW images with a resolution of 1024×1024, and their aperture size varies randomly to increase the diversity of the RAW images.

[0073] In some examples, high-speed cameras can capture images at the same frame rate in different shooting scenarios. Thus, each image set can include one RAW image and one RGB image. The RAW images in different image sets have the same frame rate. The corresponding ISP model can learn the feature representation of the RAW image to the RGB image under a frame rate-indicating degradation level and learn the model parameters that match the feature representation.

[0074] In some examples, high-speed cameras can capture images at different frame rates in different shooting scenarios. Thus, each image set can include one RAW image and one RGB image, but the frame rates corresponding to the RAW images in different image sets can be different. This allows the ISP model to learn the feature representations of RAW images to RGB images under different frame rate degradation levels and to learn the model parameters that match different feature representations. As a result, the trained ISP model can process RAW images at different frame rates.

[0075] In some examples, a high-speed camera can capture images at different frame rates in the same shooting scene. Each image set can include multiple RAW images and one RGB image. The different frame rates of the multiple RAW images allow the ISP model to learn the feature representations of RAW images to RGB images at different degradation levels indicated by different frame rates, as well as the model parameters matching different feature representations. Thus, the trained ISP model can process RAW images at different frame rates. For example, common frame rates for high-speed cameras are 100, 1000, 2000, 3500, 5000, and 10000 frames per second (fps). For each shooting scene, six RAW images are captured at these frame rates. The exposure time of the high-speed camera can be set to the maximum value of the frame rate at each frame rate, and the gain of the high-speed camera is adjusted synchronously with the frame rate to reduce errors at different frame rates.

[0076] One point needs to be clarified here: after acquiring images from high-speed cameras and SLR cameras, users can manually delete images with poor quality. For example, in moving scenes, motion may cause the captured images to be blurry, so users can manually delete these images. Of course, this embodiment can also use other methods to delete images, such as determining the image sharpness based on at least one of gradient algorithms, frequency domain analysis, and statistical algorithms, and using the image sharpness to decide whether to delete the image. This embodiment does not limit the method of image deletion.

[0077] S102. For any RAW image in the image set, generate an intermediate reference image for the RAW image, and align the RGB image to the intermediate reference image to obtain the aligned reference image.

[0078] Although RAW and RGB images in the same image set are from the same shooting scene and the field of view is consistent between the high-speed camera and the SLR camera during shooting, there are still feature differences between the RAW and RGB images. Therefore, for any RAW image in the image set, an intermediate reference image and an aligned reference image are generated. The intermediate reference image differs from the RAW image in that its colors are the same as the RGB image, thus reducing the color difference between the intermediate reference image and the RGB image. Aligning the RGB image to the intermediate reference image can be done by aligning the position of a pixel in the RGB image to the position of that pixel in the intermediate reference image, so that the same pixel is close in position in both the intermediate and aligned reference images. The aligned reference image is still an RGB image. Therefore, the pixel misalignment between the aligned reference image and the intermediate reference image is significantly less than that between the RGB image and the intermediate reference image, reducing the pixel misalignment between the aligned reference image and the intermediate reference image. The aligned reference image and the intermediate reference image can be considered as an incompletely aligned image pair.

[0079] In this embodiment, the same pixel is in the same position in the RAW image and the intermediate reference image of the RAW image. The aligned reference image is only different from the RGB image in that the pixel position is changed. Therefore, the aligned reference image is the same as the RGB image except that the pixel position is different. Using the aligned reference image of the RAW image to train the ISP model can reduce the impact of pixel misalignment between the RAW image and the aligned reference image on the performance of the ISP model.

[0080] The intermediate reference image can be generated by a trained image mapper. This trained image mapper learns the color difference features between the RAW and RGB images and uses these features to reconstruct an image with the same colors as the RGB image. This ensures that the image output by the image mapper only differs in color from the RAW image. Therefore, inputting a RAW image into the trained image mapper yields the RGB image output by the image mapper, which is designated as the intermediate reference image.

[0081] One feasible way to align an RGB image to an intermediate reference image to obtain an aligned reference image is to apply a pre-trained optical flow network to estimate the spatial relationship between the intermediate reference image and the RGB images in the image set, and then perform a warping operation to generate the aligned reference image, as shown in the following formula:

[0082] .

[0083] in, Reference image for alignment, For RGB images in the image collection, As an intermediate reference image, For pre-trained optical flow networks, This is a warping operation. The intermediate reference image and the RAW image have the same spatial relationship (the pixels are in the same position in the image). Aligning with the intermediate reference image is actually aligning with the RAW image, thereby reducing the pixel misalignment between the aligned reference image and the RAW image, and thus reducing the degree of pixel misalignment between the aligned reference image and the RAW image.

[0084] S103. Generate a binary mask for the RAW image using the intermediate reference image and the aligned reference image of the same RAW image. In this embodiment, the purpose of the binary mask is to filter out areas where there are misalignment problems between images. Areas with misalignment problems refer to areas where the positional relationship between pixels is problematic. The reason why the positional relationship between pixels in different images is due to the sensor differences between high-speed cameras and SLR cameras, as well as object movement and occlusion during shooting. These areas with misalignment problems will affect the performance of the ISP model. Therefore, in this embodiment, for each RAW image, a binary mask for the RAW image is generated based on the areas where there are misalignment problems between the intermediate reference image and the aligned reference image of that RAW image.

[0085] Since the intermediate reference image of a RAW image only changes the colors compared to the RAW image, the binary mask of the RAW image is also applicable to the RAW image and its aligned reference image. In summary, by aligning to the intermediate reference image, the alignment error caused by high noise and color difference between the RAW image and its aligned reference image is mitigated, achieving a higher-precision matching between the RAW image and the aligned reference image. This allows the ISP model to learn the feature representation of the RAW image to a higher-precision RGB image.

[0086] In one possible implementation, the process of generating the binary mask for the RAW image is as follows: Figure 2 As shown, the following steps may be included:

[0087] S201. Divide the intermediate reference image and the aligned reference image according to the preset division method, so as to divide the intermediate reference image into multiple first image blocks and the aligned reference image into multiple second image blocks.

[0088] In some examples, the preset partitioning method can divide the image into multiple image blocks of the same size. Thus, the intermediate reference image and the aligned reference image can be evenly divided into multiple image blocks. However, the dimensions of the intermediate reference image and the aligned reference image need to be adjusted to the same size before partitioning. In some examples, the preset partitioning method can specify the size of each image block, dividing the image according to its specified size. While this allows for personalized partitioning, the process is relatively complex.

[0089] S202. Extract the phase spectrum of each first image block and the phase spectrum of each second image block. For example, Fourier transform can be used to extract the phase spectrum of each image block.

[0090] S203. For the first image block and the second image block that are in the same position, calculate the alignment score between the first image block and the second image block using the phase spectrum of the first image block and the phase spectrum of the second image block. In this embodiment, the alignment score between the first image block and the second image block can be calculated using a similarity function or a distance function, such as the following possible implementation:

[0091] .

[0092] in, It is an aligned score. The cosine similarity function; This indicates that the phase spectrum of the extracted image patch is extracted using the Discrete Fourier Transform. This indicates that features are extracted from a pre-trained VGG network model; Indicates the size of the image patch, indicating that the image is divided equally, such as... Using discrete Fourier transform to extract the phase spectrum and VGG network model to extract features is an optional approach, and this embodiment does not limit it.

[0093] S204. When the alignment score is less than or equal to the threshold, the element value of the first image block and the second image block in the binary mask is determined to be a first value. When the alignment score is greater than the threshold, the element value of the first image block and the second image block in the binary mask is determined to be a second value. When the alignment score is less than or equal to the threshold, the first image block and the second image block are regions with misalignment problems. The first value is used to filter out regions with misalignment problems when calculating the first loss value.

[0094] For example, in For example, a threshold . It is a binary mask. The value of the i-th element in the array.

[0095] Although the pixel misalignment between the aligned reference image and the RAW image is reduced, the two images may still be affected by non-rigid motion and occlusion, resulting in areas of misalignment between the two images. Therefore, this embodiment further identifies areas of misalignment between the images through alignment scores, so as to filter out these areas in the loss calculation and reduce the impact on the performance of the ISP model.

[0096] S104. Using RAW images from the image set and their aligned reference images, train and test the ISP model. Training includes: inputting RAW images into the ISP model to obtain the RGB images output by the ISP model; calculating a first loss value using the binary mask of the same RAW image, the aligned reference image, and the RGB images output by the ISP model; and adjusting the model parameters of the ISP model using the first loss value so that the ISP model learns the feature representation from RAW images to RGB images and learns model parameters that match the feature representation.

[0097] In this embodiment, the aligned reference image is the RAW image used during the ISP model training process. The binary mask is used to filter out regions with misalignment issues between the images when calculating the first loss value. In this embodiment, the spatial relationships of the content in the RGB image output by the ISP model are the same as those in the RAW image. Therefore, a binary mask from a single RAW image can be applied to both the RGB image output by the ISP model and the aligned reference image. This binary mask is used to remove regions with misalignment issues between the two images, and only regions without misalignment issues are used to calculate the first loss value.

[0098] One possible implementation is to use a formula.

[0099] Calculate the first loss value , This is the RGB image of the RAW image output by the ISP model. A reference image for aligning the RAW image. For binary masks, ⊙ represents element-wise multiplication. It is an L1 norm, so that only the region without misalignment problems is considered in the calculation of the first loss value.

[0100] In summary, the high-speed imaging signal processing method provided in this embodiment can use RAW images captured by a high-speed camera and aligned reference images of those RAW images as datasets. The aligned reference images can be RGB images. Using these datasets, the ISP model is trained and tested, enabling it to learn the feature representations from the RAW images captured by the high-speed camera to RGB images, and to learn model parameters matching these feature representations. Thus, when processing RAW images captured by a high-speed camera, the ISP model can use the matching model parameters, improving accuracy and consequently enhancing the user-perceived quality of the output RGB images. The aligned reference images of the RAW images can be obtained by aligning RGB images captured by a DSLR camera to an intermediate reference image. This means that the image quality of the aligned reference images of the RAW images is close to or the same as that of the RGB images captured by the DSLR camera. Therefore, training using the aligned reference images of the RAW images allows the ISP model to learn the feature representations from the RAW images to high-quality RGB images, resulting in RGB images recovered by the ISP model with image quality close to or the same as those captured by the DSLR camera.

[0101] Furthermore, a binary mask is introduced when calculating the first loss value of the ISP model. This binary mask can filter out areas where there is misalignment between the aligned reference image and the RGB image output by the ISP model when calculating the first loss value, thereby reducing the impact of pixel misalignment between images on the performance of the ISP model. As a result, when using the trained ISP model to process RAW images, the accuracy can be improved and the user-perceived quality of the RGB image output by the ISP model can be improved.

[0102] In addition, the above Figure 1 The high-speed imaging image signal processing method shown can train an ISP model using a first loss value calculated from the phase spectra of two images. The method can also incorporate the amplitude spectra of the two images during ISP model training to consider the different sensitivities of the phase and amplitude spectra to misalignment and image degradation (such as noise and blur). Correspondingly, the training of the ISP model in the high-speed imaging image signal processing method provided in this embodiment can further include: extracting the first amplitude spectrum of an aligned reference image of the RAW image; extracting the second amplitude spectrum of the RGB image of the RAW image output by the ISP model; calculating the Walstein distance between the first and second amplitude spectra using the slice Walstein distance function, and using the Walstein distance as the second loss value; correspondingly, adjusting the model parameters of the ISP model using the first loss value includes: calculating the loss value of the ISP model using the first and second loss values, and adjusting the model parameters of the ISP model using the loss value.

[0103] For example, using formulas Calculate the second loss value , Represents the Walstein distance function for slices. This represents the amplitude spectrum of the extracted image. After obtaining the second loss value, it is used... Calculate the loss value of the ISP model. The second loss value is a balancing value used to balance the ratio of the two loss values. Its value is a constant and is not limited in this embodiment. Introducing a second loss value can effectively suppress the influence of noise and enhance image clarity, thereby further improving the user-perceived quality of the RGB image output by the ISP model.

[0104] In some examples, the high-speed imaging image signal processing method provided in this embodiment can input the gain corresponding to the RAW image into the ISP model when inputting the RAW image. The gain is related to the frame rate used for the RAW image, so as to determine the degree of degradation of the RAW image through the gain. This allows the ISP model to use more parameters to restore the RGB image and improve the user-perceived quality of the RGB image. In particular, for different frame rate scenarios, the ISP model can use the gain to help determine the degree of degradation, so that the ISP model does not need a lot of additional parameters to learn the degree of degradation, thus allowing the ISP model to focus on image restoration. Correspondingly, when using the trained ISP model to process RAW images, it is also necessary to input the gain corresponding to the RAW image. The process includes steps 1 to 4:

[0105] Step 1: Take a RAW image to be processed using a high-speed camera.

[0106] Step 2: Input the RAW image to be processed and the gain used by the high-speed camera when capturing the RAW image into the ISP model;

[0107] Step 3: The ISP model determines the degree of degradation of the RAW image to be processed through gain;

[0108] Step 4: The ISP model is guided by the degree of degradation of the RAW image to be processed to process the RAW image to obtain the RGB image output by the ISP model.

[0109] Please see Figure 3 This illustrates another optional flow of a high-speed imaging image signal processing method provided in an embodiment of this application, which may include the following steps:

[0110] S301. Acquire multiple image sets corresponding to different shooting scenarios. The image sets include an RGB image of the same shooting scenario and multiple RAW images captured by a high-speed camera at different frame rates. The RGB image can be captured by an SLR camera.

[0111] S302. For any RAW image in the image set, generate an intermediate reference image for the RAW image, and align the RGB image to the intermediate reference image to obtain the aligned reference image.

[0112] S303. Using the intermediate reference image and the aligned reference image of the same RAW image, generate the binary mask of the RAW image.

[0113] S304. For any RAW image, calculate the alignment score between the intermediate reference image and the aligned reference image of that RAW image. In some examples, the alignment score can refer to the above. Figure 2 The alignment score of any image patch in the intermediate reference image and the aligned reference image is calculated, and the sum of the alignment scores of all image patches is taken as the alignment score of the intermediate reference image and the aligned reference image. Of course, the alignment score of the intermediate reference image and the aligned reference image can also be calculated in other ways, which will not be described here.

[0114] S305. The sum of the alignment scores of all RAW images in the same image set is taken as the alignment score of the image set.

[0115] S306. Sort the alignment scores of all image sets in descending order of alignment score.

[0116] S307. Generate a test dataset for the ISP model using the first 15% of images in a pre-defined order, and generate a training dataset for the ISP model using the subsequent 15% of images in a pre-defined order. Either the test dataset or the training dataset includes multiple RAW images taken at different frame rates and reference images aligned with the RAW images. The training dataset is used to train the ISP model, and the test dataset is used to test the trained ISP model. For example, the test dataset can be generated using the first 15% of images in a pre-defined order, and the training dataset can be generated using the subsequent 15% of images in a pre-defined order.

[0117] In this embodiment, steps S301 to S303 are the same as steps S101 to S103 described above, and will not be elaborated here. Furthermore, if the alignment score is calculated using other methods in step S304, steps S304 to S307 can be processed in parallel with steps S302 and S303.

[0118] S308. Train the ISP model using RAW images and aligned reference images from the training dataset, and test the ISP model using RAW images and aligned reference images from the test dataset.

[0119] Training includes: inputting a RAW image into the ISP model to obtain an RGB image output by the ISP model; calculating a first loss value using a binary mask of the same RAW image, an aligned reference image, and the RGB image output by the ISP model; and adjusting the model parameters of the ISP model using the first loss value so that the ISP model learns the feature representation from the RAW image to the RGB image and learns model parameters that match the feature representation. For details of the training process, please refer to the above. Figure 1 As shown, the testing process is the same as the existing testing process, and will not be described in detail here.

[0120] In some examples, the image set includes a single RGB image from the same shooting scene and multiple RAW images taken by a high-speed camera at different frame rates. An intermediate reference image is generated from the RAW images, and the RGB images are aligned to this intermediate reference image to obtain the aligned reference image. One feasible approach is as follows:

[0121] The trained image mapper is used to process each RAW image in the image set to obtain an intermediate reference image for each RAW image output by the image mapper. The image mapper is trained using a specific RAW image and an RGB image in the image set to learn the color difference features between the specific RAW image and the RGB image, and to learn how to use the color difference features to reconstruct an image with the same colors as the RGB image. The specific RAW image is a RAW image in the image set captured at a specific frame rate. The spatial relationship between the intermediate reference image of the specific RAW image and the RGB image is estimated. Using the spatial relationship, an aligned reference image for each RAW image in the image set to which the specific RAW image belongs is generated. In some examples, the image mapper can be trained before the image set is acquired. For example, multiple datasets of the image mapper can be acquired, each dataset including RAW images and RGB images corresponding to the same shooting scene. The RAW images are taken by a high-speed camera at a specific frame rate, and the RGB images are taken by an SLR camera. The image mapper is trained using multiple datasets so that it learns the color difference features between the specific RAW images and RGB images, and learns to use the color difference features to reconstruct an image with the same colors as the RGB images. The specific RAW images are RAW images in the image set taken at a specific frame rate.

[0122] The specific frame rate can be the minimum frame rate used by a high-speed camera, as the RAW images captured at the minimum frame rate have the best image quality. The image mapper includes an encoder. and a decoder . study and The color difference features between them are used to output a one-dimensional vector. , represented as:

[0123] ;

[0124] RAW images captured by a high-speed camera are used during the training of the image mapper. It can be a specific RAW image. To reduce computational costs, it employs... and The downsampled image as Input, such as and It was downsampled by a factor of 2. Then, the decoder... Through a series of differentiable, a priori-guided transformation functions (including brightness adjustment, white balance, color correction, gamma transformation, and tone mapping, etc.), Mapped to an intermediate reference image with reduced chromatic aberration As shown below:

[0125] ;

[0126] in It is a fixed set of non-trainable initialization parameters. and With the same dimension, the residual learning formula This stabilized the training and accelerated convergence. Among these, Scalar digital gain for brightness correction. The white balance multiplier for the R, G, and B channels. For color correction matrix, The gamma correction index. Tone mapping function. It is an S-shaped curve, defined as:

[0127] ;

[0128] in Trainable parameters for controlling the shape of the S-curve; This represents the function that takes the maximum value, while It is a small positive constant used to prevent instability caused by values ​​close to zero, for example... 1e-8 can be used. Therefore, the parameter set of the image mapper is represented as follows: Therefore, the image mapping process of the image mapper is as follows:

[0129] ;

[0130] in These are the learnable parameters of the image mapper. Because... and The spatial relationships between them are misaligned, making direct pixel-by-pixel supervision impractical. Therefore, to train the image mapper, the image mapper can utilize global histogram matching loss. Total variational loss To calculate the loss value of the image mapper, such as: ;

[0131] in It is the loss value of the image mapper. These are empirical values, such as a balancing weight of 0.005. Global histogram matching loss. Encourage global color consistency, defined as:

[0132] ;

[0133] in This indicates the calculation of a differentiable histogram. Calculate the cumulative distribution function. The L1 norm is used to calculate the differences between elements. Because the decoder only adjusts the colors of the image, the global histogram loss ensures that the mapped image is perceptually consistent with... Alignment, while effectively reducing color and brightness differences.

[0134] Total variational loss used to suppress noise Defined as: ,in and These are the horizontal and vertical gradient operators, respectively. The image mapper is trained using the loss value calculated by the above formula, so that the image mapper learns the color difference features between a specific RAW image and an RGB image, and learns to reconstruct an image with the same colors as the RGB image using the color difference features. It should be noted here that the loss value used by the image mapper is only an example, and this embodiment is not limited to it.

[0135] After training the image mapper, the trained image mapper is used to process each RAW image in the image set to obtain an intermediate reference image for each RAW image output by the image mapper. The intermediate reference image of each RAW image is different from the RAW image except that the colors of the image are changed.

[0136] Because high-speed cameras have extremely short capture intervals, motion between RAW images within the same image set is negligible, allowing for the reuse of spatial relationships within the same image set. Based on this, this embodiment can estimate the spatial relationship between an intermediate reference image and an RGB image of a specific RAW image. This estimated spatial relationship is reused to align intermediate reference images of other RAW images, generating aligned reference images for those images. This generation process can utilize a formula... The generation process will not be detailed here. This alignment strategy is an anchor-based alignment strategy, which reduces the impact of noise.

[0137] The following is combined Figure 4 The high-speed imaging image signal processing method provided in this application is described below. Figure 4 The optional structures for the image mapper and ISP model are shown in the figure. Figure 4 The image mapper in the image mapper can be trained using RAW images captured by a high-speed camera at the lowest frame rate and RGB images captured by a DSLR camera. The RAW images captured by the high-speed camera are defined as... RGB images captured by a DSLR camera are defined as The goal of the ISP model is to learn arrive The mapping can be represented as: ,in For ISP model The learnable parameters (i.e. the parameters of the model to be trained). Gain used for high-speed cameras.

[0138] The training process of the ISP model is as follows: Figure 4 As shown, Figure 4 The training process shown can be named the MisISP training architecture. Figure 4 RAW images captured by medium-to-high-speed cameras are processed by de-mosaicing and downsampling before being input into the encoder of the image mapper. RGB images captured by SLR cameras are downsampled before being input into the encoder of the image mapper, which outputs a one-dimensional vector. One-dimensional vector The de-mosaiced RAW image is input into the decoder of the image mapper to obtain the intermediate reference image output by the decoder. The RGB image captured by the SLR camera is aligned with a central reference image to obtain the aligned reference image. The intermediate reference image and the alignment reference image of the same RAW image are divided into multiple image blocks, and the alignment score of each image block is calculated. To generate a binary mask for the RAW image using alignment fraction. .

[0139] RAW images captured by a high-speed camera and the gain used during shooting. The input is fed into the ISP model, where the gain is used to determine the degree of degradation of the RAW image. The ISP model outputs the RGB image of the RAW image. Then, using the RGB image of the RAW image output by the ISP model, the binary mask of the RAW image, and the aligned reference image of the RAW image, the first loss value is calculated; using the RGB image of the RAW image and the aligned reference image of the RAW image, the second loss value is calculated; using the first loss value and the second loss value, the loss value of the ISP model is obtained; and the ISP model is trained using the loss value. For details, please refer to the above description, which will not be elaborated here.

[0140] To evaluate the high-speed imaging signal processing method provided in this application, the applicant conducted extensive experiments. All experiments were implemented using the PyTorch framework and trained on two NVIDIA GeForce RTX 3090 graphics processing units (GPUs). For training the ISP model, the Adam optimizer was used with an initial learning rate of... The learning rate was gradually reduced using a cosine annealing strategy. The ISP model was trained for 300,000 iterations, each using a batch of 16 RAW images, randomly cropped into 128×128 image patches as input. For the image mapper, since only the lowest-degraded RAW images were used, the number of training iterations was reduced to 100,000, and random cropping was omitted to better maintain overall color consistency. Furthermore, to evaluate image quality, three widely used metrics were employed: Peak signal-to-noise ratio (PSNR), Multi-Scale Structural Similarity Index Measure (MS-SSIM), and Learned Perceptual Image Patch Similarity (LPIPS).

[0141] The experiments were primarily conducted on the RHID dataset, which comprises multiple RAW images captured by a high-speed camera at different frame rates and RGB images captured by a DSLR camera. The high-speed camera and DSLR camera are located in a coaxial imaging system. The high-speed camera can capture six RAW images of the same scene at frame rates of 100, 1000, 2000, 3500, 5000, and 10000 frames per second (fps). The RHID dataset covers a wide range of content captured at different times of day and under various weather conditions, including buildings, roads, vegetation, water bodies, vehicles, and text. In the experiments, a total of 12,375 scenes were captured under different lighting and environmental conditions. The RHID dataset includes 240,480 pairs of images for training and 42,432 pairs of images for testing. Furthermore, existing ISP datasets S7, ZRR, and MAI were used to train the ISP model to demonstrate the necessity of using a high-speed imaging dataset. The experimental procedure and results are as follows:

[0142] Figure 5 The above is shown Figure 4 The processing performance of the ISP model obtained from the MisISP training architecture is shown below. Figure 5 The RAW images in the document are captured using a high-speed camera in different shooting scenarios. Figure 5 The reference in the image is an RGB image captured using a DSLR camera. Figure 5 Ours records the RGB images output by the ISP model after processing RAW images captured by a high-speed camera. Figure 5 As shown, the image quality of the RGB image output by the ISP model is close to that of the RGB image captured by a DSLR camera, effectively improving the perceived quality for users.

[0143] Necessity of the RHID dataset: Three existing ISP models—LiteISPNet, MetaISP, and ConvIR—were employed. These models were trained on the S7, ZRR, MAI, and RHID datasets, respectively, and their performance was evaluated across these datasets. For fair comparison, all ISP models were trained using the same learning strategy and tested on RAW images captured by high-speed cameras. Visual results are shown below. Figure 6 As shown, Figure 6The RAW images in the dataset are those captured by a high-speed camera; the Built-ISP images are those output by the high-speed camera's built-in ISP; the Reference images are RGB images captured by a DSLR camera; and the other images are the processing results of the RAW images using three ISP models trained on the four datasets mentioned above. Figure 6 It is evident that although the S7, ZRR, and MAI datasets contain exposure-related degradation to varying degrees, they are difficult to generalize to the inherent complex noise patterns and color distortions in RAW images captured using high-speed imaging techniques. RGB images output by models trained on these datasets exhibit significant noise and color distortion, such as... Figure 6 The images output by the ISP models trained using the S7, ZRR, and MAI datasets, as shown in the red box, exhibit significant noise and color distortion. In contrast, the images output by the ISP models trained using the RHID dataset are closer to RGB images, reducing noise and color distortion.

[0144] Table 1 shows the comparison results of ISP models trained on different datasets. As can be seen from the comparison results in Table 1, among the ISP models trained on the S7 dataset, ZRR dataset, and MAI dataset, the ISP model trained on the S7 dataset has the best performance. However, compared with the model trained on the RHID dataset, the model trained on the S7 dataset still shows an average decrease of 4.55 dB in PSNR.

[0145] Table 1. Comparison results of ISP models trained on different datasets

[0146]

[0147] Effectiveness of the MisISP Training Architecture: To demonstrate the effectiveness of the MisISP training architecture, it was applied to several representative state-of-the-art ISP and image restoration models, including LiteISPNet, SYENet, MetaISP, FourierISP, ConvIR, and TaylorFormer. Two training methods were used: one was training directly on the RHID dataset (labeled Baseline), and the other was training using the RHID dataset and the MisISP framework (labeled Ours). The performance comparison is shown in Table 2. Table 2 shows that all models trained on the RHID dataset achieved significant improvements compared to the ISP built into high-speed cameras, with a PSNR increase of at least 3.76 dB. When further combined with the MisISP framework, the PSNR of these models improved by at least 2.83 dB. This performance improvement is mainly due to the MisISP training architecture's ability to mitigate pixel misalignment in image pairs (i.e., filtering out misaligned regions during training), effectively reducing blur and improving reconstruction quality. Figure 7 It provides a visual comparison, which mainly focuses on... Figure 7 The part shown in the red box in the middle, from Figure 7 It is known that the ISP built into high-speed cameras struggles to suppress noise, resulting in underexposure and color distortion in the output images. Compared to the ISP built into high-speed cameras, the images output by models trained on the RHID dataset, despite severe texture loss, show improved brightness and color fidelity while removing noise. When enhanced using the MisISP training architecture, the textures of the images output by all models are clearer, comparable to RGB images captured by a DSLR camera. Figure 7 The reference in the model is close to that in the model, which effectively proves the effectiveness of the MisISP training structure.

[0148] Table 2. Comparison of ISP models trained directly on the RHID dataset and models trained using the MisISP framework.

[0149]

[0150] Generalization of the MisISP training architecture: Although the MisISP training architecture was proposed based on the RHID dataset, it can be well generalized to other existing datasets used for training ISP models. For example, the existing ZRR dataset, despite pre-alignment and manual selection, still exhibits significant pixel misalignment between paired images. To further validate the effectiveness of the MisISP training architecture, experiments were conducted on the ZRR dataset. The top 1000 most aligned image pairs from the ZRR dataset were selected as the test set based on alignment scores, and the remaining image pairs were used for training. Experiments were conducted on models such as LiteISPNet, SYENet, MetaISP, FourierISP, ConvIR, and TaylorFormer. Two training methods were used for these models: one was training directly on the ZRR dataset (labeled Baseline), and the other was training using the ZRR dataset and the MisISP framework (labeled Ours). The performance comparison of the models is shown in Table 3, and the visualization comparison is as follows: Figure 8 As shown in Table 2 and Figure 8 It can be seen that the ZRR dataset combined with the MisISP training architecture improves the performance of existing ISP models and demonstrates strong generalization ability.

[0151] Table 3. Comparison of ISP models trained directly on the ZRR dataset and models trained using the MisISP framework.

[0152]

[0153] Practical Applications in Capturing Moving Objects: Moving objects captured by DSLR cameras are prone to motion blur. Therefore, to obtain high-quality RGB images as reference images, scenes with dynamic objects should be avoided as much as possible when acquiring the RHID dataset. Although the RHID dataset contains limited high-speed motion content, the ISP model trained using the proposed MisISP training architecture and the RHID dataset remains effective for real-world high-speed applications. To this end, several scenes containing fast-moving objects were captured at 10,000 fps using a high-speed camera in experiments and processed using the high-speed camera's built-in ISR. Figure 9 As shown, the image was taken by an SLR camera ( Figure 9DSLRs (DSLRs) suffer from severe motion blur, making it difficult to observe motion details. Images processed by the built-in ISP of high-speed cameras exhibit significant color distortion, noise, and poor brightness. In contrast, the ISP model obtained by combining the RHID dataset with the MisISP training architecture can successfully reconstruct sharp images of moving objects. Even at a frame rate of 10,000 fps, it can output images of the same quality as those taken by DSLR cameras, and may even achieve better sharpness of moving objects than those captured by DSLR cameras.

[0154] Ablation experiments: Table 4 evaluates PMIA, and A separate contribution to the MisISP training architecture is that PMIA utilizes an image mapper and a point-based alignment strategy to mitigate alignment errors caused by high noise and chromatic aberration. The baseline model is LiteISPNet trained on the RHID dataset with L1 norm loss calculation; that is, the baseline model does not incorporate PMIA. and The performance of the ISP model was significantly improved after the introduction of PMIA, indicating that PMIA effectively alleviated pixel misalignment (also known as pixel deviation) and improved image fidelity. The addition of [a feature] further suppresses the adverse effects of misaligned pixels (i.e., pixels in regions with misalignment issues), thereby improving image quality. Meanwhile, This reduces excessive smoothing of texture details, thereby significantly improving the user-perceived quality (measured in LPIPS) of the output image. These results demonstrate that PMIA in the MisISP training architecture... and This brings substantial performance improvements to the ISP model.

[0155] Table 4 Ablation experiments of MisISP

[0156]

[0157] Table 5 evaluates the impact of different implementation details in the MisISP training architecture on performance.

[0158] Effectiveness of the image mapper: The image mapper reduces the difference between the RAW image and the reference image, thereby effectively improving alignment performance. Alignment results with and without the image mapper were compared during implementation. Quantitative results obtained by removing the image mapper are reported in Table 5 (see label "w / o"). (The line containing "") was observed to result in a performance degradation.

[0159] Effectiveness of the alignment strategy: In the MisISP training architecture, an anchor-based alignment strategy (AAS) is used for alignment. This involves selecting the spatial relationship between the intermediate reference image of the RAW image with the lowest frame rate and the RGB image captured by the DSLR camera as the alignment reference. As shown in Table 5 (see the row labeled "w / o AAS"), when the intermediate reference image of the RAW image captured at each frame rate is aligned with the reference image (here, the reference image is a high-quality RGB image), while keeping all other training conditions unchanged, a significant decrease in the comparison metrics (PSNR and MS-SSIM) can be observed. This decrease is mainly due to the increased noise level in the RAW images obtained under high gain and short exposure conditions. Furthermore, the image mapper is designed to adjust brightness and color rather than remove noise, thus the alignment accuracy decreases under these noisy conditions.

[0160] Comparison of Optical Flow Consistency: In the field of optical flow estimation, optical flow consistency (OFC) is a commonly used metric to measure the accuracy of flow prediction. OFC can also be used to generate binary masks in regions with misalignment problems. However, OFC often fails in weakly textured regions. Table 5 (see the table marked "...") → The line showing "" indicates that OFC was used to generate the binary mask instead of The performance of the ISP model further confirms this limitation. In contrast, the proposed... Evaluating bias based on the similarity between phase spectra avoids this error and provides more reliable supervision.

[0161] Gain effectiveness: The gain associated with each RAW image during training. Used as additional input to the ISP model. As shown in Table 5 (see label "w / o") (The line containing ""), delete This will lead to a decrease in all evaluation metrics. During the reasoning process, The value can be provided automatically by the high-speed camera or adjusted manually by the user. Figure 9 Shows different manual settings The impact of the value on the output. Larger values... This value produces a strong noise reduction effect, but may also suppress fine image details. In practical applications, high-speed cameras can provide a reasonable default value, which users can then fine-tune according to their personal preferences.

[0162] Table 5. Performance impact of different implementation details

[0163]

[0164] The above describes a high-speed imaging image signal processing method provided by the embodiments of this application. The following will describe the apparatus for performing the above-described high-speed imaging image signal processing method.

[0165] Please see Figure 10 , Figure 10 This is a schematic diagram of a high-speed imaging image signal processing device provided in an embodiment of this application. Detailed descriptions of each unit in the high-speed imaging image signal processing device can be found in the above method embodiments. Figure 10 As shown, the high-speed imaging signal processing device may include: an acquisition unit 10, a first generation unit 20, a second generation unit 30, and a training unit 40.

[0166] The acquisition unit 10 is used to acquire multiple image sets corresponding to different shooting scenarios. The image set includes an RGB image and a RAW image captured by a high-speed camera in the same shooting scenario.

[0167] The first generation unit 20 is used to generate an intermediate reference image for any RAW image in any image set, and to align the RGB image to the intermediate reference image to obtain an aligned reference image. The difference between the intermediate reference image and the RAW image is that the colors of the intermediate reference image are the same as those of the RGB image.

[0168] The second generation unit 30 is used to generate a binary mask for the RAW image using an intermediate reference image and an aligned reference image of the same RAW image.

[0169] Training unit 40 is used to train and test the image signal processing model using RAW images in the image set and their aligned reference images. The training includes: inputting RAW images into the image signal processing model to obtain the RGB images output by the image signal processing model; calculating a first loss value using a binary mask of the same RAW image, the aligned reference image, and the RGB images output by the image signal processing model; and adjusting the model parameters of the image signal processing model using the first loss value so that the image signal processing model learns the feature representation from RAW image to RGB image and learns model parameters that match the feature representation. The binary mask is used to filter out regions with misalignment problems between images when calculating the first loss value.

[0170] In one possible implementation, the second generation unit 30 generates a binary mask for the RAW image using an intermediate reference image and an aligned reference image of the same RAW image, including:

[0171] The intermediate reference image and the aligned reference image are divided according to a preset division method, so that the intermediate reference image is divided into multiple first image blocks and the aligned reference image is divided into multiple second image blocks; the phase spectrum of each first image block and the phase spectrum of each second image block are extracted; for first image blocks and second image blocks with the same position, the alignment score between the first image block and the second image block is calculated using the phase spectra of the first image block and the second image block; if the alignment score is less than or equal to a threshold, the element value of the first image block and the second image block in the binary mask is determined to be a first value; if the alignment score is greater than the threshold, the element value of the first image block and the second image block in the binary mask is determined to be a second value; if the alignment score is less than or equal to the threshold, the first image block and the second image block are regions with misalignment problems, and the first value is used to filter out regions with misalignment problems when calculating the first loss value.

[0172] In one possible implementation, training unit 40 uses a binary mask of the same RAW image, an aligned reference image, and an RGB image output by the image signal processing model to calculate a first loss value, including:

[0173] Using formula Calculate the first loss value , This is the RGB image of the RAW image output by the image signal processing model. A reference image for aligning the RAW image. For binary masks, ⊙ represents element-wise multiplication. It is an L1 norm.

[0174] In one possible implementation, the training unit 40 is further configured to extract a first amplitude spectrum of an aligned reference image of the RAW image, and extract a second amplitude spectrum of the RGB image of the RAW image output by the image signal processing model; calculate the Walstein distance between the first amplitude spectrum and the second amplitude spectrum using the slice Walstein distance function, and use the Walstein distance as a second loss value; adjusting the model parameters of the image signal processing model using the first loss value includes: calculating the loss value of the image signal processing model using the first loss value and the second loss value, and adjusting the model parameters of the image signal processing model using the loss value.

[0175] In one possible implementation, when the training unit 40 inputs the RAW image into the image signal processing model, it inputs the gain corresponding to the RAW image into the image signal processing model. The gain is related to the frame rate used by the RAW image, so as to determine the degree of degradation of the RAW image through the gain.

[0176] In one possible implementation, the acquisition unit 10 in the high-speed imaging image signal processing apparatus is further configured to capture a RAW image to be processed using a high-speed camera; the high-speed imaging image signal processing apparatus further includes a processing unit configured to input the RAW image to be processed and the gain used by the high-speed camera when capturing the RAW image to be processed into an image signal processing model; the image signal processing model determines the degree of degradation of the RAW image to be processed through the gain; the degree of degradation of the RAW image to be processed guides the image signal processing model to process the RAW image to be processed, so as to obtain an RGB image output by the image signal processing model.

[0177] In one possible implementation, the image set includes a single RGB image from the same shooting scene and multiple RAW images captured by a high-speed camera at different frame rates. The aforementioned high-speed imaging image signal processing device further includes a calculation unit, a sorting unit, and a dataset generation unit. Specifically, the calculation unit calculates the alignment score of any RAW image by taking the intermediate reference image and the aligned reference image, and sums the alignment scores of all RAW images in the same image set as the alignment score of the image set. The sorting unit sorts the alignment scores of all image sets from largest to smallest. The dataset generation unit generates a test dataset for the image signal processing model using the first image set in a predetermined proportion, and generates a training dataset for the image signal processing model using the image sets in a predetermined proportion. Either the test dataset or the training dataset includes multiple RAW images captured at different frame rates and aligned reference images of the RAW images. The training dataset is used to train the image signal processing model, and the test dataset is used to test the trained image signal processing model.

[0178] In one possible implementation, the image set includes a single RGB image from the same shooting scene and multiple RAW images captured by a high-speed camera at different frame rates; the image signal processing model can learn the feature representations of the RAW images to RGB images under different degradation levels indicated by different frame rates; the first generation unit 10 generates an intermediate reference image for any RAW image in the image set, and aligns the RGB image to the intermediate reference image to obtain the aligned reference image, including:

[0179] The trained image mapper is used to process each RAW image in the image set to obtain an intermediate reference image for each RAW image output by the image mapper. The image mapper is trained using a specific RAW image and an RGB image in the image set to learn the color difference features between the specific RAW image and the RGB image, and to learn how to use the color difference features to reconstruct an image with the same colors as the RGB image. The specific RAW image is a RAW image in the image set captured at a specific frame rate. The spatial relationship between the intermediate reference image and the RGB image of the specific RAW image is estimated. Using the spatial relationship, the RGB image is aligned to the intermediate reference image to obtain the aligned reference image for each RAW image in the image set to which the specific RAW image belongs.

[0180] This application also provides an electronic device in its embodiments. (See reference...) Figure 11 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), cameras, etc. Figure 11 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0181] like Figure 11 As shown, the electronic device may include a processing unit (e.g., a graphics processor) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0182] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 11An electronic device with various means is shown; however, it should be understood that it is not required to implement or possess all of the shown means. More or fewer means may be implemented or possessed alternatively. Processing means 401 can be considered as a processor in the electronic device, and ROM 402, RAM 403, and storage means 408 can be considered as memory in the electronic device, the memory being used to store computer programs; the processor is used to execute the computer programs to enable the electronic device to implement any of the high-speed imaging image signal processing methods provided in the embodiments of this application.

[0183] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the high-speed imaging image signal processing methods provided in this application.

[0184] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the high-speed imaging image signal processing methods provided in this application.

[0185] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0186] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0187] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0188] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A high-speed imaging image signal processing method, characterized in that, include: Acquire multiple image sets corresponding to different shooting scenarios, wherein the image set includes an RGB image and a RAW image captured by a high-speed camera in the same shooting scenario; For any RAW image in the image set, generate an intermediate reference image for the RAW image, and align the RGB image to the intermediate reference image to obtain an aligned reference image. The difference between the intermediate reference image and the RAW image is that the colors of the intermediate reference image are the same as the colors of the RGB image. A binary mask for a RAW image is generated using an intermediate reference image and an aligned reference image from the same RAW image. The image signal processing model is trained and tested using RAW images from the image set and aligned reference images of those RAW images. The training includes: inputting the RAW images into the image signal processing model to obtain RGB images output by the image signal processing model; calculating a first loss value using a binary mask of the same RAW image, the aligned reference image, and the RGB images output by the image signal processing model; and adjusting the model parameters of the image signal processing model using the first loss value so that the image signal processing model learns the feature representation from RAW images to RGB images and learns model parameters that match the feature representation. The binary mask is used to filter out regions with misalignment problems between images when calculating the first loss value.

2. The method according to claim 1, characterized in that, The process of generating a binary mask for a RAW image using an intermediate reference image and an aligned reference image from the same RAW image includes: The intermediate reference image and the aligned reference image are divided according to a preset division method, so that the intermediate reference image is divided into multiple first image blocks and the aligned reference image is divided into multiple second image blocks; Extract the phase spectrum of each first image block and the phase spectrum of each second image block; For the first image block and the second image block that are in the same position, the alignment score between the first image block and the second image block is calculated using the phase spectrum of the first image block and the phase spectrum of the second image block. If the alignment score is less than or equal to a threshold, the element values ​​of the first image block and the second image block in the binary mask are determined to be a first value. If the alignment score is greater than the threshold, the element values ​​of the first image block and the second image block in the binary mask are determined to be a second value. If the alignment score is less than or equal to the threshold, the first image block and the second image block are regions with misalignment problems. The first value is used to filter out regions with misalignment problems when calculating the first loss value.

3. The method according to claim 1 or 2, characterized in that, The calculation of the first loss value using the binary mask of the same RAW image, the aligned reference image, and the RGB image output by the image signal processing model includes: Using formula Calculate the first loss value , The RGB image of the RAW image output by the image signal processing model. This is a reference image for aligning the RAW image. The binary mask is represented by ⊙, where ⊙ denotes element-wise multiplication. It is an L1 norm.

4. The method according to claim 3, characterized in that, The training also includes: extracting the first amplitude spectrum of the aligned reference image of the RAW image, and extracting the second amplitude spectrum of the RGB image of the RAW image output by the image signal processing model; The Walstein distance between the first amplitude spectrum and the second amplitude spectrum is calculated using the slice Walstein distance function, and the Walstein distance is used as the second loss value. The step of adjusting the model parameters of the image signal processing model using the first loss value includes: calculating the loss value of the image signal processing model using the first loss value and the second loss value, and adjusting the model parameters of the image signal processing model using the loss value.

5. The method according to claim 3, characterized in that, When the RAW image is input into the image signal processing model, the gain corresponding to the RAW image is input into the image signal processing model. The gain is related to the frame rate used by the RAW image, so as to determine the degree of degradation of the RAW image through the gain.

6. The method according to claim 5, characterized in that, The method further includes: A RAW image to be processed is captured using the high-speed camera; The RAW image to be processed and the gain used by the high-speed camera when capturing the RAW image to be processed are input into the image signal processing model; The degree of degradation of the RAW image to be processed is determined by the image signal processing model through the gain. The image signal processing model is guided by the degree of degradation of the RAW image to be processed to process the RAW image to obtain the RGB image output by the image signal processing model.

7. The method according to claim 1, characterized in that, The image set includes a single RGB image from the same shooting scene and multiple RAW images captured by a high-speed camera at different frame rates. The method further includes: For any RAW image, calculate the alignment score between the intermediate reference image and the aligned reference image of the RAW image; The sum of the alignment scores of all RAW images in the same image set is taken as the alignment score of the image set. Sort all image sets by alignment score from largest to smallest; A test dataset for the image signal processing model is generated using a set of images ordered at the top of a preset ratio, and a training dataset for the image signal processing model is generated using a set of images ordered at the bottom of the preset ratio. Either the test dataset or the training dataset includes multiple RAW images taken at different frame rates and reference images aligned with the RAW images. The training dataset is used to train the image signal processing model, and the test dataset is used to test the trained image signal processing model.

8. The method according to claim 1, characterized in that, The image set includes a single RGB image from the same shooting scene and multiple RAW images captured by a high-speed camera at different frame rates; The image signal processing model can learn the feature representation of RAW images to RGB images under different frame rate degradation levels; The step of generating an intermediate reference image for any RAW image in the image set, and aligning the RGB image to the intermediate reference image to obtain the aligned reference image includes: The trained image mapper is used to process each RAW image in the image set to obtain an intermediate reference image for each RAW image output by the image mapper. The image mapper is trained using a specific RAW image and an RGB image in the image set to learn the color difference features between the specific RAW image and the RGB image, and to learn how to use the color difference features to reconstruct an image with the same colors as the RGB image. The specific RAW image is a RAW image in the image set captured at a specific frame rate. Estimate the spatial relationship between the intermediate reference image of the specific RAW image and the RGB image, and use the spatial relationship to align the RGB image to the intermediate reference image to obtain an alignment reference image for each RAW image in the image set to which the specific RAW image belongs.

9. A high-speed imaging image signal processing device, characterized in that, include: The acquisition unit is used to acquire multiple image sets corresponding to different shooting scenarios. The image sets include an RGB image and a RAW image captured by a high-speed camera in the same shooting scenario. The first generation unit is used to generate an intermediate reference image for any RAW image in any image set, and to align the RGB image to the intermediate reference image to obtain an aligned reference image. The difference between the intermediate reference image and the RAW image is that the colors of the intermediate reference image are the same as the colors of the RGB image. The second generation unit is used to generate a binary mask for the RAW image using an intermediate reference image and an aligned reference image of the same RAW image. The training unit is used to train and test the image signal processing model using RAW images from the image set and aligned reference images of those RAW images. The training includes: inputting the RAW images into the image signal processing model to obtain RGB images output by the image signal processing model; calculating a first loss value using a binary mask of the same RAW image, the aligned reference image, and the RGB images output by the image signal processing model; and adjusting the model parameters of the image signal processing model using the first loss value so that the image signal processing model learns the feature representation from RAW images to RGB images and learns model parameters that match the feature representation. The binary mask is used to filter out regions with misalignment problems between images when calculating the first loss value.

10. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the high-speed imaging image signal processing method as described in any one of claims 1 to 8.

Images