Training data construction method and device based on multi-exposure wide dynamic image

Abstract

The invention discloses a training data construction method and device based on a multi-exposure wide dynamic image, and the method comprises the steps: obtaining target wide dynamic video RAW data, and carrying out the data splitting operation of the target wide dynamic video RAW data, and obtaining a plurality of exposure linear data; for each piece of exposure linear data, performing characteristic superposition operation on the exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data; constructing a training data pair based on all exposure output data; wherein the training data pair comprises a wide dynamic mapped image corresponding to the target wide dynamic video RAW data. Visibly, by implementing the method and the device, the construction of the training data can be intelligently executed on the multi-exposure wide dynamic image to obtain the corresponding data pair, so that the enhancement of the image training data set is facilitated, and the deep learning network can be helped to improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process; and the image processing quality can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for constructing training data based on multi-exposure wide dynamic images. Background Art

[0002] With the continuous development of science and technology, people's requirements for image quality have also increased. In existing technologies, image clarity is the most essential requirement of imaging technology. However, there is a difference between the dynamic range of the natural environment and the dynamic range that can be captured by ordinary linear traditional devices, which often causes overexposure of bright areas or underexposure of dark areas in images of some wide dynamic scenes.

[0003] Nowadays, most of the wide dynamic image sensors commonly used in business use a multi-exposure method, usually through time-sharing multiple wide dynamic, dual conversion gain wide dynamic, multi-pixel wide dynamic and other methods to perform image processing operations on wide dynamic images. However, most of the above-mentioned centralized image processing methods have processing limitations, such as motion ghosting in the image, poor linearity of some exposure ratios in the image, and incomplete matching of spatial positions in the image. At the same time, considering the interference of noise under different exposure ratios, with the continuous increase in the demand for dynamic range on the product side this year, the traditional algorithm is increasingly constrained by the limitations of design complexity and parameter tuning complexity. And currently it is mainly used for single network task training, and it is not possible to conduct in-depth exploration of data set construction in a targeted manner, which will lead to low accuracy and quality of image processing. It can be seen that it is particularly important to provide a method for constructing training data for wide dynamic images to enhance subsequent image training data sets and thus improve the accuracy and quality of image processing. Summary of the invention

[0004] The present invention provides a method and device for constructing training data based on multi-exposure wide dynamic images, which can intelligently perform training data construction on multi-exposure wide dynamic images to obtain corresponding data pairs, which is beneficial to enhancing the image training data set, and can help deep learning networks improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, thereby helping to improve the quality of image processing.

[0005] In order to solve the above technical problems, the first aspect of the present invention discloses a method for constructing training data based on multi-exposure wide dynamic images, the method comprising: Get the target wide dynamic video RAW data; Performing a data splitting operation on the target wide dynamic video RAW data to obtain a plurality of exposure linear data; For each exposure linear data, a characteristic superposition operation is performed on the exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data; Based on all the exposure output data, a training data pair is constructed; wherein the training data pair includes a wide dynamic mapped image corresponding to the target wide dynamic video RAW data.

[0006] A second aspect of the present invention discloses a training data construction device based on multi-exposure wide dynamic range images, the device comprising: An acquisition module is used to acquire the target wide dynamic video RAW data; A splitting module, used for performing a data splitting operation on the target wide dynamic video RAW data to obtain a plurality of exposure linear data; A superposition module, configured to perform a characteristic superposition operation on each exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data; A construction module is used to construct a training data pair based on all the exposure output data; wherein the training data pair includes a wide dynamic mapped image corresponding to the target wide dynamic video RAW data.

[0007] The third aspect of the present invention discloses another training data construction device based on multi-exposure wide dynamic image, the device comprising: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the training data construction method based on multi-exposure wide dynamic image disclosed in the first aspect of the present invention.

[0008] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the training data construction method based on multi-exposure wide dynamic image disclosed in the first aspect of the present invention.

[0009] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In an embodiment of the present invention, target wide dynamic video RAW data is obtained, and a data splitting operation is performed on the target wide dynamic video RAW data to obtain a plurality of exposure linear data; for each exposure linear data, a characteristic superposition operation is performed on the exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data; based on all exposure output data, a training data pair is constructed; wherein the training data pair includes a wide dynamic mapped image corresponding to the target wide dynamic video RAW data. It can be seen that the implementation of the present invention can intelligently perform the construction of training data on multi-exposure wide dynamic images to obtain corresponding data pairs, which is conducive to enhancing the image training data set, and can help the deep learning network improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, thereby helping to improve the quality of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 It is a flowchart of a method for constructing training data based on multi-exposure wide dynamic images disclosed in an embodiment of the present invention; Figure 2 It is a flowchart of another method for constructing training data based on multi-exposure wide dynamic images disclosed in an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a training data construction device based on multi-exposure wide dynamic image disclosed in an embodiment of the present invention; Figure 4 It is a structural schematic diagram of another training data construction device based on multi-exposure wide dynamic image disclosed in an embodiment of the present invention; Figure 5 It is a structural schematic diagram of another training data construction device based on multi-exposure wide dynamic image disclosed in an embodiment of the present invention; Figure 6 It is a schematic diagram of an exposure data relationship between an exposure data range and an actual dynamic range disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0014] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0015] The present invention discloses a method and device for constructing training data based on multi-exposure wide dynamic images, which can intelligently construct training data for multi-exposure wide dynamic images to obtain corresponding data pairs, which is beneficial to enhancing image training data sets, and can help deep learning networks improve the accuracy and intelligence of image processing in actual multi-exposure wide dynamic image processing, thereby improving the quality of image processing. The following are detailed descriptions.

[0016] Embodiment 1 See also Figure 1 , Figure 1 is a flow chart of a method for constructing training data based on multi-exposure wide dynamic range images disclosed in an embodiment of the present invention. Figure 1 The described method for constructing training data based on multi-exposure wide dynamic images can be applied to a device for constructing training data based on multi-exposure wide dynamic images, wherein the device for constructing training data based on multi-exposure wide dynamic images can be integrated in a cloud server or a local server, which is not limited in the embodiments of the present invention. Figure 1 As shown, the training data construction method based on multi-exposure wide dynamic image may include the following operations: 101. Obtain target wide dynamic video RAW data.

[0017] In an embodiment of the present invention, optionally, the target wide dynamic video RAW data can be acquired by a high-bitwidth professional dynamic camera, or by acquiring a public data set. Among them, the wide dynamic range (WDR) camera is an image processing technology that can simultaneously display highlight areas and dark details in the same image, solving the problem of overexposure of highlights or loss of dark details in high-contrast scenes in traditional photography. This technology is achieved through multi-frame exposure, image synthesis and intelligent algorithms, so that the final output image can retain details in both bright and dark areas.

[0018] In the embodiment of the present invention, optionally, the acquired public dataset is mainly a wide dynamic scene dataset constructed by the academic community, such as HDM-HDR-2014, LiU HDRv, etc. The advantages are easy acquisition and low cost, and the disadvantage is that the dataset scenes are limited.

[0019] In the embodiment of the present invention, further optionally, the finally acquired target wide dynamic video RAW data needs to be a set of continuous video data in time series.

[0020] 102. Perform a data splitting operation on the target wide dynamic video RAW data to obtain a plurality of exposure linear data.

[0021] In the embodiment of the present invention, optionally, the plurality of exposure linear data at least include one or more of long exposure pixel values / medium exposure pixel values / short exposure pixel values ​​obtained after performing a data splitting operation on the target wide dynamic video RAW data.

[0022] 103. For each exposure linear data, perform a characteristic superposition operation on the exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data.

[0023] In the embodiment of the present invention, optionally, the predetermined characteristic overlay parameters may include one or more of a camera response curve overlay parameter, a noise overlay parameter, a motion blur overlay parameter, an inter-frame motion overlay parameter, and a spatial misalignment overlay parameter.

[0024] In the embodiment of the present invention, optionally, the exposure output data may include one or more of long exposure image data, medium exposure image data, short exposure image data, and ultra-short exposure image data.

[0025] 104. Construct training data pairs based on all exposure output data.

[0026] In the embodiment of the present invention, the training data pair includes a wide dynamic range mapped image corresponding to the target wide dynamic range video RAW data.

[0027] In the embodiment of the present invention, optionally, the training data pair may also include multi-exposure images.

[0028] It can be seen that the implementation Figure 1 The described training data construction method based on multi-exposure wide dynamic images can obtain target wide dynamic video RAW data, perform data splitting operations on the target wide dynamic video RAW data to obtain a number of exposure linear data, and perform characteristic superposition operations on the exposure linear data according to predetermined characteristic superposition parameters to obtain exposure output data, and construct a training data pair including a wide dynamic mapped image corresponding to the target wide dynamic video RAW data based on all exposure output data, so as to obtain rich image information by obtaining the target wide dynamic video RAW data, provide the richest details in the image, and split the obtained RAW data into multiple linear data sets according to different exposure times, so as to obtain image data at different exposure levels, each data set highlights a specific brightness area in the image, and according to preset characteristic superposition parameters, these different exposure levels are combined Linear data is superimposed, such as pixel-level weighted superposition, to ensure that the final image retains details in each brightness area, and the exposure output data obtained by the superposition operation is paired with the original RAW data to form a training data pair. The formed training data pair can be used for training machine learning models to learn how to better process wide dynamic images, and can also reduce image noise by superimposing images with different exposure levels, which is beneficial to improving the overall quality of the image. Through the machine learning model, automated wide dynamic image processing can be achieved, and the construction of training data can be intelligently performed on multi-exposure wide dynamic images to obtain corresponding data pairs, which is beneficial to enhancing the image training data set, and can help the deep learning network improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, thereby helping to improve the quality of image processing.

[0029] Embodiment 2 See also Figure 2 , Figure 2 is a flow chart of another method for constructing training data based on multi-exposure wide dynamic range images disclosed in an embodiment of the present invention. Figure 2 The described method for constructing training data based on multi-exposure wide dynamic images can be applied to a device for constructing training data based on multi-exposure wide dynamic images, wherein the device for constructing training data based on multi-exposure wide dynamic images can be integrated in a cloud server or a local server, which is not limited in the embodiments of the present invention. Figure 2 As shown, the training data construction method based on multi-exposure wide dynamic image may include the following operations: 201. Obtain target wide dynamic video RAW data.

[0030] 202. Perform a pixel processing operation on the target wide dynamic video RAW data based on a predetermined pixel processing formula to obtain target linear data corresponding to the target wide dynamic video RAW data.

[0031] In the embodiment of the present invention, the target linear data includes RGB data of the target wide dynamic video RAW data after linearization processing.

[0032] In an embodiment of the present invention, optionally, the pixel processing operation may include a camera response processing operation and an electro-optical conversion function processing operation; wherein the camera response processing operation is mainly determined by the physical characteristics of the data acquisition camera, and the electro-optical conversion function processing operation is mainly determined by the physical characteristics of the display device / the photoelectric conversion characteristics defined by the standard.

[0033] In the embodiment of the present invention, further optionally, the electro-optical conversion function may include: ; in, For the collected / acquired video pixels (r / g / b data), is the inverse of the electro-optical conversion function and is used to complete the conversion of optical signals to electrical signals. represents the inverse of the camera response curve, It is the RGB data after linearization.

[0034] 203. Perform a Bayer format conversion operation on the target linear data to obtain target format data.

[0035] In an embodiment of the present invention, optionally, a Bayer Pattern is a color filter array for an image sensor. In the Bayer format, each pixel of the sensor is sensitive to only one color of light, and is usually arranged in a 2x2 matrix, half of which are green pixels, a quarter are red pixels, and a quarter are blue pixels.

[0036] In the embodiment of the present invention, optionally, performing the Bayer format conversion operation on the target linear data to obtain the target format data may include: According to a predetermined format conversion algorithm, a Bayer format conversion operation is performed on the target linear data to obtain target format data; The predetermined format conversion algorithm may include: ; in, is the RGB data after the previous linearization processing, Represents the RGB format to Bayer format operation and is used to extract the pixel points of the corresponding format at the corresponding spatial position. Bayer data.

[0037] In the embodiment of the present invention, further optionally, the target format data may include any one of the formats of RG / GB, BG / GR, GR / BG, and GB / RG.

[0038] 204. Perform a data splitting operation on the target format data to obtain a plurality of exposure linear data.

[0039] In the embodiment of the present invention, optionally, the target format data may be data corresponding to any one of the formats of RG / GB, BG / GR, GR / BG, and GB / RG; further, the target format data may be converted into data in the linear domain.

[0040] 205. For each exposure linear data, perform a characteristic superposition operation on the exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data.

[0041] 206. Construct training data pairs based on all exposure output data.

[0042] In the embodiment of the present invention, for the detailed description of step 201, step 205-step 206, please refer to the other descriptions of step 101, step 103-step 104 in the first embodiment, and the embodiment of the present invention will not be repeated.

[0043] It can be seen that the implementation Figure 2The described training data construction method based on multi-exposure wide dynamic images can perform pixel processing operations on target wide dynamic video data based on a predetermined pixel processing formula to obtain corresponding target linear data, and perform Bayer format conversion operations on the target linear data to obtain target format data, and then perform data splitting operations on the target format data to obtain a number of exposure linear data, and can convert RAW data into RGB data after linearization processing, which helps to reduce the impact of nonlinear factors on image quality and lays a foundation for subsequent processing, and converts the target linear data into Bayer format data, and can adapt the linearized RGB data to the Bayer array of the image sensor, which helps the subsequent image processing algorithm to process the data more effectively, and performs data splitting operations on the converted Bayer format data to obtain a number of exposure linear data, and can convert the Bayer format data into different exposure The light level is split into multiple data sets, each of which highlights a specific brightness area in the image. Through linearization processing, the nonlinear distortion of the image is reduced, making the brightness and color of the image more accurate. The Bayer format conversion makes the subsequent image processing algorithm more efficient. The data splitting operation allows the system to process data at different exposure levels separately, which helps to expand the dynamic range of the image. By processing data sets with different exposures separately, the bright and dark details in the image can be better preserved, and the overall detail performance of the image can be improved. It can effectively improve the processing quality and efficiency of wide dynamic video RAW data, provide a solid foundation for generating high-quality wide dynamic images, and help enhance image training data sets. It can also help deep learning networks improve the accuracy and intelligence of image processing in actual multi-exposure wide dynamic image processing, thereby helping to improve the quality of image processing.

[0044] In an optional embodiment, a data splitting operation is performed on the target format data to obtain a plurality of exposure linear data, including: Based on the target format data, determining an exposure data range corresponding to the target wide dynamic video RAW data, and determining an exposure data relationship between the exposure data range and the actual dynamic range; Determine the exposure ratio corresponding to the target format data according to the exposure data relationship; According to the exposure ratio and a predetermined split pixel expression, a data splitting operation is performed on the target format data to obtain a plurality of exposure linear data.

[0045] In this optional embodiment, optionally, the exposure data range corresponding to the target wide dynamic video RAW data may include a long exposure acquisition range, a medium exposure acquisition range, and a short exposure acquisition range.

[0046] In this optional embodiment, optionally, the exposure ratio mainly refers to the exposure time ratio between each wide dynamic exposure image. More specifically, the exposure ratio here mainly refers to long exposure exposure time / medium exposure exposure time, medium exposure exposure time / short exposure exposure time.

[0047] In this optional embodiment, optionally, the exposure data relationship between the exposure data range and the actual dynamic range may include a data mapping relationship between the actually acquired multi-exposure acquisition data and the actual dynamic range.

[0048] In this optional embodiment, further optionally, the exposure data relationship between the exposure data range and the actual dynamic range can be as follows: Figure 6 As shown, Figure 6 is a schematic diagram of an exposure data relationship between an exposure data range and an actual dynamic range disclosed in an embodiment of the present invention, such as Figure 6 As shown, taking 3-frame wide dynamic image acquisition as an example), let W(L), W(M), and W(S) be the actual scene dynamics recorded by long (Long Time Exposure) / middle (Middle Time Exposure) / short (Short Time Exposure) exposure respectively, then the dynamic range W(H) actually recorded by 3-frame wide dynamic is the sum of the three, which can be expressed as W(H)=W(L)∪W(M)∪W(S), ∪ represents the mathematical union. Further, corresponding to the actual response pixels, taking 3 frames of 16x exposure ratio wide dynamic (single exposure 12-bit linear ADC conversion) as an example, the corresponding split pixel value can be expressed as:

[0049] in, is the ideal wide dynamic pixel value (in the present invention, it corresponds to the data after the previous mosaic processing ), , , Corresponding to the long exposure pixel value, medium exposure pixel value, and short exposure pixel value after splitting.

[0050] In this optional embodiment, optionally, the exposure ratio mainly refers to the exposure time ratio between each wide dynamic exposure image. Furthermore, the exposure ratio here mainly refers to long exposure exposure time / medium exposure exposure time, medium exposure exposure time / short exposure exposure time; single exposure 12-bit linear ADC conversion mainly refers to the conversion accuracy of analog signals to digital signals during image sensor data acquisition. The CMOS bit width in industry scenarios is 10 / 12 / 14 bits. The wider the bit width, the wider the dynamic range recorded, and the higher the corresponding cost / power consumption.

[0051] It can be seen that the implementation of this optional embodiment can determine the exposure data range corresponding to the target wide dynamic video RAW data based on the target format data and determine the exposure data relationship between the exposure data range and the actual dynamic range, determine the exposure ratio corresponding to the target format data according to the exposure data relationship, and perform data splitting operations on the target format data according to the exposure ratio and the split pixel expression to obtain a number of exposure linear data, and can determine the exposure data range of the original wide dynamic video RAW data based on the target format data, can identify the data of the brightest and darkest parts of the image, analyze and determine the corresponding relationship between the exposure data range and the actual observable dynamic range, which helps to understand how data at different exposure levels are mapped to brightness changes that can be perceived by the human eye, calculate the exposure ratio of the target format data according to the exposure data relationship, that is, how data at different exposure levels are proportionally mapped to the final image, and use the exposure ratio and the predetermined split pixel expression to split the target format data to obtain a number of linear data sets at different exposure levels, which can be precisely controlled Data at different exposure levels can expand the dynamic range of the image. The dynamic range of the image can be expanded by precisely controlling data at different exposure levels, so that image details in both bright and low-light areas can be retained. Data splitting at different exposure levels can show more details in the image, which is beneficial to increasing the data volume of the image data set. By pre-calculating the exposure ratio and splitting the pixel expression, the complexity of post-processing can be reduced, the efficiency of image processing can be improved, and image noise and artifacts can be reduced. In particular, when synthesizing data at different exposure levels, different areas can be transitioned more smoothly. By dynamically adjusting the exposure ratio, it can adapt to different lighting environments to ensure that high-quality images can be obtained under various conditions. Precise exposure control and data splitting can significantly improve the processing effect of wide dynamic video RAW data, which is beneficial to enhancing the image training data set, and can help the deep learning network improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, thereby helping to improve the quality of image processing.

[0052] In another optional embodiment, for each exposure linear data, a characteristic superposition operation is performed on the exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data, including: For each exposure linear data, performing a first characteristic superposition operation on the exposure linear data to obtain first linear data corresponding to the exposure linear data; For each exposure linear data, performing a second characteristic superposition operation on the first linear data corresponding to the exposure linear data to obtain the second linear data corresponding to the exposure linear data; For each exposure linear data, generating exposure output data according to the second linear data corresponding to the exposure linear data; Among them, the first characteristic superposition operation includes a camera response curve superposition operation and a noise superposition operation; the second characteristic superposition operation includes a motion blur superposition operation, an inter-frame motion superposition operation, and a spatial dislocation superposition operation; the exposure output data corresponding to each exposure linear data includes the long exposure pixel value data, medium exposure pixel value data, short exposure pixel value data, and ultra-short exposure pixel value data corresponding to the exposure linear data.

[0053] In this optional embodiment, optionally, the camera response curve overlay operation mainly considers the difference in image sensor response curves at each ISO, and in the actual image acquisition process of the multi-exposure wide dynamic image sensor, actually each exposure is at a different ISO; the noise overlay operation mainly takes into account the difference in image sensor response curves at each ISO during the actual image acquisition process of the multi-exposure wide dynamic image sensor, and in fact each exposure is at a different ISO, and according to the actual image sensor characteristics, the image noise level at each ISO varies greatly.

[0054] It can be seen that the implementation of this optional embodiment can perform a first characteristic superposition operation on each exposure linear data to obtain first linear data corresponding to the exposure linear data, and perform a second characteristic superposition operation on the first linear data corresponding to each exposure linear data to obtain corresponding second linear data, and generate exposure output data according to the second linear data, wherein the first characteristic superposition operation includes a camera response curve superposition operation and a noise superposition operation; the second characteristic superposition operation includes a motion blur superposition operation, an inter-frame motion superposition operation, and a spatial dislocation superposition operation, and the exposure output data includes long exposure pixel value data, medium exposure pixel value data, and short exposure pixel value data corresponding to the exposure linear data, and can adjust the pixel value to match the actual response characteristic of the camera through the first characteristic superposition operation and perform noise superposition on the exposure linear data to simulate and reduce image noise, and perform motion blur superposition on the first linear data corresponding to each exposure linear data through the second characteristic superposition operation to simulate the blur effect of a moving object in the exposure device, and perform inter-frame motion superposition to process motion changes between consecutive frames to improve image stability in dynamic scenes, and perform spatial dislocation Superposition is used to adjust the pixel position to simulate actual motion and scene changes. By simulating camera response and noise characteristics, the generated image can be closer to the actual shooting effect. Motion blur and inter-frame motion superposition operations help to better handle dynamic scenes and reduce motion artifacts. The spatial dislocation superposition operation can improve the stability and coherence of the image in fast motion or complex scenes. By processing long exposure, medium exposure and short exposure pixel value data separately, better exposure balance can be achieved under different lighting conditions. It can adapt to different shooting environments and lighting conditions and provide consistent image quality. Therefore, through detailed feature superposition operations, not only the quality and authenticity of the image are improved, but also the processing capability of dynamic scenes is enhanced, so that the final exposure output data can maintain high quality under various conditions. Then, through precise exposure control and data splitting, the processing effect of wide dynamic video RAW data can be significantly improved, which is conducive to enhancing the image training data set, and can help the deep learning network to improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, which is conducive to improving the quality of image processing.

[0055] In yet another optional embodiment, based on all exposure output data, a training data pair is constructed, including: Based on the exposure output data and a predetermined camera response curve, a first true value data construction operation is performed on the exposure output data to obtain first true value construction data, wherein the first true value construction data includes a long exposure pixel value after the exposure output data is superimposed on the camera response curve, a medium exposure pixel value after the exposure output data is superimposed on the camera response curve, a short exposure pixel value after the exposure output data is superimposed on the camera response curve, and an ultra-short exposure pixel value after the exposure output data is superimposed on the camera response curve; Based on the first true value construction data and the predetermined noise processing parameters, a second true value data construction operation is performed on the first true value construction data to obtain second true value construction data, wherein the second true value construction data includes a medium exposure pixel value after the first true value construction data is superimposed with noise, a short exposure pixel value after the first true value construction data is superimposed with noise, and an ultra-short exposure pixel value after the first true value construction data is superimposed with noise; Generate target training data based on the first truth value construction data and the second truth value construction data, in combination with a predetermined inter-frame motion superposition truth value data construction formula and a predetermined spatial dislocation superposition truth value data construction formula; According to all target training data, training data pairs are obtained.

[0056] In this optional embodiment, optionally, performing the first true value data construction operation on the exposure output data based on the exposure output data and the predetermined camera response curve to obtain the first true value constructed data may include: Based on the exposure output data and a true value data construction algorithm corresponding to a predetermined camera response curve, a first true value data construction operation is performed on the exposure output data to obtain first true value constructed data; The algorithm for constructing the true value data corresponding to the predetermined camera response curve may include: ; in, is the Bayer pixel value of long exposure / Bayer pixel value of medium exposure / Bayer pixel value of short exposure / Bayer pixel value of ultra-short exposure obtained in the previous dynamic image splitting, represents the camera response curve, It is the long exposure pixel value / medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after superimposing the camera response curve; Furthermore, considering that low ISO usually has better linearity, it is recommended to use the camera response curve under low ISO conditions; and the biggest difference between this and the previous data construction is The curve does not change with the camera ISO, which is used as the true value construction process.

[0057] In this optional embodiment, the above-mentioned predetermined noise processing parameters may optionally include: ; in, It is the medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after the camera response curve is superimposed in the previous sequence. Represents the noise curve for long exposure, is the medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after superimposing noise; further, It can be obtained through light box calibration / noise model modeling, and the true value data corresponding to the noise superposition can maintain a signal-to-noise ratio consistent with long exposure.

[0058] In this optional embodiment, the predetermined inter-frame motion superposition true value data construction formula may include: ; in, , , It represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image after the reconstruction of the time series; further, it is used as the true value data for network training, and there is no misalignment in the time series.

[0059] In this optional embodiment, the predetermined spatial misalignment superposition true value data construction formula may include: ; in, , , , It represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image after spatial sampling reconstruction; further, the constructed true value data of its network training does not have any spatial position misalignment.

[0060] In this optional embodiment, optionally, obtaining the training data pair according to all target training data may include: All target training data are determined as training data pairs.

[0061] It can be seen that the implementation of this optional embodiment can perform a first truth value data construction operation based on the exposure output lock data and the camera response curve to obtain the first truth value construction data, and perform a second truth value data construction operation based on the first truth value construction data and the noise processing parameters to obtain the second truth value construction data, and generate target training data based on the first truth value data construction and the second truth value construction data in combination with the inter-frame motion superposition truth value data construction formula and the spatial dislocation superposition truth value data construction formula to obtain the training data pair, and can combine the long exposure, medium exposure and short exposure pixel value data with the camera response curve through the exposure output data and the camera response curve to simulate the real response when the camera captures the image, and obtain the second truth value construction data based on the first truth value construction data and the noise processing parameters, and can add the complexity of the real world to the training data by simulating the noise in the actual shooting, and combine the first truth value construction data, the second truth value construction data, the inter-frame motion superposition truth value data construction formula and the spatial dislocation superposition truth value data construction formula to generate target training data. It is used to simulate motion blur and spatial changes in dynamic scenes, provide more comprehensive training information for the algorithm, and can construct data containing inter-frame motion and spatial dislocation, so that the algorithm can process dynamic scenes more accurately. By superimposing noise in the training data, the algorithm can learn how to more effectively identify and reduce image noise. By including training data pairs with various complex situations, the model's adaptability to different shooting conditions can be improved. The authenticity and diversity of training data pairs help to improve the accuracy of image analysis tasks (such as target detection and image segmentation). By constructing training data pairs containing multiple factors and complexities, the performance and robustness of image processing algorithms can be significantly improved, especially when processing real-world image data. Through precise exposure control and data splitting, the processing effect of wide dynamic video RAW data can be significantly improved, which is conducive to enhancing image training data sets, and can help deep learning networks improve the accuracy and intelligence of image processing in actual multi-exposure wide dynamic image processing, thereby helping to improve the quality of image processing.

[0062] In yet another optional embodiment, based on all exposure output data, a training data pair is constructed, including: Determine wide dynamic linear true value data based on the exposure output data and a predetermined inverse camera response curve; Construct training data pairs based on wide dynamic linear true value data; And, after determining the wide dynamic linear true value data based on the exposure output data and the predetermined inverse camera response curve, the method further includes: According to a predetermined tone mapping algorithm, a tone mapping operation is performed on the wide dynamic linear true value data to obtain tone mapping true value data; Among them, based on the wide dynamic linear true value data, a training data pair is constructed, including: Based on the tone-mapped ground-truth data, training data pairs are constructed.

[0063] In this optional embodiment, optionally, the wide dynamic linear data generated by the exposure output data and the predetermined inverse camera response curve can be used as the true value of the wide dynamic linear data of the wide dynamic video being collected after the inverse camera response curve. At the same time, if there is inter-frame data sampling / spatial data sampling due to the characteristics of the image sensor, the wide dynamic image sampling can be performed according to the two descriptions of inter-frame motion superposition / spatial dislocation superposition in the above-mentioned multiple output, which can be described by the following algorithm: ; ; in, Corresponding to inter-frame motion superposition, Corresponding to the spatial dislocation superposition, , , , Corresponding to the wide dynamic image before and after sampling, the subscript , and , They correspond to the frame number in inter-frame sampling and the horizontal and vertical coordinate positions in spatial sampling respectively.

[0064] In this optional embodiment, the predetermined tone mapping algorithm may optionally include: ; in, is the true value data obtained in data construction, is the preset tone mapping operator, Represents the final true value dataset after tone mapping; wherein the preset tone mapping operator can be a traditional tone mapping algorithm or a tone mapping deep learning network.

[0065] It can be seen that the implementation of this optional embodiment can determine wide dynamic linear true value data based on the exposure output data and the inverse camera response curve, construct a training data pair based on the wide dynamic linear true value data, and, after determining the wide dynamic linear true value data, perform a tone mapping operation on the wide dynamic linear true value data according to the tone mapping algorithm to obtain tone mapping true value data, construct a training data pair based on the tone mapping true value data, and can use the exposure output data and the inverse camera response curve to determine the wide dynamic linear true value data, convert the data captured by the camera into a linear space to better process images with a wide dynamic range, and use the determined wide dynamic linear true value data to construct a data pair for training an image processing model, and by using the data processed by tone mapping, further construct a training data pair to train the model for color and brightness performance under different display conditions, and through the processing of the inverse camera response curve, the details of the high dynamic range scene can be better restored, and through the tone mapping algorithm The use of the method helps to ensure the consistency and accuracy of colors on different display devices. The constructed training data pairs include the complexity of wide dynamic range and color mapping, which helps to train a more robust and accurate model. By incorporating tone mapping true value data into the training process, the tone mapping algorithm can be further optimized to adapt to different visual and display needs. Through the construction of training data pairs, the model can learn how to present the best image effects on different display devices, thereby providing rich training information for the image processing model by combining wide dynamic linear true value data and tone mapping true value data, thereby achieving significant performance improvements in wide dynamic range imaging and color management, and further significantly improving the processing effect of wide dynamic video RAW data, which is conducive to enhancing the image training data set, and can help the deep learning network improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, thereby helping to improve the quality of image processing.

[0066] In yet another optional embodiment, for each exposure linear data, performing a first characteristic superposition operation on the exposure linear data to obtain first linear data corresponding to the exposure linear data includes: For each exposure linear data, performing a first characteristic superposition operation on the exposure linear data according to a predetermined camera response curve superposition algorithm and a predetermined noise superposition algorithm to obtain first linear data corresponding to the exposure linear data; The predetermined camera response curve superposition algorithm includes: ; in, is the Bayer pixel value of long exposure / Bayer pixel value of medium exposure / Bayer pixel value of short exposure / Bayer pixel value of ultra-short exposure obtained in the previous dynamic image splitting, (.) Includes camera response curves at various illumination levels, It is the long exposure pixel value / medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after superimposing the camera response curve; The predetermined noise superposition algorithm includes: ; in, It is the long exposure pixel value / medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after the camera response curve is superimposed in the previous sequence. Including noise curves at various illumination levels, It is the long exposure pixel value / medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after superimposing noise; And, for each exposure linear data, performing a second characteristic superposition operation on the first linear data corresponding to the exposure linear data to obtain the second linear data corresponding to the exposure linear data, including: For each exposure linear data, a second characteristic superposition operation is performed on the first linear data corresponding to the exposure linear data according to a predetermined motion blur superposition algorithm, an inter-frame motion superposition algorithm, and a spatial misalignment superposition algorithm to obtain second linear data corresponding to the exposure linear data; Among them, the motion blur superposition algorithm includes: ; in, It is a long exposure image / medium exposure image after motion blur superposition. It is the Bayer pixel input image of the long exposure image / medium exposure image after the previous noise is superimposed. is a spatial blur kernel function that varies with exposure; Inter-frame motion superposition algorithm, including: ; in, Represents a time series of long exposure images / medium exposure images / short exposure images / ultra-short exposure images, where the subscripts represent the corresponding frame numbers on the time axis; , , Represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image after reconstructing the time series; Spatial dislocation superposition algorithm, including: ; in, , , , represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image before sampling, and its subscript Represents the spatial position coordinates of the corresponding pixel point, , , , Represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image after spatial sampling and reconstruction.

[0067] In this optional embodiment, optionally, the camera response curve superposition is used to take into account the differences in the image sensor response curves at various ISOs, and in the actual image acquisition process of the multi-exposure wide dynamic image sensor, each exposure is actually at a different ISO.

[0068] In this optional embodiment, optionally, the noise superposition algorithm is used for a multi-exposure wide dynamic image sensor. In the actual image acquisition process, each exposure is actually at a different ISO, and according to the actual image sensor characteristics, the image noise level at each ISO varies greatly.

[0069] In this optional embodiment, further optionally, the motion blur superposition operation is mainly aimed at the situation where the actual physical exposure time corresponding to each exposure in the multi-exposure wide dynamic image sensor is inconsistent, and the clarity of the moving object corresponding to each exposure is inconsistent; for example, the short exposure image can be used as a reference (no blur), and the moving object in the corresponding medium exposure image / long exposure image is blurred by different degrees of blur kernel. Further, the motion blur superposition operation can be implemented by a motion blur superposition algorithm; Among them, the motion blur superposition algorithm may include: ; in, It is a long exposure image / medium exposure image after motion blur superposition. It is the Bayer pixel input image of the long exposure image / medium exposure image after the previous noise is superimposed. is a spatial blur kernel function that changes with exposure, wherein the longer the exposure time, the stronger the effect of the corresponding post function. Furthermore, the function can be adjusted by adjusting the blur function parameters or blur radius. The process can be described as a Gaussian blur kernel function, wherein the Gaussian blur kernel function may include: ; in, Indicates the starting and ending coordinates of the vertical filter. , Respectively represent the weights and pixels at the corresponding filtering spatial positions, Represents the spatial position coordinates of the pixel; further, The calculation process can be obtained through the following function: ; in, represents the standard deviation of the Gaussian distribution, where Affects the final filter strength.

[0070] In this optional embodiment, further optionally, the inter-frame motion superposition operation is specifically used for if multiple exposures are from the same physical pixel, then in the process of capturing a moving object, there is a misalignment of the moving object between each exposure; further, the inter-frame motion superposition can simulate the misalignment of the moving frame, which is specifically achieved by sampling and combining sequence frames without motion misalignment. For example, taking 3-frame wide dynamic as an example, the implementation process can be described as follows: ; in, , , Represents a time series of long exposure images / medium exposure images / short exposure images / ultra-short exposure images, where the subscripts represent the corresponding frame numbers on the time axis; , , Represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image after the reconstruction time series. And, from the above formula, it can be seen that the number of frames in the newly constructed long exposure time series / medium exposure time series / short exposure time series / ultra-short exposure time series is only 1 / 3 of the original data set, and the exposure order is long exposure / medium exposure / short exposure / ultra-short exposure (long exposure is extracted first, then medium exposure, and finally short exposure).

[0071] In this optional embodiment, further optional, the spatial misalignment superposition operation is mainly aimed at the situation where the physical pixels corresponding to each exposure are misaligned in space; wherein, in the actual wide dynamic image acquisition process, the four are set to different exposure times and gains to simultaneously acquire images, and the pixels between each exposure have a certain degree of spatial position misalignment, and this misalignment will cause jagged problems on the texture edge of the actual image. The superposition of spatial misalignment is mainly expected to simulate this situation so that the network has the error correction capability of this type of data. Its main means is to combine through spatial position sampling of the original sequence, and its implementation process can be described as follows: ; in, , , , represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image before sampling, and its subscript Represents the spatial position coordinates of the corresponding pixel point, , , , represents the long / medium / short / ultra-short exposure images after spatial sampling and reconstruction. It can be seen from the above formula that the spatial resolution of the newly constructed long exposure image / medium exposure image / short exposure image / ultra-short exposure image is only 1 / 4 of that before sampling, which are long exposure image / medium exposure image / short exposure image / ultra-short exposure image respectively.

[0072] It can be seen that the implementation of this optional embodiment can combine the predetermined camera response curve superposition algorithm and the noise superposition algorithm to perform the first characteristic superposition operation on the exposure linear data to obtain the first linear data corresponding to the large A, and can generate an image closer to the real shooting effect by simulating the actual response and noise characteristics of the camera, and the superposition of the camera response curve helps to expand the dynamic range of the image and better retain the details of the highlight and shadow parts, and the superposition of the noise model helps to consider the noise influence in the actual shooting in the image processing algorithm, thereby improving the effect of the noise reduction processing, the training data pair contains the camera response and noise information, which helps to improve the accuracy and reliability of the image processing algorithm in practical applications, and when synthesizing a high dynamic range image with multiple exposures, the characteristic superposition operation can ensure the consistency and naturalness of the synthesis result, and by considering different I The camera response and noise characteristics under SO settings, this method can adapt to different shooting environments and lighting conditions. In image analysis and computer vision tasks, more realistic and accurate training data can improve the accuracy of task execution. The automated image processing process can use these feature stacking operations to reduce manual intervention and improve processing efficiency and consistency. It can provide richer and more realistic training data for image processing by simulating the response and noise characteristics of the camera through feature stacking operations, thereby improving the performance and quality of image processing in many aspects, and can significantly improve the processing effect of wide dynamic video RAW data, which is conducive to enhancing the image training data set, and can help the deep learning network improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, which is conducive to improving the quality of image processing.

[0073] Embodiment 3 See also Figure 3 , Figure 3 1 is a schematic diagram of a structure of a training data construction device based on multi-exposure wide dynamic image disclosed in an embodiment of the present invention. Figure 3 As shown, the training data construction device based on multi-exposure wide dynamic image may include: An acquisition module 301 is used to acquire target wide dynamic video RAW data; The splitting module 302 is used to perform a data splitting operation on the target wide dynamic video RAW data to obtain a plurality of exposure linear data; The superposition module 303 is used to perform a characteristic superposition operation on each exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data; The construction module 304 is used to construct a training data pair based on all exposure output data; wherein the training data pair includes a wide dynamic mapped image corresponding to the target wide dynamic video RAW data.

[0074] It can be seen that the implementation Figure 3 The described device can obtain target wide dynamic video RAW data, perform a data splitting operation on the target wide dynamic video RAW data to obtain a plurality of exposure linear data, and perform a characteristic superposition operation on the exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data, and construct a training data pair including a wide dynamic mapped image corresponding to the target wide dynamic video RAW data based on all exposure output data, so that rich image information can be obtained by obtaining the target wide dynamic video RAW data, and the richest details in the image can be provided, and the obtained RAW data can be split into multiple linear data sets according to different exposure times, so that image data at different exposure levels can be obtained, each data set highlights a specific brightness area in the image, and these linear data at different exposure levels are superimposed according to preset characteristic superposition parameters. , such as pixel-level weighted superposition to ensure that the final image can retain details in each brightness area, and pair the exposure output data obtained by the superposition operation with the original RAW data to form a training data pair. The training data pair formed can be used for training machine learning models to learn how to better process wide dynamic images, and can also reduce image noise by superimposing images with different exposure levels, which is beneficial to improving the overall quality of the image. Through the machine learning model, automated wide dynamic image processing can be achieved, and the construction of training data can be intelligently performed on multi-exposure wide dynamic images to obtain corresponding data pairs, which is beneficial to enhancing the image training data set, and can help the deep learning network improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, thereby helping to improve the quality of image processing.

[0075] In an optional embodiment, if Figure 4 As shown, the device also includes: The processing module 305 is used for, after the acquisition module 301 acquires the target wide dynamic video RAW data, the splitting module 302 performs a data splitting operation on the target wide dynamic video RAW data, and before obtaining a plurality of exposure linear data, based on a predetermined pixel processing formula, performs a pixel processing operation on the target wide dynamic video RAW data to obtain target linear data corresponding to the target wide dynamic video RAW data, wherein the target linear data includes RGB data after the standard wide dynamic video RAW data is linearized; The conversion module 306 is used to perform a Bayer format conversion operation on the target linear data to obtain target format data; The specific method in which the splitting module 302 performs a data splitting operation on the target wide dynamic video RAW data to obtain a plurality of exposure linear data includes: A data splitting operation is performed on the target format data to obtain a plurality of exposure linear data.

[0076] It can be seen that the implementation Figure 4 The described device can perform pixel processing operations on target wide dynamic video data based on a predetermined pixel processing formula to obtain corresponding target linear data, and perform Bayer format conversion operations on the target linear data to obtain target format data, and then perform data splitting operations on the target format data to obtain a number of exposure linear data, and can convert RAW data into RGB data after linearization processing, which helps to reduce the impact of nonlinear factors on image quality and lays a foundation for subsequent processing, and converts the target linear data into Bayer format data, and can adapt the linearized RGB data to the Bayer array of the image sensor, which helps the subsequent image processing algorithm to process the data more efficiently, and performs data splitting operations on the converted Bayer format data to obtain a number of exposure linear data, and can split the Bayer format data into multiple according to different exposure levels. The data sets each highlight a specific brightness area in the image. Through linearization processing, the nonlinear distortion of the image is reduced, making the brightness and color of the image more accurate. The Bayer format conversion makes the subsequent image processing algorithm more efficient. The data splitting operation allows the system to process data at different exposure levels separately, which helps to expand the dynamic range of the image. By processing data sets with different exposures separately, the bright and dark details in the image can be better preserved, and the overall detail performance of the image can be improved. It can effectively improve the processing quality and efficiency of wide dynamic video RAW data, provide a solid foundation for generating high-quality wide dynamic images, and help enhance image training data sets. It can also help deep learning networks improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, thereby helping to improve the quality of image processing.

[0077] In another optional embodiment, Figure 4 As shown, the splitting module 302 performs a data splitting operation on the target format data to obtain a number of exposure linear data. The specific process includes: Based on the target format data, determining an exposure data range corresponding to the target wide dynamic video RAW data, and determining an exposure data relationship between the exposure data range and the actual dynamic range; Determine the exposure ratio corresponding to the target format data according to the exposure data relationship; According to the exposure ratio and a predetermined split pixel expression, a data splitting operation is performed on the target format data to obtain a plurality of exposure linear data.

[0078] It can be seen that the implementation Figure 4 The described device can determine the exposure data range corresponding to the target wide dynamic video RAW data and the exposure data relationship between the exposure data range and the actual dynamic range based on the target format data, determine the exposure ratio corresponding to the target format data according to the exposure data relationship, and perform data splitting operations on the target format data according to the exposure ratio and the split pixel expression to obtain a plurality of exposure linear data, can determine the exposure data range of the original wide dynamic video RAW data based on the target format data, can identify the data of the brightest and darkest parts of the image, analyze and determine the corresponding relationship between the exposure data range and the actual observable dynamic range, which helps to understand how data at different exposure levels are mapped to brightness changes perceptible to the human eye, calculate the exposure ratio of the target format data according to the exposure data relationship, that is, how data at different exposure levels are proportionally mapped to the final image, use the exposure ratio and the predetermined split pixel expression to split the target format data to obtain a plurality of linear data sets at different exposure levels, and can accurately control different exposure levels. The data of light level can expand the dynamic range of the image. The dynamic range of the image can be expanded by accurately controlling the data of different exposure levels, so that the image details in both highlight and low illumination areas can be retained, and the data splitting at different exposure levels can show more details in the image, which is beneficial to increase the data volume of the image data set, and by pre-calculating the exposure ratio and splitting the pixel expression, the complexity of post-processing can be reduced, the efficiency of image processing can be improved, and image noise and artifacts can be reduced. Especially when synthesizing data at different exposure levels, different areas can be transitioned more smoothly. By dynamically adjusting the exposure ratio, it can adapt to different lighting environments to ensure that high-quality images can be obtained under various conditions. Furthermore, through precise exposure control and data splitting, the processing effect of wide dynamic video RAW data can be significantly improved, which is beneficial to enhance the image training data set, and can help the deep learning network to improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, thereby helping to improve the quality of image processing.

[0079] In yet another optional embodiment, Figure 4 As shown, the superposition module 303 performs a characteristic superposition operation on each exposure linear data according to a predetermined characteristic superposition parameter, and a specific method of obtaining exposure output data includes: For each exposure linear data, performing a first characteristic superposition operation on the exposure linear data to obtain first linear data corresponding to the exposure linear data; For each exposure linear data, performing a second characteristic superposition operation on the first linear data corresponding to the exposure linear data to obtain the second linear data corresponding to the exposure linear data; For each exposure linear data, generating exposure output data according to the second linear data corresponding to the exposure linear data; Among them, the first characteristic superposition operation includes a camera response curve superposition operation and a noise superposition operation; the second characteristic superposition operation includes a motion blur superposition operation, an inter-frame motion superposition operation, and a spatial dislocation superposition operation; the exposure output data corresponding to each exposure linear data includes the long exposure pixel value data, medium exposure pixel value data, short exposure pixel value data, and ultra-short exposure pixel value data corresponding to the exposure linear data.

[0080] It can be seen that the implementation Figure 4The described device is capable of performing a first characteristic superposition operation on each exposure linear data to obtain first linear data corresponding to the exposure linear data, and performing a second characteristic superposition operation on the first linear data corresponding to each exposure linear data to obtain corresponding second linear data, and generating exposure output data according to the second linear data, wherein the first characteristic superposition operation includes a camera response curve superposition operation and a noise superposition operation; the second characteristic superposition operation includes a motion blur superposition operation, an inter-frame motion superposition operation, and a spatial dislocation superposition operation, and the exposure output data includes long exposure pixel value data, medium exposure pixel value data, and short exposure pixel value data corresponding to the exposure linear data, and is capable of adjusting the pixel value to match the actual response characteristic of the camera through the first characteristic superposition operation and performing noise superposition on the exposure linear data to simulate and reduce image noise, and performing motion blur superposition on the first linear data corresponding to each exposure linear data through the second characteristic superposition operation to simulate the blur effect of a moving object in an exposure device, and performing inter-frame motion superposition to process motion changes between consecutive frames to improve image stability in dynamic scenes, and performing spatial dislocation superposition to Adjusting the pixel position to simulate actual motion and scene changes can make the generated image closer to the actual shooting effect by simulating the camera response and noise characteristics. Motion blur and inter-frame motion superposition operations can help better process dynamic scenes and reduce motion artifacts. The spatial dislocation superposition operation can improve the stability and coherence of the image in fast motion or complex scenes. By processing long-exposure, medium-exposure and short-exposure pixel value data separately, better exposure balance can be achieved under different lighting conditions. It can adapt to different shooting environments and lighting conditions and provide consistent image quality. Therefore, through detailed feature superposition operations, not only the quality and authenticity of the image are improved, but also the processing ability of dynamic scenes is enhanced, so that the final exposure output data can maintain high quality under various conditions. Then, through precise exposure control and data splitting, the processing effect of wide dynamic video RAW data can be significantly improved, which is conducive to enhancing the image training data set, and can help the deep learning network to improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, which is conducive to improving the quality of image processing.

[0081] In yet another optional embodiment, Figure 4 As shown, the construction module 304 constructs the training data pair based on all exposure output data in the following specific manners: Based on the exposure output data and a predetermined camera response curve, a first true value data construction operation is performed on the exposure output data to obtain first true value construction data, wherein the first true value construction data includes a long exposure pixel value after the exposure output data is superimposed on the camera response curve, a medium exposure pixel value after the exposure output data is superimposed on the camera response curve, a short exposure pixel value after the exposure output data is superimposed on the camera response curve, and an ultra-short exposure pixel value after the exposure output data is superimposed on the camera response curve; Based on the first true value construction data and the predetermined noise processing parameters, a second true value data construction operation is performed on the first true value construction data to obtain second true value construction data, wherein the second true value construction data includes a medium exposure pixel value after the first true value construction data is superimposed with noise, a short exposure pixel value after the first true value construction data is superimposed with noise, and an ultra-short exposure pixel value after the first true value construction data is superimposed with noise; Generate target training data based on the first truth value construction data and the second truth value construction data, in combination with a predetermined inter-frame motion superposition truth value data construction formula and a predetermined spatial dislocation superposition truth value data construction formula; According to all target training data, training data pairs are obtained.

[0082] It can be seen that the implementation Figure 4The described device can perform a first truth value data construction operation based on exposure output lock data and a camera response curve to obtain first truth value construction data, and perform a second truth value data construction operation based on the first truth value construction data and noise processing parameters to obtain second truth value construction data, generate target training data based on the first truth value data construction and the second truth value construction data in combination with an inter-frame motion superposition truth value data construction formula and a spatial dislocation superposition truth value data construction formula to obtain a training data pair, can combine long exposure, medium exposure and short exposure pixel value data with the camera response curve through the exposure output data and the camera response curve to simulate the actual response when the camera captures an image, and obtain the second truth value construction data based on the first truth value construction data and the noise processing parameters, can add real-world complexity to the training data by simulating the noise in actual shooting, and generate target training data in combination with the first truth value construction data, the second truth value construction data, the inter-frame motion superposition truth value data construction formula and the spatial dislocation superposition truth value data construction formula to help simulate Motion blur and spatial changes in dynamic scenes provide more comprehensive training information for the algorithm. By constructing data containing inter-frame motion and spatial dislocation, the algorithm can process dynamic scenes more accurately. By superimposing noise in the training data, the algorithm can learn how to more effectively identify and reduce image noise. By including training data pairs with various complex situations, the model's adaptability to different shooting conditions can be improved. The authenticity and diversity of training data pairs help to improve the accuracy of image analysis tasks (such as target detection and image segmentation). By constructing training data pairs containing multiple factors and complexities, the performance and robustness of image processing algorithms can be significantly improved, especially when processing real-world image data. Precise exposure control and data splitting can significantly improve the processing effect of wide dynamic video RAW data, which is beneficial to enhancing image training data sets, and can help deep learning networks improve the accuracy and intelligence of image processing in actual multi-exposure wide dynamic image processing, thereby helping to improve the quality of image processing.

[0083] In yet another optional embodiment, Figure 4 As shown, the construction module 304 constructs the training data pair based on all exposure output data in the following specific manners: Determine wide dynamic linear true value data based on the exposure output data and a predetermined inverse camera response curve; Construct training data pairs based on wide dynamic linear true value data; And, the conversion module 306 is further configured to perform a tone mapping operation on the wide dynamic linear true value data according to a predetermined tone mapping algorithm to obtain tone mapped true value data after the construction module 304 determines the wide dynamic linear true value data based on the exposure output data and the predetermined inverse camera response curve; The construction module 304 constructs the training data pair based on the wide dynamic linear true value data, and the specific steps include: Based on the tone-mapped ground-truth data, training data pairs are constructed.

[0084] It can be seen that the implementation Figure 4 The described device can determine wide dynamic linear true value data based on exposure output data and an inverse camera response curve, construct training data pairs based on the wide dynamic linear true value data, and, after determining the wide dynamic linear true value data, perform a tone mapping operation on the wide dynamic linear true value data according to a tone mapping algorithm to obtain tone mapping true value data, construct training data pairs based on the tone mapping true value data, can use the exposure output data and the inverse camera response curve to determine the wide dynamic linear true value data, convert the data captured by the camera into a linear space to better process images with a wide dynamic range, and use the determined wide dynamic linear true value data to construct a data pair for training an image processing model, and by using the data processed by tone mapping, further construct a training data pair to train the model for color and brightness performance under different display conditions, by processing the inverse camera response curve, the details of the high dynamic range scene can be better restored, and by using the tone mapping algorithm It helps to ensure the consistency and accuracy of colors on different display devices. The constructed training data pairs include the complexity of wide dynamic range and color mapping, which helps to train more robust and accurate models. By incorporating tone mapping true value data into the training process, the tone mapping algorithm can be further optimized to adapt to different visual and display needs. Through the construction of training data pairs, the model can learn how to present the best image effects on different display devices, thereby providing rich training information for the image processing model by combining wide dynamic linear true value data and tone mapping true value data, thereby achieving significant performance improvements in wide dynamic range imaging and color management, and thus significantly improving the processing effect of wide dynamic video RAW data, which is conducive to enhancing image training data sets, and can help deep learning networks improve the accuracy and intelligence of image processing in actual multi-exposure wide dynamic image processing, thereby helping to improve the quality of image processing.

[0085] In yet another optional embodiment, Figure 4 As shown, the superposition module 303 performs a first characteristic superposition operation on each exposure linear data to obtain the first linear data corresponding to the exposure linear data in a specific manner including: For each exposure linear data, performing a first characteristic superposition operation on the exposure linear data according to a predetermined camera response curve superposition algorithm and a predetermined noise superposition algorithm to obtain first linear data corresponding to the exposure linear data; The predetermined camera response curve superposition algorithm includes: ; in, is the Bayer pixel value of long exposure / Bayer pixel value of medium exposure / Bayer pixel value of short exposure / Bayer pixel value of ultra-short exposure obtained in the previous dynamic image splitting, (.) Includes camera response curves at various illumination levels, It is the long exposure pixel value / medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after superimposing the camera response curve; The predetermined noise superposition algorithm includes: ; in, It is the long exposure pixel value / medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after the camera response curve is superimposed in the previous sequence. Including noise curves at various illumination levels, It is the long exposure pixel value / medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after superimposing noise; And, for each exposure linear data, performing a second characteristic superposition operation on the first linear data corresponding to the exposure linear data to obtain the second linear data corresponding to the exposure linear data, including: For each exposure linear data, a second characteristic superposition operation is performed on the first linear data corresponding to the exposure linear data according to a predetermined motion blur superposition algorithm, an inter-frame motion superposition algorithm, and a spatial misalignment superposition algorithm to obtain second linear data corresponding to the exposure linear data; Among them, the motion blur superposition algorithm includes: ; in, It is a long exposure image / medium exposure image after motion blur superposition. It is the Bayer pixel input image of the long exposure image / medium exposure image after the previous noise is superimposed. is a spatial blur kernel function that varies with exposure; Inter-frame motion superposition algorithm, including: ; in, , , Represents a time series of long exposure images / medium exposure images / short exposure images / ultra-short exposure images, where the subscripts represent the corresponding frame numbers on the time axis; , , Represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image after reconstructing the time series; Spatial dislocation superposition algorithm, including: ; in, , , , represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image before sampling, and its subscript Represents the spatial position coordinates of the corresponding pixel point, , , , Represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image after spatial sampling and reconstruction.

[0086] It can be seen that the implementation Figure 4 The described device can combine the predetermined camera response curve superposition algorithm and the noise superposition algorithm to perform a first characteristic superposition operation on the exposure linear data to obtain the first linear data corresponding to the large Ao, and can generate an image closer to the real shooting effect by simulating the actual response and noise characteristics of the camera, and the superposition of the camera response curve helps to expand the dynamic range of the image and better retain the details of the highlight and shadow parts, and the superposition of the noise model helps to consider the noise influence in the actual shooting in the image processing algorithm, thereby improving the effect of the noise reduction processing. The training data pair contains the camera response and noise information, which helps to improve the accuracy and reliability of the image processing algorithm in practical applications. When synthesizing a high dynamic range image with multiple exposures, the characteristic superposition operation can ensure the consistency and naturalness of the synthesis result, and by considering different ISO settings The method can adapt to different shooting environments and lighting conditions. In image analysis and computer vision tasks, more realistic and accurate training data can improve the accuracy of task execution. The automated image processing process can use these feature stacking operations to reduce manual intervention and improve processing efficiency and consistency. It can provide richer and more realistic training data for image processing by simulating the response and noise characteristics of the camera through feature stacking operations, thereby improving the performance and quality of image processing in many aspects, and can significantly improve the processing effect of wide dynamic video RAW data, which is conducive to enhancing the image training data set, and can help the deep learning network improve the accuracy and intelligence of image processing in the actual multi-exposure wide dynamic image processing process, which is conducive to improving the quality of image processing.

[0087] Embodiment 4 See also Figure 5 , Figure 5FIG. 1 is a schematic diagram of another apparatus for constructing training data based on multi-exposure wide dynamic range images disclosed in an embodiment of the present invention. Figure 5 As shown, the training data construction device based on multi-exposure wide dynamic image may include: A memory 401 storing executable program codes; a processor 402 coupled to the memory 401; The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the method for constructing training data based on multi-exposure wide dynamic range images described in the first embodiment of the present invention or the second embodiment of the present invention.

[0088] Embodiment 5 An embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps in the training data construction method based on multi-exposure wide dynamic images described in Embodiment 1 or Embodiment 2 of the present invention.

[0089] Embodiment 6 An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the method for constructing training data based on multi-exposure wide dynamic images described in Embodiment 1 or Embodiment 2.

[0090] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0091] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0092] Finally, it should be noted that the method and device for constructing training data based on multi-exposure wide dynamic images disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical scheme of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical schemes described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical schemes from the spirit and scope of the technical schemes of the embodiments of the present invention.

Claims

1. A method for constructing training data based on multi-exposure wide dynamic images, characterized in that: The method comprises: Get the target wide dynamic video RAW data; Performing a data splitting operation on the target wide dynamic video RAW data to obtain a plurality of exposure linear data; For each exposure linear data, a characteristic superposition operation is performed on the exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data; Based on all the exposure output data, construct a training data pair; wherein the training data pair includes a wide dynamic mapped image corresponding to the target wide dynamic video RAW data; After acquiring the target wide dynamic video RAW data, and before performing a data splitting operation on the target wide dynamic video RAW data to obtain a plurality of exposure linear data, the method further includes: Based on a predetermined pixel processing formula, a pixel processing operation is performed on the target wide dynamic video RAW data to obtain target linear data corresponding to the target wide dynamic video RAW data, wherein the target linear data includes RGB data of the standard wide dynamic video RAW data after linearization processing; Performing a Bayer format conversion operation on the target linear data to obtain target format data; The data splitting operation is performed on the target wide dynamic video RAW data to obtain a plurality of exposure linear data, including: A data splitting operation is performed on the target format data to obtain a plurality of exposure linear data.

2. The method for constructing training data based on multi-exposure wide dynamic range images according to claim 1, characterized in that: The data splitting operation is performed on the target format data to obtain a plurality of exposure linear data, including: Based on the target format data, determining an exposure data range corresponding to the target wide dynamic video RAW data, and determining an exposure data relationship between the exposure data range and an actual dynamic range; Determining an exposure ratio corresponding to the target format data according to the exposure data relationship; According to the exposure ratio and a predetermined pixel splitting expression, a data splitting operation is performed on the target format data to obtain a plurality of exposure linear data.

3. The method for constructing training data based on multi-exposure wide dynamic range images according to claim 1, characterized in that: For each exposure linear data, performing a characteristic superposition operation on the exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data includes: For each exposure linear data, performing a first characteristic superposition operation on the exposure linear data to obtain first linear data corresponding to the exposure linear data; For each exposure linear data, performing a second characteristic superposition operation on the first linear data corresponding to the exposure linear data to obtain second linear data corresponding to the exposure linear data; For each exposure linear data, generating exposure output data according to the second linear data corresponding to the exposure linear data; Among them, the first characteristic superposition operation includes a camera response curve superposition operation and a noise superposition operation; the second characteristic superposition operation includes a motion blur superposition operation, an inter-frame motion superposition operation, and a spatial dislocation superposition operation; the exposure output data corresponding to each of the exposure linear data includes the long exposure pixel value data, medium exposure pixel value data, short exposure pixel value data, and ultra-short exposure pixel value data corresponding to the exposure linear data.

4. The method for constructing training data based on multi-exposure wide dynamic range images according to claim 1, characterized in that: The constructing of training data pairs based on all the exposure output data comprises: Based on the exposure output data and a predetermined camera response curve, a first true value data construction operation is performed on the exposure output data to obtain first true value construction data, wherein the first true value construction data includes a long exposure pixel value after the exposure output data is superimposed on the camera response curve, a medium exposure pixel value after the exposure output data is superimposed on the camera response curve, a short exposure pixel value after the exposure output data is superimposed on the camera response curve, and an ultra-short exposure pixel value after the exposure output data is superimposed on the camera response curve; Based on the first true value construction data and the predetermined noise processing parameters, performing a second true value data construction operation on the first true value construction data to obtain second true value construction data, wherein the second true value construction data includes a medium exposure pixel value after the first true value construction data is superimposed with noise, a short exposure pixel value after the first true value construction data is superimposed with noise, and an ultra-short exposure pixel value after the first true value construction data is superimposed with noise; Generate target training data based on the first truth value construction data and the second truth value construction data, in combination with a predetermined inter-frame motion superposition truth value data construction formula and a predetermined spatial dislocation superposition truth value data construction formula; According to all the target training data, a training data pair is obtained.

5. The method for constructing training data based on multi-exposure wide dynamic range images according to claim 1, characterized in that: The constructing of training data pairs based on all the exposure output data comprises: Determining wide dynamic linear true value data based on the exposure output data and a predetermined inverse camera response curve; Based on the wide dynamic linear true value data, construct a training data pair; And, after determining the wide dynamic linear true value data based on the exposure output data and the predetermined inverse camera response curve, the method further includes: According to a predetermined tone mapping algorithm, a tone mapping operation is performed on the wide dynamic linear true value data to obtain tone mapping true value data; The step of constructing a training data pair based on the wide dynamic linear true value data includes: Based on the tone mapping true value data, a training data pair is constructed.

6. The method for constructing training data based on multi-exposure wide dynamic range images according to claim 3, characterized in that: For each exposure linear data, performing a first characteristic superposition operation on the exposure linear data to obtain first linear data corresponding to the exposure linear data includes: For each exposure linear data, performing a first characteristic superposition operation on the exposure linear data according to a predetermined camera response curve superposition algorithm and a predetermined noise superposition algorithm to obtain first linear data corresponding to the exposure linear data; The predetermined camera response curve superposition algorithm includes: ； in, is the Bayer pixel value of long exposure / Bayer pixel value of medium exposure / Bayer pixel value of short exposure / Bayer pixel value of ultra-short exposure obtained in the previous dynamic image splitting, (.) Includes camera response curves at various illumination levels, It is the long exposure pixel value / medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after superimposing the camera response curve; Wherein, the predetermined noise superposition algorithm includes: ； in, It is the long exposure pixel value / medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after the camera response curve is superimposed in the previous sequence. Including noise curves at various illumination levels, It is the long exposure pixel value / medium exposure pixel value / short exposure pixel value / ultra-short exposure pixel value after superimposing noise; And, for each of the exposure linear data, performing a second characteristic superposition operation on the first linear data corresponding to the exposure linear data to obtain the second linear data corresponding to the exposure linear data, including: For each of the exposure linear data, a second characteristic superposition operation is performed on the first linear data corresponding to the exposure linear data according to a predetermined motion blur superposition algorithm, an inter-frame motion superposition algorithm, and a spatial misalignment superposition algorithm to obtain second linear data corresponding to the exposure linear data; Wherein, the motion blur superposition algorithm includes: ； in, It is a long exposure image / medium exposure image after motion blur superposition. It is the Bayer pixel input image of the long exposure image / medium exposure image after the previous noise is superimposed. is a spatial blur kernel function that varies with exposure; The inter-frame motion superposition algorithm comprises: ； in, , , Represents a time series of long exposure images / medium exposure images / short exposure images, and its subscript represents the corresponding frame number on the time axis; , , Represents the long exposure image / medium exposure image / short exposure image after reconstructing the time series; The spatial dislocation superposition algorithm comprises: ； in, , , , represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image before sampling, and its subscript Represents the spatial position coordinates of the corresponding pixel point, , , , Represents the long exposure image / medium exposure image / short exposure image / ultra-short exposure image after spatial sampling and reconstruction.

7. A training data construction device based on multi-exposure wide dynamic image, characterized in that: The device comprises: An acquisition module is used to acquire the target wide dynamic video RAW data; A splitting module, used for performing a data splitting operation on the target wide dynamic video RAW data to obtain a plurality of exposure linear data; A superposition module, configured to perform a characteristic superposition operation on each exposure linear data according to a predetermined characteristic superposition parameter to obtain exposure output data; A construction module is used to construct a training data pair based on all the exposure output data; wherein the training data pair includes a wide dynamic mapped image corresponding to the target wide dynamic video RAW data.

8. A training data construction device based on multi-exposure wide dynamic image, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the training data construction method based on multi-exposure wide dynamic image according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the training data construction method based on multi-exposure wide dynamic image according to any one of claims 1 to 6.

Images