A data processing method and related apparatus

Abstract

The application provides a data processing method and related device, the data processing method comprises: generating a corresponding blurred image based on a plurality of clear images collected by an APS image sensor; determining an exposure time interval corresponding to the blurred image; obtaining target event stream data within the exposure time interval from event stream data collected by an EVS image sensor; and associating the blurred image, the target event stream data, and a ground truth image in the plurality of clear images to obtain training data for a deblurring network model. Through implementation of the application, the image data and event data stream captured by a real camera are used to generate triplet data for training the deblurring network model, which reduces the difficulty of data collection and improves the authenticity and effectiveness of the training data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method and related apparatus. Background Technology

[0002] In action shooting scenarios, images captured by cameras often suffer from motion blur, resulting in image quality that fails to meet users' actual needs. To improve the clarity of blurred images, researchers have developed an image deblurring algorithm model based on a large amount of training data.

[0003] Event-based image deblurring algorithms are a typical current approach. These algorithms process blurred images captured by a camera and the corresponding event data stream to obtain a sharp image. To improve the algorithm's performance, the training process is crucial. The training data for event-based image deblurring algorithms consists of triplet data, which includes the blurred image, the sharp image, and the corresponding event data stream.

[0004] Currently, the acquisition of the aforementioned triplet data typically involves using a high-frame-rate camera to capture high-frame-rate, clear images, using an event simulator to generate the corresponding event data stream, merging multiple clear images to generate a blurred image, and finally combining a clear image, a blurred image, and the event data stream to form the triplet data. However, since the event data stream in the training data is simulated, the authenticity and validity of the data cannot be fully guaranteed. Summary of the Invention

[0005] This application provides a data processing method and related apparatus, which can at least solve the problem that the event stream data used in the training data of the image deblurring algorithm model based on event data provided in the related art is simulated data, resulting in the inability to fully guarantee the authenticity and validity of the data.

[0006] The first aspect of this application provides a data processing method, including: generating a corresponding blurred image based on multiple clear images acquired by an APS image sensor; determining the exposure time interval corresponding to the blurred image; obtaining target event stream data within the exposure time interval from event stream data acquired by an EVS image sensor; and associating the blurred image, the target event stream data, and ground truth images from the multiple clear images to obtain training data for a deblurring network model.

[0007] A second aspect of this application provides a data processing apparatus, comprising: a generation module for generating a corresponding blurred image based on multiple clear images acquired by an APS image sensor; a determination module for determining an exposure time interval corresponding to the blurred image; an acquisition module for acquiring target event stream data within the exposure time interval from event stream data acquired by an EVS image sensor; and an association module for associating the blurred image, the target event stream data, and ground truth images from the multiple clear images to obtain training data for a deblurring network model.

[0008] A third aspect of this application provides a data processing system, including: an APS image sensor, an EVS image sensor, a memory, and a processor, wherein the processor is used to execute a computer program stored in the memory, and when the processor executes the computer program, it implements the steps of the data processing method provided in the first aspect of this application.

[0009] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the steps of the data processing method provided in the first aspect of this application.

[0010] As can be seen from the above, according to the data processing method and related apparatus provided in this application, based on multiple clear images acquired by an APS image sensor, a corresponding blurred image is generated; the exposure time interval corresponding to the blurred image is determined; target event stream data within the exposure time interval is obtained from the event stream data acquired by an EVS image sensor; and the blurred image, target event stream data, and ground truth images from the multiple clear images are correlated to obtain training data for the deblurring network model. Through the implementation of this application, image data and event data streams captured by a real camera are used to generate triplet data for training the deblurring network model, reducing data acquisition difficulty and improving the authenticity and effectiveness of the training data. Attached Figure Description

[0011] Figure 1 This is a basic flowchart illustrating the data processing method provided in the first embodiment of this application;

[0012] Figure 2 A schematic diagram of the exposure time of an APS image provided in the first embodiment of this application;

[0013] Figure 3 A schematic diagram illustrating a blurry image generation method provided in the first embodiment of this application;

[0014] Figure 4 A schematic diagram illustrating another method for generating blurred images provided in the first embodiment of this application;

[0015] Figure 5 A schematic diagram of the exposure time of three clear images provided in the first embodiment of this application;

[0016] Figure 6 A schematic diagram illustrating a frame interpolation method provided in the first embodiment of this application;

[0017] Figure 7 A detailed flowchart illustrating the data processing method provided in the second embodiment of this application;

[0018] Figure 8 This is a schematic diagram of the program modules of the data processing apparatus provided in the third embodiment of this application;

[0019] Figure 9 This is a schematic diagram of the structure of the data processing system provided in the fourth embodiment of this application. Detailed Implementation

[0020] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In the description of the embodiments of this application, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the present invention.

[0022] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0023] In the embodiments of this application, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0024] To address the problem that the event stream data used in the training data of image deblurring algorithms based on event data in related technologies is simulated data, leading to insufficient assurance of data authenticity and validity, the first embodiment of this application provides a data processing method, such as... Figure 1 This is a basic flowchart of the data processing method provided in this embodiment. The data processing method includes the following steps:

[0025] Step 101: Generate a corresponding blurred image based on multiple clear images acquired by the APS image sensor.

[0026] Specifically, an active-pixel sensor (APS) is a commonly used image sensor, in which each pixel sensor unit has a photodetector and at least one active transistor. In metal-oxide-semiconductor (MOS) active-pixel sensors, MOS field-effect transistors (MOSFETs) are used as amplifiers. There are various types of APS, including the earlier NMOS type APS and the more common complementary MOS (CMOS) type APS. In this embodiment, the frame rate of the APS image sensor can preferably be set to 28 FPS, and the exposure time can be set to 3–10 Ms. Of course, the exposure time can be adaptively adjusted according to the ambient light intensity.

[0027] It is worth noting that the APS image sensor in this embodiment can use a rolling shutter exposure method. The exposure start time and exposure end time of each row in the clear image acquired by the APS image are different, such as... Figure 2 The diagram shown is a schematic of the exposure time of an APS image provided in this embodiment. For an APS image, the exposure end time of the first row is represented as sof, and the exposure end time of the last row is represented as eof. The exposure time of each row is fixed and is called exposure. The time difference between the exposure end time of the i-th row and the (i+1)-th row is the data reading time of the i-th row, readouttime, which can also be called gap_time.

[0028] Step 102: Determine the exposure time range corresponding to the blurred image.

[0029] Specifically, in this embodiment, for a single-frame image formed by the roller shutter exposure method, the exposure start time of the first row of pixels of the image sensor and the exposure end time of the last row of pixels constitute the exposure time interval of the acquired single-frame image. In addition, the time difference between the two corresponds to the exposure time of the single-frame image.

[0030] Step 103: Obtain the target event stream data within the exposure time interval from the event stream data collected by the EVS image sensor.

[0031] Specifically, the Event-based Vision Sensor (EVS) in this embodiment is a novel sensor that mimics the human retina and responds to pixel pulses caused by brightness changes due to motion. Therefore, it can capture scene brightness changes at an extremely high frame rate, record events at specific times and locations in the image, and form an event stream instead of a frame stream. This can solve problems such as information redundancy, large data storage requirements, and large real-time processing requirements of traditional cameras.

[0032] In this embodiment, the frame rate of the EVS image sensor can be 800 FPS and the sensitivity can be 4 or 5. In practical applications, event data exists independently at a very high frame rate. Regardless of whether it is within the exposure time of the APS image sensor, as long as there is a change in light, event data will be output. Therefore, this embodiment needs to align the APS image and event data stream, that is, to obtain specific event stream data corresponding to the exposure time of the blurred image from all event data streams.

[0033] Step 104: Associate the blurred image, the target event stream data, and the ground truth image in multiple clear images to obtain the training data for the deblurring network model.

[0034] Specifically, in this embodiment, for multiple clear images acquired by the APS image sensor, a ground truth image is obtained from them. Then, a blurred image obtained based on the multiple clear images, a ground truth image from the multiple clear images, and an event data stream corresponding to the exposure time of the blurred image are associated to obtain training data in the form of triplets.

[0035] In one optional embodiment of this example, the step of associating the blurred image, target event stream data, and ground truth images from multiple clear images to obtain training data for the deblurring network model includes: obtaining the optical flow between the current frame blurred image and the previous frame blurred image as a mask; performing denoising processing on the target event stream data based on the mask to obtain denoised target event stream data; and associating the current frame blurred image, the denoised target event stream data, and ground truth images from multiple clear images to obtain training data for the deblurring network model.

[0036] Specifically, in practical applications, due to the presence of noise in EVS data, an optical flow algorithm is used to obtain the optical flow between the previous frame's blurred APS image and the current frame's blurred APS image as a mask to remove noise from the current frame's EVS data. It should be noted that the current frame's blurred APS image is the blurred image generated based on multiple clear images from the current data processing stage, while the previous frame's blurred APS image is the blurred APS image from the training data obtained in the previous data processing stage. Of course, if the current data processing stage is the first stage, then the current frame's blurred APS image can be used instead of the previous frame's blurred APS image.

[0037] It should be noted that, in this embodiment, the aforementioned optical flow algorithm is expressed as:

[0038] Mask i =(OpticalFlow(APS) i-1 APS i )!=0)+(OpticalFlow(APS i APS i-1 )! = 0);

[0039] Among them, Mask i Represents the mask, APS i and APS i-1 These represent the current frame APS image and the previous frame APS image, respectively.

[0040] Furthermore, the denoising algorithm in this embodiment is expressed as: EVS i =Mask i *EVS i If the current frame is the first frame in a video segment, then set Mask... i =1 is sufficient.

[0041] In one optional embodiment of this example, after the step of associating the blurred image, the target event stream data, and the ground truth image in multiple clear images to obtain the training data for the deblurring network model, the method further includes: training the deblurring network model using the training data to obtain the trained deblurring network model; and processing the input APS image and the event stream data in the same exposure time interval based on the trained deblurring network model to output the deblurred APS image.

[0042] Specifically, the deblurring network model in this embodiment is an image deblurring algorithm model based on event data. In this embodiment, the training data in the form of the above triplet is input into the deblurring network model based on a self-supervised framework for training. When the loss value no longer decreases and the model converges, the network model training is complete. In actual deblurring applications, for the single-frame APS image actually acquired by the APS image sensor (i.e., the blurred image to be deblurred), the event data stream acquired by the EVS image sensor during the exposure time of the APS image is simultaneously acquired. Then, the APS image and the event data stream are input into the aforementioned trained deblurring network model for processing. The final model output is the clear APS image corresponding to the original acquired APS image.

[0043] In one optional embodiment of this example, the step of processing the input APS image and event stream data within the same exposure time interval based on the trained deblurring network model and outputting a deblurred APS image includes: when processing multiple consecutive frames of APS images based on the trained deblurring network model, the current frame APS image, the current frame event stream data within the same exposure time interval, the previous frame APS image, the previous frame event stream data, and the output deblurred APS image of the previous frame are used as input to the deblurring network model for processing, and the current frame deblurred APS image is output.

[0044] Specifically, in this embodiment, during the deblurring process of multiple consecutive APS images captured using the trained deblurring network model, for each frame, the current frame's APS image, current frame's EVS data, the previous frame's APS image, the previous frame's EVS data, and the previous frame's model output are used as the model input for deblurring the current frame's APS image. The final output is the deblurring result of the current frame's APS image. It should be noted that if the current frame is the first frame of a video segment, then the previous frame's APS image and the previous frame's model output are replaced by the current frame's APS image, and the previous frame's EVS data are replaced by the current frame's EVS data.

[0045] In one optional embodiment of this example, the step of generating a corresponding blurred image from multiple clear images acquired by an APS image sensor includes: for the odd-numbered clear images acquired by the APS image sensor, performing equidistant frame interpolation on adjacent clear images in the time dimension; using the acquisition time of the middle image in the odd-numbered clear images as a reference, shifting the target time forward and backward respectively to obtain the start time and the end time; merging the target images between the start time and the end time in the multiple frames obtained after frame interpolation to generate the corresponding blurred image.

[0046] Specifically, in this embodiment, considering the limited frame rate of the APS image sensor and the large displacement of the moving object in two frames of data, directly merging multiple acquired APS images to generate a blurred image would result in an unrealistic merging effect. Figure 3 The diagram illustrates a blurred image generation method provided in this embodiment, showing how a blurred image is obtained by directly merging three clear images and taking the average value. To improve the realism of the blurred image, this embodiment proposes a blurred image generation method based on a frame interpolation algorithm, such as... Figure 4 The diagram shown is a schematic of another blurry image generation method provided in this embodiment. That is, the original odd-numbered clear images are interpolated to increase the number of frames in the image to be merged, and then the images with the increased number of frames are merged, thereby generating a more realistic blurry image.

[0047] This embodiment will continue to use three clear images originally acquired by the APS image sensor as an example for explanation. Figure 5 The diagram shown illustrates the exposure time of a clear image provided in this embodiment. To simulate blurry images captured with different exposure times, it is assumed that the shooting times of the first row of the three frames are 0s and 1s respectively. i-1 -exposure i-1 ,sof i -exposure i / 2、sof i+1 The shooting times in the last line are eof i-1 -exposure i-1 、eof i -exposure i / 2、eof i+1 .

[0048] like Figure 6The diagram illustrates a frame interpolation method provided in this embodiment. Next, this embodiment performs frame interpolation on every two images (e.g., the capture times of the first row are t0 and t1). The time point of the first interpolation result is the midpoint between the timestamps of the two input images: t2 = (t1 + t0) / 2. The time points of the second interpolation result are t3 = (t2 + t0) / 2, t4 = (t1 + t2) / 2, and so on. Using this frame interpolation method, three, five, seven, nine, or more images can be inserted between every two images, and these images are equidistant in the time dimension, which can greatly expand the number of images to be merged.

[0049] In this embodiment, the middle image among all images (e.g., the second frame in a three-frame image set) is used as a reference. Equal target durations *t* are shifted left and right along the time axis. The start and end times after these shifts represent different exposure durations. The interpolated frames within these time periods are then merged to generate the blurred image. It is worth noting that the target duration *t* satisfies the following condition:

[0050] t<(sof m -sof n +exposure n ) / 2

[0051] t>(sof m -sof n +exposure n ) / (N-1),

[0052] Where t represents the target duration, sof m The sof value represents the end time of the first row exposure of the last frame in an odd-numbered sharp image set. n Exposure indicates the end time of the first line exposure of the first frame in an odd-numbered sharp image set. n This represents the single-line exposure time of the first frame image, and N represents the total number of frames in the image obtained after interpolation. The value of N is an integer greater than 1.

[0053] For the example of three clear images originally acquired by the aforementioned APS image sensor, adaptively, t satisfies the following condition:

[0054] t<(sof i+1 -sof i-1 +exposure i-1 ) / 2

[0055] t>(sof i+1 -sof i-1 +exposure i-1 ) / (N-1),

[0056] It should be noted that if seven frames are inserted between every two clear frames in three clear images, then the total number of frames N after interpolation is: 7*2+3=17.

[0057] Furthermore, in an optional embodiment of this example, before the step of performing equidistant frame interpolation on adjacent clear frames in the time dimension for the odd-numbered clear images acquired by the APS image sensor, the method further includes: obtaining the actual frame rate of the APS image sensor; comparing the actual frame rate with a preset frame rate threshold; if the actual frame rate is lower than the frame rate threshold, then performing the aforementioned step of performing equidistant frame interpolation on adjacent clear frames in the time dimension for the odd-numbered clear images acquired by the APS image sensor; if the actual frame rate is higher than the frame rate threshold, then merging the odd-numbered clear images to generate a corresponding blurred image.

[0058] Specifically, in practical applications, the image acquisition frame rate of APS image sensors varies. If the image acquisition frame rate is high, it can ensure that there are enough original acquired images. In this case, the blurry image can be generated by directly merging the original clear images, thus ensuring the authenticity of the blurry image. Conversely, if the image acquisition frame rate is low and it is not possible to ensure that there are enough original acquired images, then the method of interpolation and merging described in this embodiment can be used to generate the blurry image.

[0059] In one optional embodiment of this example, the step of determining the exposure time interval corresponding to the blurred image includes: determining the first line exposure end time of the last frame of the multi-frame clear images as the first line exposure end time of the blurred image, and determining the last line exposure end time of the last frame of the multi-frame clear images as the last line exposure end time of the blurred image; calculating the single-line exposure time of the blurred image based on a preset exposure time calculation formula; and determining the exposure time interval corresponding to the blurred image based on the first line exposure end time, the last line exposure end time, and the single-line exposure time of the blurred image.

[0060] The above formula for calculating exposure time is expressed as follows:

[0061] exposure synthetic =sof m -sof n +exposure n ,

[0062] Among them, exposure synthetic The sof value represents the single-line exposure time of a blurred image. m This indicates the end time of the first line exposure of the last frame image, sof n Exposure indicates the end time of the first line exposure of the first frame in a multi-frame clear image set.n This indicates the single-line exposure time of the first frame image.

[0063] Continuing with the previous example of three clear images originally acquired by the APS image sensor, the first line exposure end time of the synthesized blurred image is represented as: sof synthetic =sof i+1 The end time of tailgating exposure is indicated as eof synthetic =eof i+1 Single-line exposure time is expressed as: exposure synthetic =sof i+1 -sof i-1 +exposure i-1 Finally, by summing the first line exposure end time, the last line exposure end time, and the single line exposure time of the blurred image, the exposure time range corresponding to the blurred image can be obtained.

[0064] In one optional embodiment of this example, before the step of associating the blurred image, the target event stream data, and the ground truth image in the multiple clear images, the method further includes: determining the (M+1)th clear target image in the 2M+1 clear images as the ground truth image; wherein, M is an integer greater than 1.

[0065] Specifically, because the exposure time set when the APS image sensor acquires data is relatively short, the captured images are relatively clear and can be used as ground truth. The ground truth image for synthesizing a blurred image from every 2M+1 frames is the M+1th frame. For example, when three clear frames are synthesized into a blurred image, the corresponding ground truth image is the second clear frame.

[0066] Based on the technical solution of the embodiments of this application described above, a corresponding blurred image is generated from multiple clear images acquired by an APS image sensor; the exposure time interval corresponding to the blurred image is determined; target event stream data within the exposure time interval is obtained from the event stream data acquired by an EVS image sensor; and the blurred image, target event stream data, and ground truth images from the multiple clear images are correlated to obtain training data for the deblurring network model. By implementing the solution of this application, image data and event data streams captured by a real camera are used to generate triplet data for training the deblurring network model, reducing the difficulty of data acquisition and improving the authenticity and effectiveness of the training data.

[0067] Figure 7 The method described in the second embodiment of this application is a refined data processing method, which includes:

[0068] Step 701: For the odd-numbered clear images acquired by the APS image sensor, perform equidistant frame interpolation on adjacent clear images in the time dimension.

[0069] In this embodiment, the original odd-numbered clear images are interpolated to increase the number of frames in the images to be merged. Then, the images with increased number of frames are merged, thereby generating a more realistic blurred image.

[0070] Step 702: Using the acquisition time of the intermediate image in the odd-numbered clear images as a reference, shift the target duration forward and backward respectively to obtain the start time and end time.

[0071] Step 703: Merge all target images between the start and end times in the multi-frame images obtained after frame interpolation to generate the corresponding blurred image.

[0072] Specifically, in this embodiment, a portion of the images obtained after frame interpolation are selected as images to be merged according to the offset time period, and then merged to obtain a blurred image.

[0073] Step 704: Determine the first line exposure end time of the last frame of all clear images as the first line exposure end time of the blurred image, and determine the last line exposure end time of the last frame of all clear images as the last line exposure end time of the blurred image.

[0074] Step 705: Calculate the single-line exposure time of the blurred image based on the preset exposure time calculation formula.

[0075] Specifically, the formula for calculating exposure time is: exposure time synthetic =sof m -sof n +exposure n Among them, exposure synthetic The sof value represents the single-line exposure time of a blurred image. m This indicates the end time of the first line exposure of the last frame image, sof n Exposure indicates the end time of the first line exposure of the first frame in a multi-frame clear image set. n This indicates the single-line exposure time of the first frame image.

[0076] Step 706: Based on the first line exposure end time, the last line exposure end time, and the single line exposure time of the blurred image, determine the exposure time range corresponding to the blurred image.

[0077] In this embodiment, the exposure time interval corresponding to the blurred image can be obtained by summing the first line exposure end time, the last line exposure end time, and the single line exposure time of the blurred image.

[0078] Step 707: Obtain target event stream data within the exposure time interval from the event stream data collected by the EVS image sensor.

[0079] Specifically, this embodiment requires aligning the APS image and event data stream, that is, obtaining specific event stream data corresponding to the exposure time of the blurred image from all event data streams.

[0080] Step 708: Associate the blurred image, the target event stream data, and the ground truth image in multiple clear images to obtain the training data for the deblurring network model.

[0081] In this embodiment, the ground truth image for synthesizing a blurred image from every 2M+1 frames is the M+1th frame. For example, when three clear images are synthesized into a blurred image, the corresponding ground truth image is the second clear image.

[0082] Step 709: Train the deblurring network model using the training data to obtain the trained deblurring network model.

[0083] Step 710: Based on the trained deblurring network model, process the actual input APS image and event stream data within the same exposure time interval, and output the deblurred APS image.

[0084] It should be understood that the sequence number of each step in this embodiment does not imply the order in which the steps are executed. The execution order of each step should be determined by its function and internal logic, and should not constitute a unique limitation on the implementation process of this application embodiment.

[0085] Based on the technical solutions of the embodiments of this application described above, on the one hand, frame interpolation is performed on the original odd-numbered clear images to increase the number of frames in the images to be merged, and then the images with increased number are merged, thereby generating more realistic blurred images and further improving the accuracy of the training data; on the other hand, image data and event data streams captured by real cameras are used to generate triplet data for training the deblurring network model, which reduces the difficulty of data collection and improves the authenticity and effectiveness of the training data; finally, based on the deblurring network model obtained using high-quality training data, the accuracy and robustness of APS image deblurring processing can be effectively guaranteed.

[0086] Figure 8 A data processing apparatus is provided in the third embodiment of this application. This data processing apparatus can be used to implement the data processing methods in the foregoing embodiments, and mainly includes:

[0087] The generation module 801 is used to generate a corresponding blurred image based on multiple clear images acquired by the APS image sensor.

[0088] The determination module 802 is used to determine the exposure time interval corresponding to the blurred image;

[0089] The acquisition module 803 is used to acquire target event stream data within the exposure time interval from the event stream data collected by the EVS image sensor;

[0090] The association module 804 is used to associate blurred images, target event stream data and ground truth images in multiple clear images to obtain training data for the deblurring network model.

[0091] In some embodiments of this example, the generation module is specifically used to: for the odd-numbered clear images acquired by the APS image sensor, perform equidistant frame interpolation processing on adjacent clear images in the time dimension; based on the acquisition time of the middle image in the odd-numbered clear images, shift the target time forward and backward respectively to obtain the start time and the end time; merge the target images between the start time and the end time in the multi-frame images obtained after frame interpolation to generate the corresponding blurred image.

[0092] Furthermore, in some implementations of this embodiment, the target duration satisfies the following condition:

[0093] t<(sof m -sof n +exposure n ) / 2

[0094] t>(sof m -sof n +exposure n ) / (N-1)

[0095] Where t represents the target duration, sof m The sof value represents the end time of the first row exposure of the last frame in an odd-numbered sharp image set. n Exposure indicates the end time of the first line exposure of the first frame in an odd-numbered sharp image set. n This represents the single-line exposure time of the first frame image, and N represents the total number of frames in the image obtained after interpolation. The value of N is an integer greater than 1.

[0096] Furthermore, in some other embodiments of this example, the data processing device further includes: a comparison module, used to obtain the actual frame rate of the APS image sensor; and compare the actual frame rate with a preset frame rate threshold. Correspondingly, the generation module is specifically used to: if the actual frame rate is lower than the frame rate threshold, then for the odd-numbered clear images acquired by the APS image sensor, perform equidistant frame interpolation processing on adjacent clear images in the time dimension; the generation module is also used to: if the actual frame rate is higher than the frame rate threshold, then merge the odd-numbered clear images to generate a corresponding blurred image.

[0097] In some embodiments of this example, the determining module is specifically used to: determine the first-line exposure end time of the last frame in a multi-frame clear image as the first-line exposure end time of the blurred image, and determine the last-line exposure end time of the last frame in a multi-frame clear image as the last-line exposure end time of the blurred image; calculate the single-line exposure time of the blurred image based on a preset exposure time calculation formula; the exposure time calculation formula is expressed as: exposure synthetic =sof m -sof n +exposure n Among them, exposure synthetic The sof value represents the single-line exposure time of a blurred image. m This indicates the end time of the first line exposure of the last frame image, sof n Exposure indicates the end time of the first line exposure of the first frame in a multi-frame clear image set. n This represents the single-line exposure time of the first frame image; based on the first-line exposure end time of the blurred image, the last-line exposure end time of the blurred image, and the single-line exposure time of the blurred image, the corresponding exposure time interval of the blurred image is determined.

[0098] In some embodiments of this example, the determining module is further configured to: determine the (M+1)th clear target image in the 2M+1 clear images as the ground truth image; wherein, M is an integer greater than 1.

[0099] In some implementations of this embodiment, the association module is specifically used to: obtain the optical flow between the current frame blurred image and the previous frame blurred image as a mask; perform denoising processing on the target event stream data based on the mask to obtain denoised target event stream data; and associate the current frame blurred image, the denoised target event stream data, and the ground truth images in multiple clear images to obtain training data for the deblurring network model.

[0100] In some embodiments of this example, the data processing device further includes a training module and a processing module, wherein the training module is used to train the deblurring network model using training data to obtain a trained deblurring network model; the processing module is used to process the input APS image and event stream data with the same exposure time interval based on the trained deblurring network model, and output the deblurred APS image.

[0101] Furthermore, in some embodiments of this example, the processing module is specifically used to: when processing multiple consecutive APS images based on the trained deblurring network model, take the current frame APS image, the current frame event stream data in the same exposure time interval, the previous frame APS image, the previous frame event stream data, and the output deblurred APS image of the previous frame as input to the deblurring network model for processing, and output the current frame deblurred APS image.

[0102] It should be noted that the data processing methods in the first and second embodiments can be implemented based on the data processing device provided in this embodiment. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the data processing device described in this embodiment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0103] According to the data processing apparatus provided in this embodiment, based on multiple clear images acquired by an APS image sensor, a corresponding blurred image is generated; the exposure time interval corresponding to the blurred image is determined; target event stream data within the exposure time interval is obtained from the event stream data acquired by an EVS image sensor; and the blurred image, target event stream data, and ground truth images from the multiple clear images are correlated to obtain training data for the deblurring network model. Through the implementation of this application's solution, image data and event data streams captured by a real camera are used to generate triplet data for training the deblurring network model, reducing data acquisition difficulty and improving the authenticity and effectiveness of the training data.

[0104] Figure 9 A data processing system is provided in the fourth embodiment of this application. This data processing system can be used to implement the data processing methods in the foregoing embodiments, and mainly includes:

[0105] The system includes a memory 901, a processor 902, an APS image sensor 903, and an EVS image sensor 904. The memory 901 stores a computer program 905 that can run on the processor 902. The memory 901 and the processor 902 are communicatively connected. When the processor 902 executes the computer program 905, it implements the data processing method described in the foregoing embodiments. The number of processors 902 can be one or more.

[0106] The memory 901 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk drive. The memory 901 is used to store executable program code, and the processor 902 is coupled to the memory 901.

[0107] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the data processing system described in the above embodiments, and the computer-readable storage medium may be as described above. Figure 9 The memory in the illustrated embodiment.

[0108] The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the data processing method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, a portable hard drive, a read-only memory (ROM), RAM, a magnetic disk, or an optical disk, or any other medium capable of storing program code.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0110] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0111] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0112] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0113] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0114] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0115] The above is a description of the data processing method and related apparatus provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: generating a corresponding blurred image based on a plurality of clear images collected by an APS image sensor; determining an exposure time interval corresponding to the blurred image; obtaining target event stream data within the exposure time interval from event stream data collected by an EVS image sensor; associating the blurred image, the target event stream data and a ground truth image in the plurality of clear images to obtain training data of a deblurring network model; the step of generating a corresponding blurred image based on a plurality of clear images collected by an APS image sensor comprises: for odd-numbered frames of clear images collected by the APS image sensor, performing equidistant interpolation processing on adjacent frames of the clear images in the time dimension; taking the middle image in the odd-numbered frames of clear images as a reference, shifting forward and backward by a target time length respectively to obtain a starting time and an ending time; merging target images between the starting time and the ending time in the plurality of images obtained after interpolation to generate a corresponding blurred image.

2. The data processing method according to claim 1, characterized in that, The target time length satisfies the following condition: t < (sof m -sof n + exposure n ) / 2 t>(sof m -sof n +exposure n ) / (N-1) wherein t represents the target time length, sof m represents the first row exposure end time of the last frame image in the even frame clear image, sof n represents the first row exposure end time of the first frame image in the odd frame clear image, exposure n represents the single row exposure time of the first frame image, and N represents the total frame number of the image obtained after the frame insertion, and N is an integer greater than 1.

3. The data processing method of claim 1, wherein, Before the step of performing equidistant interpolation processing on adjacent frames of the clear images in the time dimension for odd-numbered frames of clear images collected by the APS image sensor, the method further comprises the following steps: obtaining an actual frame rate of the APS image sensor; comparing the actual frame rate with a preset frame rate threshold; if the actual frame rate is lower than the frame rate threshold, performing the step of performing equidistant interpolation processing on adjacent frames of the clear images in the time dimension for odd-numbered frames of clear images collected by the APS image sensor; if the actual frame rate is higher than the frame rate threshold, merging the odd-numbered frames of clear images to generate a corresponding blurred image.

4. The data processing method of claim 1, wherein, The step of determining an exposure time interval corresponding to the blurred image comprises: determining the exposure end time of the first row of the last frame of the clear images as the exposure end time of the first row of the blurred image, and determining the exposure end time of the last row of the last frame of the clear images as the exposure end time of the last row of the blurred image; calculating a single row exposure time of the blurred image based on a preset exposure time calculation formula; the exposure time calculation formula is expressed as: exposure synthetic = sof m -sof n +exposure n , wherein exposure synthetic represents the single row exposure time of the blurred image, sof m represents an end time of first row exposure of the last frame of image, sof n represents an end time of first row exposure of a first frame of image in the plurality of frames of the clear image, exposure n represents the single row exposure time of the first frame of image; determining the exposure time interval corresponding to the blurred image based on the exposure end time of the first row of the blurred image, the exposure end time of the last row of the blurred image and the single-row exposure time of the blurred image.

5. The data processing method of claim 1, wherein, Before the step of associating the blurred image, the target event stream data and the ground truth image in the plurality of clear images, the method further comprises the following step: determining the (M+1)th target clear image in the 2M+1 frames of clear images as the ground truth image, wherein M is an integer greater than 1.

6. The data processing method of claim 1, wherein, The step of associating the blurred image, the target event stream data and the ground truth image in the plurality of clear images to obtain training data of a deblurring network model comprises: obtaining an optical flow between the current frame of the blurred image and the previous frame of the blurred image as a mask; performing denoising processing on the target event stream data based on the mask to obtain denoised target event stream data; Correlate the blurred image of the current frame, the denoised target event stream data, and the true value image in the multiple frames of clear images to obtain training data of a deblurring network model.

7. The data processing method according to any one of claims 1 to 6, characterized in that, The step of correlating the blurred image, the target event stream data, and the true value image in the multiple frames of clear images to obtain training data of a deblurring network model further comprises: Training the deblurring network model using the training data to obtain a trained deblurring network model; Based on the trained deblurring network model, processing the input APS image and event stream data in the same exposure time interval to output an APS image after deblurring processing.

8. The data processing method according to claim 7, characterized in that, The step of processing the input APS image and event stream data in the same exposure time interval based on the trained deblurring network model to output an APS image after deblurring processing comprises: When processing multiple consecutive frames of APS images based on the trained deblurring network model, the current frame of APS image, the current frame of event stream data in the same exposure time interval, the previous frame of APS image, the previous frame of event stream data, and the output previous frame of APS image after deblurring processing are used as inputs of the deblurring network model for processing to output a current frame of APS image after deblurring processing.

9. A data processing apparatus, characterized by, Comprise: A generation module configured to generate a blurred image based on multiple frames of clear images collected by an APS image sensor; A determination module configured to determine an exposure time interval corresponding to the blurred image; An acquisition module configured to acquire target event stream data within the exposure time interval from event stream data collected by an EVS image sensor; An association module configured to correlate the blurred image, the target event stream data, and a true value image in the multiple frames of clear images to obtain training data of a deblurring network model; The step of generating a blurred image based on multiple frames of clear images collected by an APS image sensor comprises: For odd frames of clear images collected by the APS image sensor, performing equidistant interpolation processing on adjacent frames of the clear images in the time dimension; Taking a middle image in the odd frames of clear images as a reference, shifting forward and backward by a target time length respectively to obtain a start time and an end time; Merging target images between the start time and the end time in the multiple frames of images obtained after interpolation to generate a corresponding blurred image.

10. A data processing system, characterized by Comprise an APS image sensor, an EVS image sensor, a memory, and a processor, wherein: The processor is configured to execute a computer program stored in the memory; The processor executes the computer program to implement the steps in the data processing method of any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps in the data processing method of any one of claims 1 to 8.

Images