Image super-resolution reconstruction method and system based on deep learning

Abstract

The present invention proposes an image super-resolution reconstruction method and system based on deep learning, which belongs to the field of image reconstruction technology. The method includes steps S1: obtaining multiple original video frames of a first resolution combination; S2: grouping the multiple original video frames to obtain multiple grouped video frames; S3: performing super-resolution reconstruction for each grouped video frame to obtain multiple reconstructed video frames of a second resolution combination; S4: fusing the multiple reconstructed video frames to obtain super-resolution reconstruction results of the original video frames. The system includes a video frame acquisition unit, a video frame grouping unit, a group reconstruction unit and a video frame fusion unit, which are used to implement the method. The technical solution of the present invention can select a corresponding deep learning model to perform resolution reconstruction based on the timestamp and perspective information of the original video frame, avoiding the perspective difference and time error problems caused by a single reconstruction method.

Description

Technical Field

[0001] The present invention belongs to the technical field of image reconstruction, and in particular relates to an image super-resolution reconstruction method and system based on deep learning. Background Art

[0002] Image super-resolution reconstruction (superresolution, SR) refers to an image processing technology that uses a computer to process a low-resolution image (lowresolution, LR) or image sequence to restore a high-resolution image (highresolution, HR). HR means that the image has a high pixel density and can provide more details, which often play a key role in applications. Super-resolution reconstruction is mainly divided into super-resolution reconstruction of a single image and super-resolution reconstruction of multiple images.

[0003] Important reconstruction models of deep learning mainly involve convolutional neural networks, and their general structure includes convolution layer, fully connected layer, pooling layer, Softmax function, loss function, back propagation algorithm, etc., which are well known to those skilled in the art.

[0004] Among the classic super-resolution reconstruction algorithms based on deep learning, SRCNN is the earliest super-resolution reconstruction algorithm. It first uses bilinear interpolation to scale the image to the desired size, and then uses a nonlinear network for feature extraction and reconstruction, using only two convolutional layers.

[0005] Some reconstruction algorithms or models require some prior knowledge, such as prediction models, edge-based methods, image statistics methods, and block-based (sample-based) methods.

[0006] However, in some super-resolution reconstruction scenarios, such as images produced by light field camera arrays, traditional reconstruction methods using a single neural network model will be obviously unsuitable, and reconstruction algorithms based on prior knowledge will almost fail. This is because when a light field camera produces an image frame, due to different acquisition modes, the shooting perspective, timestamp, etc. change at any time, and the timestamp and perspective information are misaligned, resulting in obvious perspective differences and time errors in the reconstructed structure. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes an image super-resolution reconstruction method and system based on deep learning.

[0008] The first aspect of the present invention proposes an image super-resolution reconstruction method based on deep learning, which is implemented by an image reconstruction model including a multi-scale attention residual network model and a generative adversarial network model in parallel.

[0009] In specific implementation, the method includes the following steps:

[0010] S1: Acquire a plurality of original video frames of a first resolution combination;

[0011] S2: grouping the multiple original video frames to obtain multiple grouped video frames;

[0012] S3: performing super-resolution reconstruction on each grouped video frame to obtain a plurality of reconstructed video frames of a second resolution combination;

[0013] S4: fusing the multiple reconstructed video frames to obtain a super-resolution reconstruction result of the original video frame;

[0014] Wherein, the first resolution combination includes a plurality of first resolutions; the second resolution combination includes a plurality of second resolutions;

[0015] Each of the second resolutions is greater than each of the first resolutions.

[0016] As a further improvement of the above method, step S2 specifically includes:

[0017] Get the timestamp and viewing angle information of each original video frame;

[0018] Grouping the multiple original video frames based on the timestamp and the viewing angle information to obtain multiple grouped video frames;

[0019] The viewing angle information of the video frames in each group is the same and the difference of the timestamps is within a preset time threshold.

[0020] As a further improvement of the above method, step S3 specifically includes:

[0021] Using a deep learning-based image reconstruction model to perform the super-resolution reconstruction;

[0022] The deep learning-based image reconstruction model includes a multi-scale attention residual network model and a generative adversarial network model.

[0023] The multi-scale attention residual network model is connected in parallel with the generative adversarial network model.

[0024] For each group, obtain the number of video frames contained in the group;

[0025] When the number of the video frames is greater than a preset number value, a multi-frame super-resolution reconstruction method is used to perform the super-resolution reconstruction to obtain a reconstructed video frame of a second resolution corresponding to the group;

[0026] When the number of video frames is less than a preset number value, the video frame corresponding to the median timestamp in the group is selected, and the super-resolution reconstruction is performed using a single-frame super-resolution reconstruction method to obtain a reconstructed video frame with a second resolution corresponding to the group.

[0027] In a second aspect of the present application, a deep learning-based image super-resolution reconstruction system is proposed, wherein the reconstruction system comprises a video frame acquisition unit, a video frame grouping unit, a grouping reconstruction unit, and a video frame fusion unit;

[0028] In terms of specific structure, the functions performed by the above-mentioned units are as follows:

[0029] The video frame acquisition unit is used to acquire multiple light field video frames generated by the light field camera array;

[0030] The video frame grouping unit is used to group the multiple light field video frames to obtain multiple grouped video frames;

[0031] The grouping reconstruction unit is used to perform super-resolution reconstruction on each grouped video frame to obtain multiple reconstructed video frames;

[0032] The video frame fusion unit is used to fuse the multiple reconstructed video frames to obtain super-resolution reconstruction results of the multiple light field video frames generated by the light field camera array this time;

[0033] Wherein, the grouping reconstruction unit adopts an image reconstruction model based on deep learning to perform the super-resolution reconstruction.

[0034] The multiple light field video frames generated by the light field camera array have different time stamps and different shooting angles;

[0035] The video frame grouping unit groups the plurality of light field video frames based on the timestamp and the shooting angle.

[0036] a plurality of light field video frames combined at a first resolution generated by the light field camera array;

[0037] The grouping reconstruction unit is used to perform super-resolution reconstruction on each grouped video frame to obtain a plurality of reconstructed video frames of a second resolution combination;

[0038] The first resolution combination includes a plurality of first resolutions; the second resolution combination includes a plurality of second resolutions;

[0039] Each of the second resolutions is greater than each of the first resolutions.

[0040] The grouping reconstruction unit includes a multi-frame super-resolution reconstruction processing unit and a single-frame super-resolution reconstruction processing unit;

[0041] For each group, the group reconstruction unit obtains the number of video frames contained in the group;

[0042] When the number of the video frames is greater than a preset number value, a multi-frame super-resolution reconstruction processing unit is used to perform the super-resolution reconstruction to obtain a reconstructed video frame of a second resolution corresponding to the group;

[0043] When the number of video frames is less than a preset number value, the video frame corresponding to the median timestamp in the group is selected, and the single-frame super-resolution reconstruction processing unit is used to perform the super-resolution reconstruction to obtain a reconstructed video frame of the second resolution corresponding to the group.

[0044] The video frame grouping unit groups the multiple light field video frames based on the timestamp and the shooting angle, and the grouping standard is: the shooting angles of the video frames in each group are the same and the differences of the timestamps are within a preset time threshold.

[0045] The technical solution of the present invention, when performing super-resolution reconstruction based on video frames generated by light field camera arrangement, firstly groups multiple original video frames based on timestamp and viewing angle information to obtain multiple grouped video frames, then selects a corresponding matching super-resolution reconstruction method for each number of grouped video frames to perform super-resolution reconstruction to obtain reconstructed video frames and perform final video fusion, which can ensure the accuracy of the super-resolution reconstruction results of the video frames of the light field camera array and avoid the viewing angle difference and time error problems caused by a single reconstruction method.

[0046] Further advantages of the present invention will be further reflected in detail in the specific embodiments section in conjunction with the drawings of the specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flowchart of an image super-resolution reconstruction method based on deep learning according to an embodiment of the present invention;

[0049] Figure 2-Figure 3 yes Figure 1A schematic diagram of a specific implementation of some sub-steps of the image super-resolution reconstruction method based on deep learning;

[0050] Figure 4 It is a schematic diagram of a unit architecture of an image super-resolution reconstruction system based on deep learning according to an embodiment of the present invention;

[0051] Figure 5 yes Figure 4 A schematic diagram of a further preferred embodiment of the deep learning-based image super-resolution reconstruction system. DETAILED DESCRIPTION

[0052] The invention is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0053] Figure 1 It is a flowchart of an image super-resolution reconstruction method based on deep learning according to an embodiment of the present invention.

[0054] It should be pointed out that the original image targeted by the image super-resolution reconstruction method described in the present invention comes from a video frame captured by a light field camera.

[0055] Specifically, the original image comes from a video frame sequence captured by multiple light field cameras of the light field camera array.

[0056] When shooting videos, a light field camera usually needs to change the shooting angle at any time due to different depth of field requirements and angle shooting requirements.

[0057] Taking the Cartesian coordinate system as an example, the shooting perspectives include perspective 1 parallel to the first coordinate axis, perspective 2 parallel to the second coordinate axis, and perspective 3 parallel to the third coordinate axis; the corresponding depths of field are perspective 1, perspective 2, and perspective 3, respectively.

[0058] The first coordinate axis, the second coordinate axis and the third coordinate axis are selected from one of the XYZ axes.

[0059] Of course, other perspectives of other coordinate systems may also be used, and the present invention is not limited thereto.

[0060] For the convenience of description, the above Cartesian coordinate system is used in the following. The different perspectives are X-axis perspective, Y-axis perspective, and Z-axis perspective, which are expressed as Eye. x ,Eye y ,Eye z However, this does not constitute a limitation of the present invention.

[0061] At the same time, when the light field camera shoots a video, each video contains multiple video frames with a timestamp, which is used to mark the time when the video frame is generated. Different video frames are generated at different times, so their timestamps are different.

[0062] For the convenience of description, the timestamps are represented by a digital sequence t1, t2, t3, ..., and the larger the digital subscript, the later the time of generation. However, this does not constitute a limitation to the present invention.

[0063] In addition, during the shooting process of the light field camera, the shooting mode at each time point and each depth of field direction can also be adjusted, resulting in different resolutions of different video frames.

[0064] For the convenience of description, the resolution of the original video frame is the first resolution, denoted as Pix 1st .

[0065] Therefore, for the n original video frames {F1, F2, …, F n}, each original video frame F i (i=1,2,…,n), each contains at least three attributes:

[0066] Perspective attribute, perspective is selected from Eye x ,Eye y ,Eye z one;

[0067] Original video frame F i The viewing angle is represented by Eye ik ,k∈{x,y,z};

[0068] Timestamp attribute, the timestamp is selected from one of the digital sequences t1, t2, t3, ...;

[0069] Original video frame F i The timestamp is denoted as t i ;

[0070] Resolution attribute, original video frame F i The resolution is Pix 1sti .

[0071] On this basis, Figure 1 The method shown includes the following steps:

[0072] S1: Acquire a plurality of original video frames of a first resolution combination;

[0073] S2: grouping the multiple original video frames to obtain multiple grouped video frames;

[0074] S3: performing super-resolution reconstruction on each grouped video frame to obtain a plurality of reconstructed video frames of a second resolution combination;

[0075] S4: fusing the multiple reconstructed video frames to obtain a super-resolution reconstruction result of the original video frame;

[0076] Wherein, the first resolution combination includes a plurality of first resolutions; the second resolution combination includes a plurality of second resolutions;

[0077] Each of the second resolutions is greater than each of the first resolutions.

[0078] Taking the above exemplary example, step S1 obtains a plurality of original video frames of the first resolution combination as follows:

[0079] Get n original video frames {F1, F2, ..., F n}, each original video frame F i The first resolution of (i=1,2,…,n) is Pix 1st1 ,Pix 1st2 ,…,Pix 1stn ;

[0080] The viewing angle information is: Eye 1k ,Eye 2k ,…Eye ik , k∈{x, y, z}; k∈{x, y, z} means k is selected from one of x, y, z;

[0081] The timestamps are t1, t2, t3, ...;

[0082] It can be seen that Pix 1st1 ,Pix 1st2 ,…,Pix 1stn That is the aforementioned first resolution combination.

[0083] The specific implementation of steps S2-S3 can be found in Figure 2-Figure 3 .

[0084] exist Figure 2 In the step S2, the step S2 specifically includes:

[0085] Get the timestamp and viewing angle information of each original video frame;

[0086] Grouping the multiple original video frames based on the timestamp and the viewing angle information to obtain multiple grouped video frames;

[0087] The viewing angle information of the video frames in each group is the same and the difference of the timestamps is within a preset time threshold.

[0088] It can be seen that the grouping criterion of step S2 is that the shooting angles of the video frames in each group are the same and the differences of the timestamps are within the preset time threshold.

[0089] The shooting angle is the same, that is, if the original video frame F i Corresponding viewing angle information Eyeik The original video frame F j The corresponding viewing angle information Eye jk If k is the same (both x, both y, or both z), then the original video frame F i and the original video frame F j have the same shooting viewing angle.

[0090] The differences in timestamps are all within a preset time threshold, that is, if the original video frame F i corresponds to the timestamp t i and the original video frame F j corresponds to the timestamp t j , if |i - j| ≤ timeThreshold, then the difference in timestamps between the original video frame F i and the original video frame F j is within the preset time threshold.

[0091] When the original video frame F i and the original video frame F j simultaneously satisfy the above two conditions, the original video frame F i and the original video frame F j are grouped into the same group.

[0092] Among them, timeThreshold is the preset time threshold, 1 < timeThreshold < timeUp, timeUP is an integer value set according to needs, and timeUp is not greater than 5.

[0093] Step S2 can group the original videos to obtain multiple groups, and each group contains at least one original video frame;

[0094] Next, see Figure 3 The processing steps for the video frames in each group are as follows:

[0095] For each group, obtain the number of video frames contained in the group;

[0096] When the number of video frames is greater than the preset number value, perform the super-resolution reconstruction using the multi-frame super-resolution reconstruction method to obtain the reconstructed video frames with the second resolution corresponding to the group;

[0097] When the number of video frames is less than the preset number value, select the video frame corresponding to the median timestamp in the group and perform the super-resolution reconstruction using the single-frame super-resolution reconstruction method to obtain the reconstructed video frames with the second resolution corresponding to the group.

[0098] Figure 3 Partially shows the computer pseudo-code implementation form of this step S3 as follows:

[0099] That is, suppose there are multiple groups, and each group is identified by a group ID, group ID = 1, 2, ...,

[0100] The computer pseudo code implementation of step S3 is as follows:

[0101] S301: Set group ID = 1;

[0102] S302: Determine the number of video frames contained in the group ID SumGroup ID Is it greater than the set value SetSum?

[0103] If yes, a multi-frame super-resolution reconstruction method is used to perform the super-resolution reconstruction.

[0104] If not, selecting the video frame corresponding to the median timestamp in the group, and performing the super-resolution reconstruction using a single-frame super-resolution reconstruction method;

[0105] It is understandable that in some extreme cases, there is no median, for example, if the number of video frames contained in the group ID SumGroup ID If it is equal to 1, there is only one single frame available. If the group ID contains the number of video frames SumGroup ID If it is equal to 2, you can choose any one of them or their average value;

[0106] Of course, depending on the set threshold, more than two can be obtained, thus ensuring the existence of the median;

[0107] It can be understood that the set value SetSum can be set based on experience, for example, the value SetSum is equal to 3; preferably, the value SetSum is equal to 5;

[0108] S303: Obtain a reconstructed video frame of a second resolution corresponding to the group;

[0109] S304: Increment the group ID by 1 and return to step S302 until all groups have been processed.

[0110] More specifically, the step S3 uses a deep learning-based image reconstruction model to perform the super-resolution reconstruction;

[0111] The deep learning-based image reconstruction model includes a multi-scale attention residual network model and a generative adversarial network model.

[0112] As a further preference, the multi-scale attention residual network model is connected in parallel with the generative adversarial network model.

[0113] The inventors noticed that traditional deep learning super-resolution reconstruction models usually use a series connection of multiple modules and cannot fully utilize the contextual association information between features in each layer.

[0114] To this end, this embodiment adopts a deep learning reconstruction model including a multi-scale attention residual network model and a generative adversarial network model, and the multi-scale attention residual network model is connected in parallel with the generative adversarial network model, which can effectively obtain the contextual association information between the features of each layer.

[0115] To achieve Figure 1-Figure 3 For the method steps, see Figure 4 , Figure 4 It is a schematic diagram of the unit architecture of an image super-resolution reconstruction system based on deep learning according to an embodiment of the present invention.

[0116] exist Figure 4 In the embodiment, the reconstruction system includes a video frame acquisition unit, a video frame grouping unit, a grouping reconstruction unit and a video frame fusion unit.

[0117] Specifically, the video frame acquisition unit is used to acquire a plurality of light field video frames generated by a light field camera array;

[0118] The video frame grouping unit is used to group the multiple light field video frames to obtain multiple grouped video frames;

[0119] The grouping reconstruction unit is used to perform super-resolution reconstruction on each grouped video frame to obtain multiple reconstructed video frames;

[0120] The video frame fusion unit is used to fuse the multiple reconstructed video frames to obtain super-resolution reconstruction results of the multiple light field video frames generated by the light field camera array this time;

[0121] Preferably, Figure 4 The system also includes a video monitoring terminal for receiving the super-resolution reconstruction result and playing and displaying it on a human-computer interface.

[0122] Wherein, the grouping reconstruction unit adopts an image reconstruction model based on deep learning to perform the super-resolution reconstruction.

[0123] The multiple light field video frames generated by the light field camera array have different time stamps and different shooting angles;

[0124] The video frame grouping unit groups the plurality of light field video frames based on the timestamp and the shooting angle.

[0125] a plurality of light field video frames combined at a first resolution generated by the light field camera array;

[0126] The grouping reconstruction unit is used to perform super-resolution reconstruction on each grouped video frame to obtain a plurality of reconstructed video frames of a second resolution combination;

[0127] The first resolution combination includes multiple first resolutions; the second resolution combination includes multiple second resolutions; each of the second resolutions is larger than each of the first resolutions.

[0128] exist Figure 4 Based on this, see Figure 5 . Figure 5 Show Figure 4 A schematic diagram of a further preferred embodiment of the deep learning-based image super-resolution reconstruction system.

[0129] exist Figure 5 in Figure 4 On the basis, the grouping reconstruction unit includes a multi-frame super-resolution reconstruction processing unit and a single-frame super-resolution reconstruction processing unit;

[0130] For each group, the group reconstruction unit obtains the number of video frames contained in the group;

[0131] When the number of the video frames is greater than a preset number value, a multi-frame super-resolution reconstruction processing unit is used to perform the super-resolution reconstruction to obtain a reconstructed video frame of a second resolution corresponding to the group;

[0132] When the number of video frames is less than a preset number value, the video frame corresponding to the median timestamp in the group is selected, and the single-frame super-resolution reconstruction processing unit is used to perform the super-resolution reconstruction to obtain a reconstructed video frame of the second resolution corresponding to the group.

[0133] The video frame grouping unit groups the multiple light field video frames based on the timestamp and the shooting angle, and the grouping standard is: the shooting angles of the video frames in each group are the same and the differences of the timestamps are within a preset time threshold.

[0134] The technical solution of the present invention, when performing super-resolution reconstruction based on video frames generated by light field camera arrangement, firstly groups multiple original video frames based on timestamp and viewing angle information to obtain multiple grouped video frames, then selects a corresponding matching super-resolution reconstruction method for each number of grouped video frames to perform super-resolution reconstruction to obtain reconstructed video frames and perform final video fusion, which can ensure the accuracy of the super-resolution reconstruction results of the video frames of the light field camera array and avoid the viewing angle difference and time error problems caused by a single reconstruction method.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

[0136] The present invention does not particularly specify the structure of some modules, which shall be subject to the contents recorded in the prior art. The prior art mentioned in the above background technology section of the present invention can be used as part of the present invention to understand the meaning of some technical features or parameters. The scope of protection of the present invention shall be subject to the contents actually recorded in the claims.

Claims

1. A method for image super-resolution reconstruction based on deep learning, characterized in that: The method comprises the following steps: S1: Acquire a plurality of original video frames of a first resolution combination; S2: grouping the multiple original video frames to obtain multiple grouped video frames; S3: performing super-resolution reconstruction on each grouped video frame to obtain a plurality of reconstructed video frames of a second resolution combination; S4: fusing the multiple reconstructed video frames to obtain a super-resolution reconstruction result of the original video frame; Wherein, the first resolution combination includes a plurality of first resolutions; the second resolution combination includes a plurality of second resolutions; each of the second resolutions is greater than each of the first resolutions; The step S2 specifically includes: Get the timestamp and viewing angle information of each original video frame; Grouping the multiple original video frames based on the timestamp and the viewing angle information to obtain multiple grouped video frames; The video frames in each group have the same viewing angle information and the difference of the timestamps is within a preset time threshold; The step S3 specifically comprises: For each group, obtain the number of video frames contained in the group; When the number of the video frames is greater than a preset number value, a multi-frame super-resolution reconstruction method is used to perform the super-resolution reconstruction to obtain a reconstructed video frame of a second resolution corresponding to the group; When the number of video frames is less than a preset number value, the video frame corresponding to the median timestamp in the group is selected, and the super-resolution reconstruction is performed using a single-frame super-resolution reconstruction method to obtain a reconstructed video frame with a second resolution corresponding to the group.

2. The image super-resolution reconstruction method based on deep learning according to claim 1, characterized in that: The step S3 uses a deep learning-based image reconstruction model to perform the super-resolution reconstruction; The deep learning-based image reconstruction model includes a multi-scale attention residual network model and a generative adversarial network model.

3. The image super-resolution reconstruction method based on deep learning as claimed in claim 2, characterized in that: The multi-scale attention residual network model is connected in parallel with the generative adversarial network model.

4. A deep learning-based image super-resolution reconstruction system, the reconstruction system comprising a video frame acquisition unit, a video frame grouping unit, a grouping reconstruction unit and a video frame fusion unit, Features: The video frame acquisition unit is used to acquire multiple light field video frames generated by the light field camera array; The video frame grouping unit is used to group the multiple light field video frames to obtain multiple grouped video frames; The grouping reconstruction unit is used to perform super-resolution reconstruction on each grouped video frame to obtain multiple reconstructed video frames; the video frame fusion unit is used to fuse the multiple reconstructed video frames to obtain super-resolution reconstruction results of the multiple light field video frames generated by the light field camera array this time; Wherein, the grouping reconstruction unit uses an image reconstruction model based on deep learning to perform the super-resolution reconstruction; The multiple light field video frames generated by the light field camera array have different time stamps and different shooting angles; The video frame grouping unit groups the plurality of light field video frames based on the timestamp and the shooting angle; The video frames in each group have the same viewing angle information and the difference of the timestamps is within a preset time threshold; A plurality of light field video frames combined at a first resolution generated by the light field camera array; The grouping reconstruction unit is used to perform super-resolution reconstruction on each grouped video frame to obtain a plurality of reconstructed video frames of a second resolution combination; The first resolution combination includes a plurality of first resolutions; the second resolution combination includes a plurality of second resolutions; each of said second resolutions is greater than each of said first resolutions; The grouping reconstruction unit includes a multi-frame super-resolution reconstruction processing unit and a single-frame super-resolution reconstruction processing unit; For each group, the group reconstruction unit obtains the number of video frames contained in the group; When the number of the video frames is greater than a preset number value, a multi-frame super-resolution reconstruction processing unit is used to perform the super-resolution reconstruction to obtain a reconstructed video frame of a second resolution corresponding to the group; When the number of video frames is less than a preset number value, the video frame corresponding to the median timestamp in the group is selected, and the single-frame super-resolution reconstruction processing unit is used to perform the super-resolution reconstruction to obtain a reconstructed video frame of the second resolution corresponding to the group.

Images