Image processing method and device, processor, and storage medium

Abstract

The application relates to an image processing method and device, a processor and a storage medium. The image processing method comprises the following steps: a first device encodes an acquired image to generate a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is higher than that of the second resolution image frame, and the number of the first resolution image frames is less than that of the second resolution image frames; the first device packages the first resolution image frame and the second resolution image frame into a video stream, and sends the video stream to a second device; and the second device reconstructs an image corresponding to the second resolution image frame by taking an image corresponding to the first resolution image frame as a reference. In the premise of guaranteeing backhaul video quality under limited uplink bandwidth resources, more ultra-high-definition IPCs in the same base station cell are accessed, and the back end can realize real-time reconstruction of the video.

Description

Technical Field

[0001] This application relates to the field of video technology, and in particular to an image processing method and apparatus, processor, and storage medium. Background Technology

[0002] Video is composed of individual frames. Video capture devices encode the raw images they capture using a video encoder (such as H.264 / AVC, H.265 / HEVC, or H.266 / VVC encoders), generating image frames. These image frames are then sent to video processing devices for further processing. Encoded image frames can be categorized into three types: I-frames, P-frames, and B-frames. The video capture device can be a camera, a terminal device with a camera, etc., while the video processing device can be a video server, etc.

[0003] I-frames, also called keyframes, are produced by intra-frame encoding of raw images using a video encoder. I-frames utilize the spatial correlation within a single frame but not the temporal correlation between frames, resulting in a relatively low compression ratio. In short, I-frames can be encoded and decoded independently, without relying on other frames. Because they are independent, I-frames serve as the entry point for random access and also as the reference frame for decoding. I-frames appear periodically in the image sequence, and the encoder can select their frequency. An important subset of I-frames is called IDR (Instantaneous Decoding Refresh) frames. IDR frames immediately refresh the decoding, preventing errors from propagating. IDR frames empty the DPB (Decoded Picture Buffer, also called the reference frame list), while ordinary I-frames do not. All frames after an IDR frame cannot reference the content of frames preceding it, but B / P frames after an ordinary I-frame can reference frames preceding it.

[0004] P-frames, also called predictive pictures, belong to the forward-predictive inter-frame coding method. P-frames use motion compensation to transmit the difference and motion vectors between themselves and the preceding I or P-frames. When decoding a P-frame, the predicted values ​​from the I-frame are summed with the prediction error to reconstruct the complete P-frame image. Because it uses interpolation for transmission, P-frames can improve compression efficiency.

[0005] B-frames, also known as bi-predictive pictures, are a type of bi-directional predictive coded frame. They use preceding I or P frames and following P frames as reference frames, transmitting the prediction error and motion vectors between themselves and these frames. Because B-frames can use future frames as references, the transmission order and display order of image frames in the MPEG-2 encoded stream can be different. B-frames employ bi-directional time prediction, which can significantly improve compression ratios.

[0006] Image frames obtained after encoding the original image by the video encoder can be stored and transmitted over the network through NAL units. An image frame can be composed of one or more NAL units. Figure 1 A schematic diagram of the NAL unit structure in related technologies is shown. For example... Figure 1 As shown, a NAL unit may include a header and a body. The header may contain a Type field, which indicates the frame type of the image frame. The body is used to record data.

[0007] An I-frame and the frame preceding the next I-frame form a GOP (Group of Pictures). The number of frames contained within a GOP is called the GOP length. If an encoder encodes a video stream with a GOP length of 50, then each I-frame in the encoded video stream is followed by 49 P and / or B frames.

[0008] 5G (5th generation mobile networks or 5th generation wireless systems, 5th-Generation, the fifth generation of mobile communication technology, abbreviated as 5G or 5G technology) has made significant progress compared with the previous generation of wireless communication technology in terms of network bandwidth, latency and concurrency. Therefore, for scenarios where large-scale cabling is not feasible, such as industrial parks, mines and ports, high-definition and ultra-high-definition video surveillance via 5G wireless backhaul is gradually becoming a major industry demand. Summary of the Invention

[0009] In view of this, an image processing method, device, processor, and storage medium are proposed, which enables more ultra-high-definition IPCs to be accessed in the same base station cell under the premise of limited uplink bandwidth resources and ensuring the quality of back-transmitted video, and the backend can realize real-time reconstruction of video.

[0010] In a first aspect, embodiments of this application provide an image processing method in which a first device encodes a captured image to generate a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of the first resolution image frames is less than the number of the second resolution image frames; the first device packages the first resolution image frames and the second resolution image frames into a video stream, and sends the video stream to a second device.

[0011] Compared to image processing methods in related technologies, the image processing method of this application has a lower frame rate for high-resolution image frames obtained by encoding the acquired images by the first device, and most of the encoded image frames are low-resolution image frames. In other words, the number of high-resolution image frames is relatively small. Therefore, the amount of data sent by the first device to the second device is smaller, and the bandwidth occupied by the video stream is narrower. Thus, with the same bandwidth, more video streams can be accessed.

[0012] In a first possible implementation of the first aspect, the first device encodes an image to generate a first resolution image frame and a second resolution image frame, including: the first device encodes the acquired image to generate a high-resolution image frame and a low-resolution image frame; wherein the low-resolution image frame is the second resolution frame; and the first device extracts a portion of the high-resolution image frame as the first resolution image frame.

[0013] The frame extraction process described in the above embodiments of this application can reuse the high-resolution bitstream and low-resolution bitstream already generated by the camera. By extracting frames from the already generated high-resolution bitstream and low-resolution bitstream, a first resolution image frame and a second resolution image frame can be obtained. In other words, the image frames of the main bitstream and sub-bitstream in the dual bitstream technology can be reused without secondary encoding, without additional computing power, and without hardware dependence.

[0014] In a second possible implementation of the first aspect, the first device encodes the acquired image to generate a first resolution image frame and a second resolution image frame, including: the first device extracting all or part of the acquired image and encoding it to generate a second resolution image frame, and extracting part of the acquired image and encoding it to generate a first resolution image frame. The frame extraction method in the above embodiments of this application, since it performs frame extraction on the acquired image before encoding, reduces the amount of image encoding required, thus improving encoding efficiency.

[0015] In conjunction with the first aspect, or the first or second possible implementation of the first aspect, in a third possible implementation, the first device packages the first resolution image frame and the second resolution image frame into a video stream, including: the first device packages the first resolution image frame and the second resolution image frame into a first video stream; and / or, the first device packages the first resolution image frame into a second video stream and packages the second resolution image frame into a third video stream.

[0016] In conjunction with the third possible implementation of the first aspect, in the fourth possible implementation, the second resolution image frame contains a source image frame corresponding to the first resolution image frame. The first resolution image frame and the corresponding source image frame are generated by encoding the same image. In the first video stream, the first resolution image frame and the corresponding source image frame are arranged according to rules, or the first resolution image frame and the corresponding source image frame are combined into a combined frame.

[0017] The above packetization method can maintain a consistent correspondence between the high-resolution and low-resolution image frames acquired by the first device and the video processor. In this way, the second device can directly use the pairs of high-resolution and low-resolution image frames as a reference for low-resolution image reconstruction, simplifying the processing of the second device.

[0018] In conjunction with the fourth possible implementation of the first aspect, in the fifth possible implementation, in the first video stream, the first resolution image frame is adjacent to the corresponding source image frame.

[0019] In conjunction with the fourth possible implementation of the first aspect, in the sixth possible implementation, the first device packages the first resolution image frame and the second resolution image frame into a video stream, including:

[0020] The first device inserts the Network Abstraction Layer (NAL) packet of the first resolution image frame into the NAL packet of the corresponding source image frame to form the combined frame.

[0021] In conjunction with the third possible implementation of the first aspect, in the seventh possible implementation, the value of the Type field of the NAL packet of the first resolution image frame and the value of the Type field of the NAL packet of the second resolution image frame are different in the first video stream.

[0022] Secondly, embodiments of this application provide an image processing method in which a second device processes a received video stream to obtain a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of the first resolution image frames is less than the number of the second resolution image frames; the second device uses the first resolution image decoded from the first resolution image frame as a reference to reconstruct the second resolution image decoded from the second resolution image frame.

[0023] Because fewer high-resolution images are used as references for super-resolution reconstruction of low-resolution images, the image processing method of the present application can achieve image super-resolution reconstruction faster, reduce computation time, and achieve real-time reconstruction of video.

[0024] In related technologies, a large number of images are pre-collected to train a super-resolution reconstruction neural network. However, since the actual application scenario is often not necessarily related to the pre-collected scenario, the super-resolution reconstruction neural network structure is relatively complex, the training and inference process is time-consuming, and the image reconstruction quality cannot be guaranteed in order to improve the accuracy of super-resolution reconstruction. In contrast, the image processing method according to the embodiments of this application uses a high-resolution image of the current scene as a reference to perform super-resolution reconstruction on a low-resolution image. Therefore, the scene correlation is relatively strong, which can achieve high-quality super-resolution reconstruction and further shorten the computation time while improving the quality of the reconstructed image.

[0025] In the first possible implementation of the second aspect, the second device processes the received video stream to obtain a first resolution image frame and a second resolution image frame, including:

[0026] If the received video stream is a first video stream, the second device processes the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frame in the first video stream to obtain a first resolution image frame and a second resolution image frame.

[0027] In conjunction with the first possible implementation of the second aspect, in the second possible implementation, the second device processes the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frame in the first video stream to obtain a first resolution image frame and a second resolution image frame, including:

[0028] If the value of the Type field of the NAL packet of an image frame in the first video stream is a first extended value, then the second device processes the image frame to obtain a first resolution image frame.

[0029] If the Type field of the NAL packet of an image frame in the first video stream has a non-extended value, the second device processes the image frame to obtain a second resolution image frame.

[0030] In conjunction with the first possible implementation of the second aspect, in the third possible implementation, the second device processes the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frame in the first video stream to obtain a first resolution image frame and a second resolution image frame, including:

[0031] If the first video stream includes a composite frame, the second device processes the composite frame to obtain a first resolution image frame and a second resolution image frame corresponding to the first resolution image frame. The second resolution image frame corresponding to the first resolution image frame and the first resolution image frame are obtained by encoding the same image. The value of the Type field of the NAL packet of the composite frame is a second extended value.

[0032] In conjunction with the second aspect, in a fourth possible implementation, the second device processes the received video stream to obtain a first resolution image frame and a second resolution image frame, including:

[0033] If the received video stream includes a second video stream and a third video stream, the second device processes the second video stream and the third video stream respectively to obtain a first resolution image frame corresponding to the second video stream and a second resolution image frame corresponding to the third video stream.

[0034] In conjunction with the second aspect, or any one of the first to fourth possible implementations of the second aspect, in the fifth possible implementation, the second device uses the first resolution image decoded from the first resolution image frame as a reference to reconstruct the second resolution image decoded from the second resolution image frame, including:

[0035] The second device decodes the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image;

[0036] The second device inputs the first resolution image, the image corresponding to the first resolution image, and the second resolution image into the neural network to perform super-resolution reconstruction on the second resolution image;

[0037] Wherein, the image corresponding to the first resolution image is the image in the second resolution image that corresponds to the first resolution image; the image in the second resolution image that corresponds to the first resolution image is the same image acquired by the first device as the first resolution image;

[0038] Alternatively, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image.

[0039] In conjunction with the fifth possible implementation of the second aspect, in the sixth possible implementation, the second device inputs the first resolution image, the image corresponding to the first resolution image, and the second resolution image into a neural network to perform super-resolution reconstruction on the second resolution image, including:

[0040] The second device convolves the second resolution image to obtain a first feature map, and convolves the image corresponding to the first resolution image to obtain a second feature map;

[0041] The second device stitches the first feature map and the second feature map together to obtain a first stitched feature map;

[0042] The second device performs convolution on the first spliced ​​feature map to obtain the third feature map;

[0043] The second device convolves the first resolution image to obtain a fourth feature map;

[0044] The second device splices the third feature map and the fourth feature map to obtain a second spliced ​​feature map;

[0045] The second device performs convolution on the second stitched feature map to obtain the reconstructed image of the second resolution image.

[0046] The image processing method according to the embodiments of this application uses a high-resolution image of the current scene as a reference to train and infer a super-resolution neural network. A simple network structure is employed to achieve high-quality super-resolution reconstruction, further reducing computation time while improving the quality of the reconstructed image. Furthermore, a small number of reference image pairs allows for asynchronous parallel training and inference. The network parameters generated during online training can be updated in real time to guide the reconstruction process of subsequent frames, ensuring the quality of the reconstructed image.

[0047] In a seventh possible implementation, combining the second aspect or any one of the first to fourth possible implementations of the second aspect, the second device uses the first resolution image decoded from the first resolution image frame as a reference to reconstruct the second resolution image decoded from the second resolution image frame, including:

[0048] The second device decodes the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image;

[0049] The second device inputs the first resolution image and the image corresponding to the first resolution image into the first neural network for training to obtain network parameters; wherein, the image corresponding to the first resolution image is the image in the second resolution image that corresponds to the first resolution image; the image in the second resolution image that corresponds to the first resolution image is the same image acquired by the first device as the first resolution image; or, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image.

[0050] The second device inputs the second resolution image into the second neural network for processing, and performs super-resolution reconstruction on the second resolution image;

[0051] The network parameters of the second neural network are obtained by real-time synchronization based on the network parameters of the first neural network.

[0052] The super-resolution reconstruction method described in the above embodiments of this application can use a high-resolution image of the current scene as a reference for training and inference of the super-resolution neural network. Therefore, a simple network structure can achieve high-quality super-resolution reconstruction, improving the quality of the reconstructed image while reducing computation time. Furthermore, a small number of reference image pairs allows for asynchronous parallel training and inference. The network parameters generated during online training can be updated in real time to guide the reconstruction process of subsequent frames, ensuring the quality of the reconstructed image.

[0053] Thirdly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0054] The encoding unit is used to encode the acquired image to generate a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of the first resolution image frames is less than the number of the second resolution image frames.

[0055] The packet assembly unit is used to assemble the first resolution image frame and the second resolution image frame into a video stream, and send the video stream to the second device.

[0056] Compared to image processing in related technologies, the image processing apparatus of this application has a low frame rate for high-resolution image frames obtained by encoding the acquired images by the first device, and most of the encoded image frames are low-resolution image frames, meaning that the number of high-resolution image frames is relatively small. Therefore, the amount of data sent from the first device to the second device is small, and the bandwidth occupied by the video stream is narrow. Thus, with the same bandwidth, more video streams can be accessed.

[0057] In the first possible implementation of the third aspect, the encoding unit includes:

[0058] The first encoding subunit is used to encode the acquired image to generate a high-resolution image frame and a low-resolution image frame; wherein the low-resolution image frame is a second-resolution frame.

[0059] A frame extraction unit is used to extract a portion of frames from a high-resolution image frame as a first-resolution image frame.

[0060] The frame extraction process described in the above embodiments of this application can reuse the high-resolution and low-resolution bitstreams already generated by the camera. By extracting frames from the already generated high-resolution and low-resolution bitstreams, a first-resolution image frame and a second-resolution image frame can be obtained. In other words, the image frames of the main bitstream and sub-bitstream in the dual-bitstream technology can be reused without secondary encoding, without additional computing power, and without hardware dependence.

[0061] In a second possible implementation of the third aspect, the encoding unit includes:

[0062] The second encoding subunit is used to extract all or part of the acquired images for encoding to generate a second-resolution image frame, and to extract part of the acquired images for encoding to generate a first-resolution image frame. The frame extraction method described in the above embodiments of this application performs frame extraction on the acquired images before encoding, thus reducing the amount of image encoding required and improving encoding efficiency.

[0063] In conjunction with the third aspect, or the first or second possible implementation of the third aspect, in the third possible implementation, the packaging unit includes:

[0064] The first group of packet units is used to package the first resolution image frame and the second resolution image frame into a first video stream;

[0065] And / or,

[0066] The second group of sub-units is used to package the first resolution image frames into a second video stream, and to package the second resolution image frames into a third video stream.

[0067] In conjunction with the third possible implementation of the third aspect, in the fourth possible implementation, the second resolution image frame contains a source image frame corresponding to the first resolution image frame. The first resolution image frame and the corresponding source image frame are generated by encoding the same image. In the first video stream, the first resolution image frame and the corresponding source image frame are arranged according to rules, or the first resolution image frame and the corresponding source image frame are combined into a combined frame. The above packetization method can maintain a consistent correspondence between the high-resolution image frames and low-resolution image frames obtained by the first device and the video processing unit. In this way, the second device can directly use the paired high-resolution image frames and low-resolution image frames as a reference for reconstructing the low-resolution image, simplifying the processing of the second device.

[0068] In conjunction with the fourth possible implementation of the third aspect, in the fifth possible implementation, in the first video stream, the first resolution image frame is adjacent to the corresponding source image frame.

[0069] In conjunction with the fourth possible implementation of the third aspect, in the sixth possible implementation, the packet assembly unit includes:

[0070] The third group of sub-units is used to insert the Network Abstraction Layer (NAL) packet of the first resolution image frame into the NAL packet of the corresponding source image frame to form the combined frame.

[0071] In a seventh possible implementation, in combination with any of the third possible implementations of the third aspect, the value of the Type field of the NAL packet of the first resolution image frame and the value of the Type field of the NAL packet of the second resolution image frame are different in the first video stream.

[0072] Fourthly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0073] The processing unit is configured to process the received video stream to obtain a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of the first resolution image frames is less than the number of the second resolution image frames.

[0074] The reconstruction unit is used to reconstruct the second resolution image after decoding the second resolution image frame, using the first resolution image frame decoded as a reference.

[0075] Because fewer high-resolution images are used as references for super-resolution reconstruction of low-resolution images, the image processing apparatus of the present application can achieve image super-resolution reconstruction faster, reduce computation time, and achieve real-time reconstruction of video.

[0076] In related technologies, a large number of images are pre-collected to train a super-resolution reconstruction neural network. However, since the actual application scenario is often not related to the pre-collected scenario, the super-resolution reconstruction neural network structure is relatively complex, the training and inference process is time-consuming, and the image reconstruction quality cannot be guaranteed in order to improve the accuracy of super-resolution reconstruction. In contrast, the image processing apparatus according to the embodiments of this application uses a high-resolution image of the current scene as a reference to perform super-resolution reconstruction on a low-resolution image. Therefore, the scene correlation is relatively strong, and high-quality super-resolution reconstruction can be achieved. This further reduces the computation time while improving the quality of the reconstructed image.

[0077] In conjunction with the first possible implementation of the fourth aspect, the processing unit includes:

[0078] The first processing subunit is configured to, if the received video stream is a first video stream, process the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frame in the first video stream to obtain a first resolution image frame and a second resolution image frame.

[0079] In conjunction with the first possible implementation of the fourth aspect, in the second possible implementation, the first processing subunit is further used for

[0080] If the Type field of the NAL packet of an image frame in the first video stream has a first extended value, the image frame is processed to obtain a first resolution image frame; if the Type field of the NAL packet of an image frame in the first video stream has a non-extended value, the image frame is processed to obtain a second resolution image frame.

[0081] Combining the first possible implementation of the fourth aspect, in the third possible implementation,

[0082] The first processing subunit is further configured to, if the first video stream includes a combined frame, process the combined frame to obtain a first resolution image frame and a second resolution image frame corresponding to the first resolution image frame, wherein the second resolution image frame corresponding to the first resolution image frame and the first resolution image frame are obtained by encoding the same image.

[0083] In conjunction with the fourth aspect, in a fourth possible implementation, the processing unit further includes:

[0084] The second processing subunit is configured to process the second video stream and the third video stream respectively if the received video stream includes a second video stream and a third video stream, to obtain a first resolution image frame corresponding to the second video stream and a second resolution image frame corresponding to the third video stream.

[0085] In combination with the fourth aspect, or any one of the first to fourth possible implementations of the fourth aspect, in the fifth possible implementation, the reconstruction unit includes:

[0086] The decoding subunit is used to decode the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image;

[0087] The first reconstruction subunit is used to input the first resolution image, the image corresponding to the first resolution image, and the second resolution image into the neural network to perform super-resolution reconstruction on the second resolution image.

[0088] Wherein, the image corresponding to the first resolution image is the image in the second resolution image that corresponds to the first resolution image; the image in the second resolution image that corresponds to the first resolution image is the same image acquired by the first device as the first resolution image;

[0089] Alternatively, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image.

[0090] In conjunction with the fifth possible implementation of the fourth aspect, in the sixth possible implementation, the first reconstruction subunit is further configured to:

[0091] The second resolution image is convolved to obtain the first feature map, and the image corresponding to the first resolution image is convolved to obtain the second feature map;

[0092] The first feature map and the second feature map are concatenated to obtain a first concatenated feature map;

[0093] Convolution is performed on the first spliced ​​feature map to obtain the third feature map;

[0094] Convolve the first resolution image to obtain the fourth feature map;

[0095] The third feature map and the fourth feature map are concatenated to obtain the second concatenated feature map;

[0096] The second stitched feature map is convolved to obtain the reconstructed image of the second resolution image.

[0097] The image processing apparatus according to the embodiments of this application uses a high-resolution image of the current scene as a reference to train and infer a super-resolution neural network. Employing a simple network structure, it achieves high-quality super-resolution reconstruction, further reducing computation time while improving the quality of the reconstructed image. Furthermore, the use of a small number of reference image pairs allows for asynchronous parallel training and inference. The network parameters generated during online training can be updated in real time to guide the reconstruction process of subsequent frames, ensuring the quality of the reconstructed image.

[0098] In combination with the fourth aspect, or any one of the first to fourth possible implementations of the fourth aspect, in the seventh possible implementation, the reconstruction unit further includes:

[0099] The decoding subunit is used to decode the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image;

[0100] The training subunit is used to input the first resolution image and the image corresponding to the first resolution image into the first neural network for training to obtain network parameters; wherein, the image corresponding to the first resolution image is the image corresponding to the first resolution image in the second resolution image; the image corresponding to the first resolution image in the second resolution image is the same image acquired by the first device corresponding to the first resolution image; or, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image.

[0101] The inference subunit is used to input the second resolution image into the second neural network for computation and to perform super-resolution reconstruction of the second resolution image.

[0102] The network parameters of the second neural network are obtained by real-time synchronization based on the network parameters of the first neural network.

[0103] The super-resolution reconstruction process described in the above embodiments of this application can use a high-resolution image of the current scene as a reference for training and inference of the super-resolution neural network. Therefore, a simple network structure can achieve high-quality super-resolution reconstruction, improving the quality of the reconstructed image while reducing computation time. Furthermore, a small number of reference image pairs allows for asynchronous parallel training and inference. The network parameters generated during online training can be updated in real time to guide the reconstruction process of subsequent frames, ensuring the quality of the reconstructed image.

[0104] Fifthly, embodiments of this application provide an image processing method in which a first device encodes a captured image to generate two video streams with different resolutions: a high-resolution main stream and a low-resolution sub-stream; wherein the high-resolution main stream includes high-resolution image frames, and the low-resolution sub-stream includes low-resolution image frames; the first device extracts a reference high-resolution image frame from the high-resolution image frame, and combines the reference high-resolution image frame with the low-resolution image frame to generate a first stream; the first device sends the first stream to a second device.

[0105] Compared to image processing methods in related technologies, the image processing method of this application has a lower frame rate for high-resolution image frames obtained by encoding the acquired images by the first device, and most of the encoded image frames are low-resolution image frames. In other words, the number of high-resolution image frames is relatively small. Therefore, the amount of data sent by the first device to the second device is smaller, and the bandwidth occupied by the video stream is narrower. Thus, with the same bandwidth, more video streams can be accessed.

[0106] The frame extraction process described in the above embodiments of this application can reuse the high-resolution bitstream and low-resolution bitstream already generated by the camera. By extracting frames from the already generated high-resolution bitstream and low-resolution bitstream, a first resolution image frame and a second resolution image frame can be obtained. In other words, the image frames of the main bitstream and sub-bitstream in the dual bitstream technology can be reused without secondary encoding, without additional computing power, and without hardware dependence.

[0107] In a first possible implementation of the fifth aspect, in the first bitstream, the value of the Type field of the NAL packet of the high-resolution image frame is different from the value of the Type field of the NAL packet of the low-resolution image frame.

[0108] In conjunction with the fifth aspect or the first possible implementation of the fifth aspect, in the second possible implementation, in the first bitstream, the reference high-resolution image frame is located adjacent to the corresponding low-resolution image frame, and the reference high-resolution image frame and the corresponding low-resolution image frame are generated by encoding the same image.

[0109] The above packetization method can maintain a consistent correspondence between the high-resolution and low-resolution image frames acquired by the first device and the video processor. In this way, the second device can directly use the pairs of high-resolution and low-resolution image frames as a reference for low-resolution image reconstruction, simplifying the processing of the second device.

[0110] In conjunction with the second possible implementation of the fifth aspect, in the third possible implementation, the reference high-resolution image frame is arranged after the corresponding low-resolution image frame in the first bitstream.

[0111] In conjunction with the fourth possible implementation of the fifth aspect, the method further includes:

[0112] The first device will package reference high-resolution image frames into a second bitstream.

[0113] The first device sends the second bitstream and the sub-bitstream to the processing terminal.

[0114] The above package assembly method is compatible with existing encoders.

[0115] In a sixth aspect, embodiments of this application provide an image processing method in which a second device processes a received bitstream to obtain a reference high-resolution image frame and a low-resolution image frame, wherein the number of the reference high-resolution image frames is less than the number of the low-resolution image frames; the second device uses the reference high-resolution image after the reference high-resolution image frame is encoded as a reference to reconstruct the low-resolution image after the low-resolution image frame is encoded.

[0116] Because fewer high-resolution images are used as references for super-resolution reconstruction of low-resolution images, the image processing method of the present application can achieve image super-resolution reconstruction faster, reduce computation time, and achieve real-time reconstruction of video.

[0117] In related technologies, a large number of images are pre-collected to train a super-resolution reconstruction neural network. However, since the actual application scenario is often not necessarily related to the pre-collected scenario, the super-resolution reconstruction neural network structure is relatively complex, the training and inference process is time-consuming, and the image reconstruction quality cannot be guaranteed in order to improve the accuracy of super-resolution reconstruction. In contrast, the image processing method according to the embodiments of this application uses a high-resolution image of the current scene as a reference to perform super-resolution reconstruction on a low-resolution image. Therefore, the scene correlation is relatively strong, which can achieve high-quality super-resolution reconstruction and further shorten the computation time while improving the quality of the reconstructed image.

[0118] In conjunction with the first possible implementation of the sixth aspect, the second device processes the received bitstream to obtain a reference high-resolution image frame and a low-resolution image frame, including:

[0119] If the received bitstream is a first bitstream, the second device processes the image frames in the first bitstream according to the value of the Type field of the NAL packet of the image frame in the first bitstream to obtain a reference high-resolution image frame and a low-resolution image frame.

[0120] In conjunction with the first possible implementation of the sixth aspect, in the second possible implementation, the second device processes the image frames in the first bitstream according to the value of the Type field of the NAL packet of the image frame in the first bitstream to obtain a reference high-resolution image frame and a low-resolution image frame, including:

[0121] If the value of the Type field of the NAL packet of an image frame in the first bitstream is a first extended value, then the second device processes the image frame to obtain a reference high-resolution image frame.

[0122] If the Type field value of the NAL packet of the image frame in the first bitstream is a non-extended value, then the second device processes the image frame to obtain a low-resolution image frame.

[0123] In the third possible implementation of the sixth aspect, the second device processes the received bitstream to obtain a reference high-resolution image frame and a low-resolution image frame, including:

[0124] If the received bitstream includes a second bitstream and a third bitstream, the second device processes the second bitstream and the third bitstream respectively to obtain a reference high-resolution image frame corresponding to the second bitstream and a low-resolution image frame corresponding to the third bitstream.

[0125] In conjunction with the fourth possible implementation of the sixth aspect, the second device uses the reference high-resolution image after decoding the reference high-resolution image frame as a reference to reconstruct the low-resolution image after decoding the low-resolution image frame, including:

[0126] The second device decodes the reference high-resolution image frame and the low-resolution image frame respectively to obtain the reference high-resolution image and the low-resolution image;

[0127] The second device inputs the reference high-resolution image, the image in the low-resolution image corresponding to the reference high-resolution image, and the low-resolution image into a neural network to perform super-resolution reconstruction on the low-resolution image; wherein, the image in the low-resolution image corresponding to the reference high-resolution image is the same image as the reference high-resolution image.

[0128] In conjunction with the fourth possible implementation of the sixth aspect, in the fifth possible implementation, the second device inputs the reference high-resolution image, the image in the low-resolution image corresponding to the reference high-resolution image, and the low-resolution image into a neural network to perform super-resolution reconstruction on the low-resolution image, including:

[0129] The low-resolution image is convolved to obtain a first feature map, and the image in the low-resolution image that corresponds to the reference high-resolution image is convolved to obtain a second feature map.

[0130] The first feature map and the second feature map are concatenated to obtain a first concatenated feature map;

[0131] Convolution is performed on the first spliced ​​feature map to obtain the third feature map;

[0132] The reference high-resolution image is convolved to obtain the fourth feature map;

[0133] The third feature map and the fourth feature map are concatenated to obtain the second concatenated feature map;

[0134] The second stitched feature map is convolved to obtain the reconstructed image of the low-resolution image.

[0135] The image processing method according to the embodiments of this application uses a high-resolution image of the current scene as a reference to train and infer a super-resolution neural network. A simple network structure is employed to achieve high-quality super-resolution reconstruction, further reducing computation time while improving the quality of the reconstructed image. Furthermore, a small number of reference image pairs allows for asynchronous parallel training and inference. The network parameters generated during online training can be updated in real time to guide the reconstruction process of subsequent frames, ensuring the quality of the reconstructed image.

[0136] In a sixth possible implementation of the sixth aspect, the second device uses the reference high-resolution image decoded from the reference high-resolution image frame as a reference to reconstruct the low-resolution image decoded from the low-resolution image frame, including:

[0137] The second device decodes the reference high-resolution image frame and the low-resolution image frame respectively to obtain the reference high-resolution image and the low-resolution image;

[0138] The second device inputs the reference high-resolution image and the image in the low-resolution image that corresponds to the reference high-resolution image into the first neural network for training to obtain network parameters; wherein, the image in the low-resolution image that corresponds to the reference high-resolution image is the same image as the reference high-resolution image;

[0139] The second device inputs the low-resolution image into the second neural network for processing, and performs super-resolution reconstruction on the low-resolution image;

[0140] The network parameters of the second neural network are obtained by real-time synchronization based on the network parameters of the first neural network.

[0141] The super-resolution reconstruction process described in the above embodiments of this application can use a high-resolution image of the current scene as a reference for training and inference of the super-resolution neural network. Therefore, a simple network structure can achieve high-quality super-resolution reconstruction, improving the quality of the reconstructed image while reducing computation time. Furthermore, a small number of reference image pairs allows for asynchronous parallel training and inference. The network parameters generated during online training can be updated in real time to guide the reconstruction process of subsequent frames, ensuring the quality of the reconstructed image.

[0142] In a seventh aspect, embodiments of this application provide a terminal device that can execute one or more of the image processing methods described in the first aspect or various implementations of the first aspect.

[0143] Eighthly, embodiments of this application provide a terminal device that can execute one or more of the image processing methods described in the second aspect or various implementations of the second aspect above.

[0144] Ninthly, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in an electronic device, a processor in the electronic device executes one or more of the image processing methods described in the first aspect or multiple implementations of the first aspect.

[0145] In a tenth aspect, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in an electronic device, a processor in the electronic device executes one or more of the image processing methods described in the second aspect or multiple implementations of the second aspect.

[0146] Eleventhly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement one or more of the image processing methods of the first aspect or multiple implementations of the first aspect.

[0147] In a twelfth aspect, embodiments of this application provide a non-volatile computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement one or more of the image processing methods of the second aspect or multiple implementations of the second aspect. Attached Figure Description

[0148] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0149] Figure 1 A schematic diagram of the structure of the NAL unit in the related technology is shown.

[0150] Figure 2 This section illustrates an example of an application scenario for video capture and transmission, representing one implementation of a related technology.

[0151] Figure 3 A schematic diagram illustrating an application scenario of an image processing method according to an embodiment of this application is shown.

[0152] Figure 4 A flowchart illustrating an image processing method according to an embodiment of this application is shown.

[0153] Figure 5 The diagram illustrates an end-to-end module and flowchart according to one embodiment of this application.

[0154] Figure 6 A flowchart illustrating an image processing method according to an embodiment of this application is shown.

[0155] Figure 7 A schematic diagram illustrating a frame-drawing process according to an embodiment of this application is shown.

[0156] Figure 8 A flowchart illustrating an image processing method according to an embodiment of this application is shown.

[0157] Figure 9 A schematic diagram illustrating a frame-drawing process according to an embodiment of this application is shown.

[0158] Figure 10 An example is shown showing the correspondence between a first resolution image frame and a second resolution image frame according to an embodiment of this application.

[0159] Figure 11a A schematic diagram of a package according to an embodiment of this application is shown.

[0160] Figure 11b A schematic diagram of a package according to an embodiment of this application is shown.

[0161] Figure 11c An example of a package according to an embodiment of this application is shown.

[0162] Figure 11d A schematic diagram illustrating the frame extraction and packet assembly process according to an embodiment of this application is shown.

[0163] Figure 12A flowchart of a super-resolution reconstruction method according to an embodiment of this application is shown.

[0164] Figure 13 A flowchart of a reconstruction method according to an embodiment of this application is shown.

[0165] Figure 14 A schematic diagram illustrating the reconstruction process according to an embodiment of this application is shown.

[0166] Figure 15 A flowchart of a super-resolution reconstruction method according to another embodiment of this application is shown.

[0167] Figure 16 A schematic diagram of a super-resolution reconstruction method according to an embodiment of this application is shown.

[0168] Figure 17 A schematic diagram of a second device performing super-resolution reconstruction according to an embodiment of this application is shown.

[0169] Figure 18 and Figure 19 Schematic diagrams showing the effects of the image processing method of this application compared to related technologies are provided.

[0170] Figure 20 A block diagram of an image processing apparatus according to an embodiment of this application is shown.

[0171] Figure 21 A block diagram of an image processing apparatus according to an embodiment of this application is shown.

[0172] Figure 22 A block diagram of a first device according to an embodiment of this application is shown.

[0173] Figure 23 A block diagram of a second device according to an embodiment of this application is shown. Detailed Implementation

[0174] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0175] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0176] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0177] Figure 2 This illustrates an example of an application scenario for video capture and transmission, representing one implementation of a related technology. For example... Figure 2 As shown, the video acquisition system in this scenario comprises video capture equipment, a 5G base station, video processing equipment, and display equipment. The video capture equipment can be a network camera (IP CAMERA, or IPC), the video processing equipment can be a video server, and the display equipment can be a large monitoring screen or a mobile phone or other terminal device.

[0178] IPC can encode the acquired raw images into image frames using a video encoder, and then transmit the image frames to a 5G base station via a 5G wireless network. The 5G base station then transmits the image frames to a video server, which can decode and reconstruct the received image frames to obtain the image. The reconstructed image is then sent to a large monitoring screen for display.

[0179] IPC transmits video data directly through a 5G wireless network. Compared to wired networks, it does not require cabling and has advantages such as flexible installation, removal, and modification.

[0180] An IPC encoder can encode and generate video streams in one or more encoding formats, such as two or more. For example, in dual-stream technology, the IPC encoder can digitize the original image to generate two video streams, called the main stream and the sub-stream. The main stream can include high-resolution image frames, while the sub-stream can include low-resolution image frames. The encoding formats of the main stream and the sub-stream can be the same or different. In one possible implementation, the IPC encoder can use the same encoding format and the same digitization algorithm to digitize the original image. The IPC encoder can control the image sharpness and network bandwidth usage in the main stream and sub-stream by setting different configuration parameters. The image sharpness and network bandwidth usage are mainly determined by three configuration parameters: resolution, bitrate, and frame rate.

[0181] Dual-stream technology is designed to resolve the conflict between the image quality requirements of local storage in IPCs and the bandwidth requirements of remote network transmission. Dual-stream technology can meet two different bandwidth requirements for local and remote transmission: local transmission can use a high-resolution main stream, resulting in clearer stored recordings locally; due to bandwidth limitations, remote transmission can use a low-resolution sub-stream, thus obtaining smoother images for easier previewing.

[0182] Users often desire a high-quality, clear video experience. While increasing video resolution, frame rate, and bitrate can improve clarity, video transmission requires more bandwidth. Considering cost and other factors, network uplink bandwidth resources are limited, and these resources are particularly valuable in wireless air interface uplink scenarios.

[0183] A single 5G base station has limited uplink bandwidth, while ultra-high-definition video streaming demands extremely high network bandwidth. Take video streaming in a 4K IPC monitoring scenario as an example:

[0184] 1. The main stream resolution is 4K, the frame rate is 25Hz, and the bitrate is configured to be 12Mbps.

[0185] 2. The sub-stream resolution is 1080P, the frame rate is 25Hz, and the bitrate is configured to be 4Mbps.

[0186] Because encoded I-frames are large and encoded P-frames are small, video streams experience peak bandwidth fluctuations. Therefore, to achieve a stable, smooth, high-quality video experience, a bandwidth of approximately 20Mbps is typically required for a single video stream. If a single 5G base station has an uplink bandwidth of around 200Mbps, in practical engineering deployments, fewer than 10 ultra-high-definition video streams can be deployed within a single 5G base station cell, and all uplink bandwidth is occupied, resulting in low engineering efficiency. Configuring a lower bitrate and sacrificing image quality allows for the deployment of more IPCs, but the lower image quality is detrimental to the security requirements of the surveillance industry. Furthermore, it affects the accuracy of intelligent analysis services, such as facial recognition.

[0187] Therefore, transmitting higher-quality video under limited bandwidth has become a common industry demand, and related technologies have been developed to meet this need. Currently, the relevant technologies that meet these needs include super-resolution reconstruction methods or systems. A system can include two parts: a front-end and a back-end. The front-end can be a network camera or a terminal device with a camera, etc. The front-end is responsible for video acquisition, encoding and compression to generate a video stream, and sending the video stream to the wireless network for transmission to the back-end. The back-end is responsible for receiving the video stream through the wireless network and performing reconstruction and other processing on the video stream.

[0188] In related technologies, the backend uses a super-resolution reconstruction neural network to perform super-resolution reconstruction on the received video stream. Before using the super-resolution reconstruction neural network, a large number of repeated training sessions are required for the complex deep neural network, which is time-consuming. Furthermore, the inference process during reconstruction is also very time-consuming. Therefore, the training and reconstruction processes in related technologies need to be performed offline and non-real-time. To meet the requirements of real-time processing, the deep neural network needs to be simplified. However, the image quality obtained by super-resolution reconstruction using the simplified deep neural network may not meet the requirements.

[0189] Therefore, the technical problem to be solved by this application is: how to enable more ultra-high-definition IPCs to be accessed in the same base station cell under the premise of limited uplink bandwidth resources and ensuring the quality of back-transmitted video, and how to enable the backend to realize real-time reconstruction of video.

[0190] To address the aforementioned technical problems, this application provides an image processing method. Figure 3 A schematic diagram illustrating an application scenario of an image processing method according to an embodiment of this application is shown. For example... Figure 3 As shown, the image processing method of the embodiments of this application can include a first device and a second device in its application scenario. The first device can be a device capable of video and image acquisition, and encoding, compressing, or otherwise processing the acquired images or videos. The first device can be a camera, or a terminal device with a camera, such as a mobile phone, drone, or autonomous vehicle. The first device may not have image or video acquisition capabilities, but can encode and compress images. That is, the first device can be connected to a video acquisition device to obtain images or videos acquired by the video acquisition device. The second device can be a video server, an edge computing node in a network, or the like. Both the first and second devices can include modules or units with image processing capabilities. For example, the first device can include a general-purpose processor, an FPGA (Field Programmable Gate Array), or the like, while the second device can include a graphics processing unit (GPU) or a GPU cluster.

[0191] The first device can send a video stream to the second device via a wired or wireless network. In one example, the first device can send the video stream to the second device via a 5G wireless network. For instance, the first device can first send the video stream to a 5G base station, and the 5G base station will then send the received video stream to the second device.

[0192] Figure 4 A flowchart illustrating an image processing method according to an embodiment of this application is shown. Figure 3 and Figure 4 The image processing method of this application is described.

[0193] like Figure 3 and Figure 4 As shown, in one embodiment of this application, the image processing method may include the following steps:

[0194] In step S400, the first device encodes the acquired image to generate a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of the first resolution image frames is less than the number of the second resolution image frames.

[0195] In step S401, the first device packages the first resolution image frame and the second resolution image frame into a video stream and sends the video stream to the second device.

[0196] In one embodiment of this application, the image processing method may further include the following steps:

[0197] In step S500, the second device processes the received video stream to obtain a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of the first resolution image frames is less than the number of the second resolution image frames.

[0198] In step S501, the second device uses the first resolution image after decoding the first resolution image frame as a reference to reconstruct the second resolution image after decoding the second resolution image frame.

[0199] The statement that the resolution of a first resolution image frame is greater than the resolution of a second resolution image frame can mean that the resolution of the first resolution image frame is higher than that of the second resolution image frame. For example, if the resolution of the first resolution image frame is 4K and the resolution of the second resolution image frame is 1080P, then 4K resolution is greater than 1080P resolution. In one embodiment of this application, the length and width of the high-resolution image frame can be twice the length and width of the low-resolution image frame, respectively. For example, the high-resolution image frame can be 3140x2160, and the low-resolution image frame can be 1920x1080.

[0200] The number of first-resolution image frames being less than the number of second-resolution image frames can mean that the frame rate of the first-resolution image frames is less than the frame rate of the second-resolution image frames. For example, if the frame rate of the first-resolution image frames is 10 and the frame rate of the second-resolution image frames is 30, then frame rate 10 is less than frame rate 30.

[0201] The first device can capture video or images. It can generate a first-resolution image frame with a low frame rate and high resolution, and a second-resolution image frame with a high frame rate and low resolution, by performing high-resolution and low-resolution encoding on the captured images and then extracting frames. The frame extraction process can occur before or after encoding; the embodiments of this application do not limit this. If the frame extraction process occurs after encoding, the first device can first perform high-resolution and low-resolution encoding on the captured images to generate high-resolution and low-resolution image frames respectively. Then, the first device can extract a portion of the image frames from the high-resolution image frames as the first-resolution image frames, and extract all or part of the image frames from the low-resolution image frames as the second-resolution image frames. The number of extracted first-resolution image frames is less than the number of extracted second-resolution image frames.

[0202] For step S401, the first device can either package two image frames of different resolutions into one video stream, or package two image frames of different resolutions into two separate video streams.

[0203] In one possible implementation, the first device can sequentially package first-resolution image frames and second-resolution image frames into a video stream for transmission, or sequentially package first-resolution image frames into a video stream for transmission and sequentially package second-resolution image frames into another video stream for transmission.

[0204] In another possible implementation, the image frames obtained by the first device after encoding the acquired images can be I-frames, P-frames, or B-frames. I-frames, P-frames, and B-frames can be composed of multiple units; for example, an image frame can be composed of multiple NAL units. The first device can insert multiple units of a first-resolution image frame into multiple units of a second-resolution image frame to form a video stream, which is then sent to the second device.

[0205] The second device receives a video stream and can unpack the video stream to obtain first-resolution image frames and second-resolution image frames. It can also further decode the first-resolution image frames to obtain a first-resolution image and the second-resolution image frames to obtain a second-resolution image. The second device can input the first-resolution image corresponding to the first-resolution image frame and the second-resolution image corresponding to the second-resolution image frame into a super-resolution reconstruction neural network. Using the first-resolution image as a reference, the super-resolution reconstruction neural network is trained and inferred to achieve super-resolution reconstruction of the second-resolution image, thus obtaining the corresponding high-resolution image. The super-resolution reconstruction neural network can be a neural network trained with the aim of super-resolution reconstruction of low-resolution images to obtain high-resolution images. Forward inference using the super-resolution reconstruction neural network can achieve super-resolution reconstruction of the input low-resolution image to obtain the corresponding high-resolution image. The super-resolution reconstruction neural network can be a convolutional neural network or a fully connected neural network; this application does not limit the specific type of the super-resolution reconstruction neural network.

[0206] It should be noted that the second device may also use other super-resolution reconstruction methods to perform super-resolution reconstruction of the second resolution image with the first resolution image as a reference. For example, the second device may use interpolation to perform super-resolution reconstruction, and this application does not limit this.

[0207] Compared to image processing methods in related technologies, the image processing method of this application has a lower frame rate for high-resolution image frames obtained by encoding the acquired images by the first device, and most of the encoded image frames are low-resolution image frames. In other words, the number of high-resolution image frames is relatively small. Therefore, the amount of data sent by the first device to the second device is smaller, and the bandwidth occupied by the video stream is narrower. Thus, with the same bandwidth, more video streams can be accessed.

[0208] Furthermore, since fewer high-resolution images are used as references for super-resolution reconstruction of low-resolution images, the image processing method of the embodiments of this application can achieve image super-resolution reconstruction faster, reduce computation time, and achieve real-time reconstruction of video.

[0209] In related technologies, a large number of images are pre-collected to train a super-resolution reconstruction neural network. However, since the actual application scenario is often not necessarily related to the pre-collected scenario, the super-resolution reconstruction neural network structure is relatively complex, the training and inference process is time-consuming, and the image reconstruction quality cannot be guaranteed in order to improve the accuracy of super-resolution reconstruction. In contrast, the image processing method according to the embodiments of this application uses a high-resolution image of the current scene as a reference to perform super-resolution reconstruction on a low-resolution image. Therefore, the scene correlation is relatively strong, which can achieve high-quality super-resolution reconstruction and further shorten the computation time while improving the quality of the reconstructed image.

[0210] As can be seen from the above description of the image processing method of this application, the image processing method of this application can be divided into two parts: front-end and back-end. Figure 5 An end-to-end module and flowchart according to an embodiment of this application is shown. Figure 5 As shown, the first device at the front end mainly implements the process of acquiring images and videos through a photosensor, encoding the acquired images, extracting frames, and assembling them into a video stream for transmission to the second device. The encoding can be divided into high-resolution encoding and low-resolution encoding. The second device at the back end mainly implements the process of unpacking, decoding, and super-resolution reconstruction of the received video stream. Super-resolution reconstruction can be achieved through training and inference of a convolutional neural network. This application provides several different implementation methods for both the front end and the back end.

[0211] First Equipment

[0212] Frame skipping

[0213] Figure 6 A flowchart illustrating an image processing method according to an embodiment of this application is shown. Figure 6 As shown, in step S400, the first device encodes the acquired image to generate a first resolution image frame and a second resolution image frame, which may include:

[0214] In step S4000, the first device encodes the acquired image to generate a high-resolution image frame and a low-resolution image frame; wherein, the low-resolution image frame is a second-resolution frame.

[0215] Step S4001: The first device extracts a portion of the frames from the high-resolution image frames as the first resolution image frames.

[0216] Figure 7 A schematic diagram illustrating a frame-drawing process according to an embodiment of this application is shown. Figure 5 and Figure 7As shown, the first device acquires images through a photosensor, inputs the images to a low-resolution encoder for encoding and outputting low-resolution image frames, and inputs the images to a high-resolution encoder for encoding and outputting high-resolution image frames. The encoder used in this application can be an H.264 / AVC encoder, an H.265 / HEVC encoder, an H.266 / VVC encoder, etc., and this application does not specifically limit the type of encoder.

[0217] It should be noted that the terms "high-resolution image frame" and "low-resolution image frame" are relative; the resolution of a high-resolution image frame is higher than that of a low-resolution image frame.

[0218] The first device can extract frames from the encoded image frames, such as... Figure 7 As shown, in this embodiment, the first device can extract a portion of image frames from high-resolution image frames as first-resolution image frames. The first device may not extract low-resolution image frames, meaning it can use all low-resolution image frames as second-resolution image frames. It should be noted that the first device can also extract a portion of image frames from low-resolution image frames as second-resolution image frames, with the extraction frequency from low-resolution image frames being higher than the extraction frequency from high-resolution image frames. In other words, the number of second-resolution image frames extracted from low-resolution image frames is greater than the number of first-resolution image frames extracted from high-resolution image frames.

[0219] Regarding the specific frame extraction method, taking the extraction of high-resolution image frames as an example, one frame can be extracted every N frames, where N is an integer greater than or equal to 2. Alternatively, multiple frames can be extracted every N frames. Irregular frame extraction can also be used; this application does not limit this approach.

[0220] As mentioned earlier, the encoded image frames can be I-frames, P-frames, or B-frames. Figure 7 For example, this application does not limit the type of high-resolution image frames extracted. The first device can extract any one or more types of I-frames, P-frames, or B-frames from the high-resolution image frames. In one example, the first device can extract I-frames from the high-resolution image frames and package the extracted I-frames and low-resolution image frames into a video stream. In another example, the first device can extract P-frames or B-frames from the high-resolution image frames. Since P-frames and B-frames require a reference frame for decoding, a long-term reference frame (I-frame) also needs to be extracted. The first device packages the extracted P-frames or B-frames, the reference frame, and the low-resolution image frames into a video stream. Because P-frames and B-frames have higher compression rates and smaller data sizes, extracting P-frames or B-frames can reduce the size of transmitted data and reduce peak traffic fluctuations, helping to obtain a stable bitstream, improve bandwidth utilization, and enable more video streams to be accessed with the same bandwidth.

[0221] The frame extraction process described in the above embodiments of this application can reuse the high-resolution bitstream and low-resolution bitstream already generated by the camera. By extracting frames from the already generated high-resolution bitstream and low-resolution bitstream, a first resolution image frame and a second resolution image frame can be obtained. In other words, the image frames of the main bitstream and sub-bitstream in the dual bitstream technology can be reused without secondary encoding, without additional computing power, and without hardware dependence.

[0222] Figure 8 A flowchart illustrating an image processing method according to an embodiment of this application is shown. Figure 8 As shown, in step S400, the first device encodes the acquired image to generate a first resolution image frame and a second resolution image frame, which may include:

[0223] In step S4003, the first device extracts all or part of the images from the acquired images and encodes them to generate a second resolution image frame, and extracts part of the images from the acquired images and encodes them to generate a first resolution image frame.

[0224] Figure 9 This diagram illustrates a frame extraction process according to an embodiment of the present application. A first device acquires an image using a photosensor. The first device extracts a portion of the acquired image and inputs it into a high-resolution encoder for encoding to generate a first-resolution image frame. The first device also extracts all or part of the acquired image and inputs it into a low-resolution encoder for encoding to generate a second-resolution image frame. The first-resolution image frames generated by the first device are fewer than the second-resolution image frames; that is, the first device generates fewer high-resolution image frames than low-resolution image frames.

[0225] The generated high-resolution image frames and low-resolution image frames (or, first-resolution image frames and second-resolution image frames) may or may not have a corresponding relationship; this application does not limit this. For example, in one example, the first device starts extracting from the first frame and can extract one frame of the acquired image every three frames to perform high-resolution encoding to generate a first-resolution image frame, that is, extracting the acquired frames 1, 5, 9, 13, 17... to perform high-resolution encoding to generate a first-resolution image frame; the first device also starts extracting from the first frame and extracts one frame of the acquired image every other frame to perform low-resolution encoding to generate a second-resolution image frame, that is, extracting the acquired frames 1, 3, 5, 7, 9, 11, 13, 15... to perform low-resolution encoding to generate a second-resolution image frame. In this case, in the second-resolution image frames, every other frame contains a second-resolution image frame and a corresponding first-resolution image frame that are encoded from the same image. For example, frames 1, 5, 9, 13... of the acquired image simultaneously contain corresponding first-resolution image frames and second-resolution image frames. In another example, the first device starts extracting from the first frame and can extract one captured image frame every 3 frames to generate a first resolution image frame through high-resolution encoding; the first device starts extracting from the second frame and extracts one captured image frame every 1 frame to generate a second resolution image frame through low-resolution encoding; in this example, there is no image frame in the second resolution image frame that corresponds to the first resolution image frame.

[0226] The frame extraction method described in the above embodiments of this application performs frame extraction on the acquired image before encoding, thus reducing the amount of image data that needs to be encoded and improving encoding efficiency. The above are some embodiments of frame extraction and encoding provided in this application; however, the type, frequency, and order of frame extraction and encoding do not limit this application in any way.

[0227] Package

[0228] In the embodiments of this application, the first device provides multiple packet assembly methods, which can generate one video stream or two video streams after packet assembly. The first device can generate one video stream by inserting a first-resolution image frame into the video stream containing a second-resolution image frame, or it can generate two video streams by separately packetizing the first-resolution image frame and the second-resolution image frame.

[0229] In one possible implementation, step S401, where the first device packages the first resolution image frame and the second resolution image frame into a video stream, may include:

[0230] The first device packages the first resolution image frame and the second resolution image frame into a first video stream;

[0231] And / or,

[0232] The first device packages the first resolution image frames into a second video stream, and packages the second resolution image frames into a third video stream.

[0233] Package Assembly Example 1

[0234] In one possible implementation, the second resolution image frame contains a source image frame corresponding to the first resolution image frame, and the first resolution image frame and the corresponding source image frame are generated by encoding the same image.

[0235] for Figure 6 and Figure 7 The frame extraction process shown involves the first device simultaneously inputting the acquired image into both a low-resolution encoder and a high-resolution encoder to generate low-resolution and high-resolution image frames. Therefore, the low-resolution and high-resolution image frames generated by the first device are paired. Since the low-resolution image frame is a second-resolution image frame, the first device extracts frames from the high-resolution image frame to obtain a first-resolution image frame. Therefore, the second-resolution image frame contains an image frame corresponding to the first-resolution image frame, which is the aforementioned homologous image frame.

[0236] for Figure 6 and Figure 7 The frame extraction process shown involves extracting a portion of image frames from low-resolution image frames to create second-resolution image frames, and extracting a portion of image frames from high-resolution image frames to create first-resolution image frames. Based on the extraction rules, it can be determined whether there exists an image frame in the second-resolution image frames that corresponds to the first-resolution image frame. For example, images a, b, c, d, e… after high-resolution encoding, can result in a high-resolution image frame a. H b H c H d H and e H ..., the low-resolution image frame obtained after low-resolution encoding can be a L b L c L d L e L ...If a portion of image frames is extracted from the low-resolution image frames to obtain a second-resolution image frame, then a... L c L e L ...(Starting from the first frame, extract one frame every other frame), extract a portion of image frames from the high-resolution image frames as the first-resolution image frames to obtain a. H e H...(Starting from the first frame, one frame is extracted every three frames). This method of frame extraction results in a second-resolution image frame containing an image frame corresponding to the first-resolution image frame. If a portion of the image frames are extracted from the low-resolution image frames to obtain the second-resolution image frame, then b... L d L ...(Starting from the second frame, extract one frame every other frame), extract a portion of image frames from the high-resolution image frames as the first-resolution image frames to obtain a. H e H ...(Starting from the first frame, one frame is extracted every three frames). The second resolution image frame obtained by this frame extraction method does not contain an image frame corresponding to the first resolution image frame.

[0237] Similarly, for Figure 8 and Figure 9 As can be seen from the above description, depending on the frame extraction method, the second resolution image frame may contain a source image frame corresponding to the first resolution image frame, or it may not contain a source image frame corresponding to the first resolution image frame.

[0238] In other words, if an image frame encoded at high resolution is extracted as a first-resolution image frame and an image frame encoded at low resolution is extracted as a second-resolution image frame, the second-resolution image frame is a source image frame of the first-resolution image frame.

[0239] Figure 10 An example illustrating the correspondence between a first-resolution image frame and a second-resolution image frame according to an embodiment of this application is shown. For example... Figure 10 As shown, the images acquired by the first device are a, b, c, d, e... in sequence. After low-resolution encoding, the second-resolution image frame obtained can be a. L b L c L d L e L ...The first resolution image frame obtained after high-resolution encoding and frame extraction can be b H and d H Frame extraction can be performed before or after high-resolution encoding; this application does not impose any limitation on this. Wherein, b H and b L Correspondingly, both are generated by encoding image b; d H and d L Correspondingly, both are generated by encoding the image d.

[0240] In one possible implementation, where a source image frame exists within a second-resolution image frame that corresponds to the first-resolution image frame, the first-resolution image frame and its corresponding source image frame are arranged according to a rule in the first video stream. This rule-based arrangement could mean that the first-resolution image frame and its corresponding source image frame are adjacent in the first video stream. For example, the first-resolution image frame might be located after or before its corresponding source image frame in the first video stream.

[0241] It should be noted that the above is merely an example of a rule-based arrangement, and this application is not limited to this. For instance, in the first video stream, the first resolution image frame can also be arranged with its corresponding source image frame at intervals of N frames, where N is an integer greater than or equal to 2, etc., as long as the first device uses the same rules when assembling the packets and the second device uses the same rules when unpacking them. The above packet assembly method can maintain a consistent correspondence between high-resolution and low-resolution image frames obtained by the first and second devices. In this way, the second device can directly use the paired high-resolution and low-resolution image frames as a reference for reconstructing the low-resolution image, simplifying the processing of the second device.

[0242] Figure 11a This diagram illustrates a packet assembly process according to an embodiment of the present application. The packet assembly process will be described using the example of a first-resolution image frame following a corresponding source image frame. Figure 11a As shown, L represents a low-resolution image frame, which is the second-resolution image frame, and H represents a high-resolution image frame, which is the first-resolution image frame. During packet assembly, the first device finds the corresponding source image frame of the first-resolution image frame in the sequence of second-resolution image frames, and inserts the first-resolution image frame after the corresponding source image frame to assemble the first video stream.

[0243] In one possible implementation, in the first video stream, the value of the Type field in the NAL packet of the first resolution image frame is different from the value of the Type field in the NAL packet of the second resolution image frame. For example, the Type field of the NAL packet of the first resolution image frame can use a first extended value to represent the frame type, and this first extended value is different from the existing value of the Type field. The value of the Type field of the NAL packet of the second resolution image frame can still use an existing value to represent the frame type; in other words, the value of the Type field of the NAL packet of the second resolution image frame is a non-extended value, which can refer to the value of the Type field specified by the encoding format standard.

[0244] For example, the NAL header consists of one byte (8 bits), of which 5 bits are the Type field, which can represent values ​​from 0 to 31. Different values ​​can represent different meanings. Among them, 13-23 are reserved values ​​of the Type field, which can be expanded to represent specific meanings. Other values ​​can be the non-expanded values ​​as described above. When expanding, the value of the Type field can be set to any value from 13 to 23.

[0245] In one possible implementation, the first device can modify the value of the Type field of the NAL packet of the first resolution image frame or the second resolution image frame during frame extraction or packet assembly. For example, in the packet assembly process described above, the first device can modify the value of the Type field of the NAL packet of the first resolution image frame to a first extended value. It should be noted that the above method of modifying the value of the Type field of the NAL packet is also applicable to other embodiments of this application.

[0246] By using the extended value of the Type field to distinguish between first-resolution image frames and second-resolution image frames, the second device can differentiate between different types of image frames during decoding based on whether the value of the Type field in the NAL packet of the image frame is an extended value.

[0247] It should be noted that the "first" in the first extended value mentioned above, the "second" in the second extended value below, and the "first" and "second" in other terms are used to distinguish different objects and are not used to describe a specific order.

[0248] In another possible implementation, in the first video stream, the first resolution image frame and the corresponding source image frame are combined into a composite frame. Here, a composite frame can refer to two image frames combined to form one image frame. This can be achieved by combining smaller units that make up the image frame. Taking an image frame composed of NAL packets as an example, in the embodiments of this application, step S401, where the first device packages the first resolution image frame and the second resolution image frame into a video stream, may include: the first device inserting the Network Abstraction Layer (NAL) packet of the first resolution image frame into the NAL packet of the corresponding source image frame to form the composite frame. Specifically, the first device may insert the NAL packet of the first resolution image frame into the main body of the NAL packet of the corresponding second resolution image frame to form the composite frame.

[0249] In one possible implementation, the first device may further modify the value of the Type field of the NAL packet of the same source image frame corresponding to the first resolution image frame to the second extended value during frame extraction or packet assembly. For example, in the packet assembly process of the above embodiment, the first device may modify the value of the Type field of the NAL packet of the corresponding second resolution image frame to the second extended value after inserting the NAL packet of the first resolution image frame into the main body of the NAL packet of the corresponding second resolution image frame.

[0250] Figure 11b A schematic diagram of a package according to an embodiment of this application is shown. Figure 11b As shown, HL represents a combined frame composed of a first-resolution image frame and a corresponding second-resolution image frame, and L represents a low-resolution image frame, which is the second-resolution image frame.

[0251] In one possible implementation, in the combined frame, the value of the Type field in the NAL packet of the first resolution image frame is different from the value of the Type field in the NAL packet of the second resolution image frame. For example, the Type field of the NAL packet of the second resolution image frame can use a second extended value to represent the frame type, and this second extended value is different from the existing Type field value. The Type field of the NAL packet of the first resolution image frame can still use the existing value to represent the frame type; that is, the Type field value of the NAL packet of the first resolution image frame is a non-extended value. In other words, the Type field value of the NAL packet of the low-resolution image frame can use an extended value to distinguish it from the existing image frame. In this way, the Type field value of the NAL packet of the combined frame is different from the existing Type field value. When the second device receives the video stream and performs unpacking processing, it can identify the combined frame through the Type field value and perform unpacking processing on the combined frame.

[0252] According to the video stream formed by the above-described embodiments of this application, when the second device receives the video stream sent by the first device and unpacks the video stream, it can identify different image frames based on the value of the Type field of the NAL packet of the image frame. For example, if the value of the Type field of the NAL packet of the image frame is identified as a first extended value, the second device can unpack the image frame to obtain a first resolution image frame; if the value of the Type field of the NAL packet of the image frame is identified as a non-extended value, the second device can unpack the image frame to obtain a second resolution image frame; if the value of the Type field of the NAL packet of the image frame is identified as a second extended value, the second device can unpack the image frame to obtain a first resolution image frame and a second resolution image frame corresponding to the first resolution image frame.

[0253] The above packetization method can maintain a consistent correspondence between the high-resolution and low-resolution image frames acquired by the first device and the video processor. In this way, the second device can directly use the pairs of high-resolution and low-resolution image frames as a reference for low-resolution image reconstruction, simplifying the processing of the second device.

[0254] Package Example 2

[0255] In one possible implementation, the second resolution image frame does not contain any source image frames corresponding to the first resolution image frame. In this case, either one video stream or two video streams can be used for transmission.

[0256] When transmitting a video stream, the first device can arbitrarily insert first resolution image frames into the queue of second resolution image frames to form a first video stream. Similarly, in the first video, the value of the Type field of the NAL packet of the first resolution image frame is different from the value of the Type field of the NAL packet of the second resolution image frame. The Type field of the NAL packet of the first resolution image frame can use a first extended value to represent the frame type.

[0257] In one possible implementation, the first-resolution image frame can be inserted into the queue of the second-resolution image frames according to the acquisition time sequence of the images corresponding to the first-resolution image frame and the second-resolution image frame. For example, if the acquisition time of image a by the first device is before that of image b, then encoding image a yields the first-resolution image frame a. H After encoding image b, a second-resolution image frame b is obtained. L During packet assembly, the first resolution image frame a can be... H Inserted into second resolution image frame b L Previously, this meant that when using neural networks for inference and training to reconstruct high-resolution image frames from the second resolution image frame, the first resolution image frame, which was closest in time to the second resolution image frame, could be referenced, improving the accuracy of the reconstruction and thus enhancing video quality.

[0258] When using two video streams for transmission, as described above, the first device can package the first resolution image frames into a second video stream and package the second resolution image frames into a third video stream.

[0259] Figure 11c An example of a package according to an embodiment of this application is shown. For example... Figure 11c As shown, when the data is packetized into two video streams for transmission, in the second video stream corresponding to the first resolution image frame, the header of the NAL packet of the first resolution image frame does not need to include an extended field to identify the first resolution image frame.

[0260] In one possible implementation, when two video streams are transmitted, the Type field values ​​of the NAL packets for both the first and second resolution image frames can use the existing Type field values ​​instead of extended values. The second device can then distinguish between the two different video streams using other parameters of the image frames.

[0261] The above are examples of several packet-gathering methods provided in this application. The packet-gathering implementation methods in the above examples do not limit this application in any way. Except for the frame combination method, other packet-gathering implementation methods are compatible with existing encoders.

[0262] Figure 11d A schematic diagram illustrating the frame extraction and packet assembly process according to an embodiment of this application is shown. Figure 11d As shown, the first device encodes the acquired images to generate two video streams with different resolutions: a high-resolution main stream and a low-resolution sub-stream. The high-resolution main stream includes high-resolution image frames, and the low-resolution sub-stream includes low-resolution image frames. The main stream's GOP can be 50 frames, with a bitrate of 12 Mbps and a resolution of 4K; the sub-stream's GOP can be 50 frames, with a bitrate of 4 Mbps and a resolution of 1080P. It should be noted that the GOPs of the main stream and sub-stream are not limited to the above example, and the GOPs of the main stream and sub-stream can be equal or unequal; the embodiments of this application do not limit this.

[0263] The first device can extract a reference high-resolution image frame from the high-resolution image frame, and then combine the reference high-resolution image frame with the low-resolution image frame to generate a first bitstream. The first device can extract any portion of the high-resolution image frame as the reference high-resolution image frame, without limiting the type of the extracted image frame, such as... Figure 11d As shown, I-frames can be extracted and grouped with low-resolution image frames, or P-frames or B-frames can be extracted. This does not affect the result of super-resolution reconstruction, as long as the extracted image frames can be decoded normally after being sent to the second device.

[0264] like Figure 11dAs shown, taking the extracted high-resolution image frame as an I-frame as an example, the packet assembly process is explained. The low-resolution I-frames in the sub-stream correspond to the extracted high-resolution I-frames, and are obtained by performing low-resolution encoding and high-resolution encoding on the same acquired image. Therefore, in one example, the first device can insert a reference high-resolution image frame into the adjacent position of the corresponding low-resolution image frame. For example, the high-resolution I-frame can be inserted after (or before) the low-resolution I-frame and packetized with the sub-stream to form a first bitstream. The GOP of the first bitstream can be 50 frames, and the bitrate can be 6 Mbps. In another example, the first device can insert the NAL packet of the high-resolution I-frame into the main body of the NAL packet of the low-resolution I-frame to form a combined frame (combined I-frame). The combined frame is then packetized with the low-resolution P-frame in the sub-stream to form a first bitstream. The GOP of the first bitstream can be 50 frames, and the bitrate can be 6 Mbps. The first device sends the first bitstream to the second device.

[0265] In another example, the first device may also package reference high-resolution image frames into a second bitstream, for example, package high-resolution I-frames into a second bitstream, and send the second bitstream and sub-bitstream to the second device.

[0266] In one possible implementation, the first device can send the first bitstream, or the second bitstream and sub-bitstream, to the base station via a 5G wireless network, and the base station then forwards it to the second device. Since the embodiments of this application transmit a small number of high-resolution image frames, the amount of data sent from the first device to the second device is relatively small, and the bandwidth occupied by the video stream is narrow. Therefore, with the same bandwidth, a single base station can access more video streams.

[0267] Second equipment

[0268] Unpacking

[0269] The first device encodes, extracts, and packages the acquired images into a data stream, then transmits the data stream to the second device via a wireless or wired network. The second device receives the video stream, unpacks it, and obtains first-resolution image frames and second-resolution image frames from the unpacked video stream.

[0270] In one possible implementation, step S500 may include:

[0271] Step S5000: If the received video stream is a first video stream, the second device processes the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frame in the first video stream to obtain a first resolution image frame and a second resolution image frame.

[0272] Step S5001: If the received video stream includes a second video stream and a third video stream, the second device processes the second video stream and the third video stream respectively to obtain a first resolution image frame corresponding to the second video stream and a second resolution image frame corresponding to the third video stream.

[0273] As described above, this application provides a variety of different packaging methods. For different packaging methods, the second device in the embodiments of this application can perform a variety of different unpacking processes.

[0274] If the first device, during packet assembly, inserts first-resolution image frames into the queue of second-resolution image frames according to rules or the time of the first-resolution and second-resolution image frames, assembling them into a video stream, then in this video stream, the values ​​of the Type field in the NAL packets of the first-resolution image frames and the Type field in the NAL packets of the second-resolution image frames are different. Therefore, in step S5000, the second device can process the image frames in the first video stream according to the values ​​of the Type field in the NAL packets of the image frames in the first video stream to obtain the first-resolution image frames and the second-resolution image frames.

[0275] In one possible implementation, the Type field of the NAL packet of the first resolution image frame is set to a first extended value, which is used to identify the first resolution image frame. The Type field of the NAL packet of the second resolution image frame is set to a non-extended value. That is, the Type field of the NAL packet of the first resolution image frame is set to the first extended value to distinguish it from the existing value of the Type field, thereby achieving the distinction between high-resolution and low-resolution image frames.

[0276] In one possible implementation, if the Type field of the NAL packet of an image frame in the first video stream has a first extended value, the second device processes the image frame to obtain a first resolution image frame; if the Type field of the NAL packet of an image frame in the first video stream has a non-extended value, the second device processes the image frame to obtain a second resolution image frame.

[0277] In embodiments of this application, the decoder in the second device can identify the first resolution image frame and the second resolution image frame based on the value of the Type field in the header of the NAL packet of the image frame in the first video stream, and can further identify the second resolution image frame (same source image frame) corresponding to the first resolution image frame according to rules.

[0278] In one possible implementation, if the value of the Type field is a first extended value, the decoder of this embodiment can unpack the image frame to obtain a first resolution image frame, and can also decode the first resolution image frame, and determine that the previous low-resolution image frame is a source image frame corresponding to the first resolution image frame. If the value of the Type field is a non-extended value, the decoder of this embodiment can unpack the image frame to obtain a second resolution image frame, and can further decode the second resolution image frame.

[0279] If the second device is equipped with a standard decoder, when the standard processor receives the first video stream, it can perform normal unpacking and decoding of the second resolution image frames, ignoring the first resolution image frames whose Type field value of the NAL packet is the first extended value.

[0280] If, during packet assembly, the first device inserts the Network Abstraction Layer (NAL) packet of the first resolution image frame into the NAL packet of the corresponding source image frame to form a combined frame, then step S5000, in which the second device processes the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frame in the first video stream to obtain a first resolution image frame and a second resolution image frame, may include: if the first video stream includes a combined frame, then the second device processes the combined frame to obtain a first resolution image frame and a second resolution image frame corresponding to the first resolution image frame, wherein the second resolution image frame corresponding to the first resolution image frame and the first resolution image frame are encoded from the same image. Specifically, the second device can identify the combined frame based on whether the value of the Type field of the image frame is a second extended value. If the value of the Type field of the NAL packet of the image frame in the first video stream is a second extended value, then the second device processes the combined frame to obtain a first resolution image frame and a second resolution image frame corresponding to the first resolution image frame.

[0281] In one possible implementation, the second device can parse the first resolution image frame from the main body of the NAL packet of the combined frame, and the header and the remaining part of the main body of the NAL packet of the combined frame form the second resolution image frame corresponding to the first resolution image frame.

[0282] If the first device, during packet assembly, assembles the first-resolution image frames into a second video stream and the second-resolution image frames into a third video stream, then when the second device receives the two video streams, it can identify the high-resolution video stream and the low-resolution video stream based on their resolution, and unpack the two video streams respectively to obtain the first-resolution image frames and the second-resolution image frames.

[0283] It should be noted that this application provides various packet assembly methods and corresponding unpacking methods. The first device and the second device can use one or more of these packet assembly and unpacking methods. As long as the first device assembles image frames into a video stream and sends it to the second device, the second device, upon receiving the video stream, can correctly unpack the video stream using the unpacking method corresponding to the packet assembly method of the first device.

[0284] Decoding, Super-resolution Reconstruction

[0285] After unpacking, the second device can decode the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image. Then, the second device can use the first resolution image corresponding to the first resolution image frame as a reference to reconstruct the second resolution image corresponding to the second resolution image frame. This application provides various implementation methods for super-resolution reconstruction.

[0286] Figure 12 A flowchart illustrating a super-resolution reconstruction method according to an embodiment of this application is shown. Figure 12 As shown, step S501 may include:

[0287] In step S5010, the second device decodes the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image.

[0288] Step S5011: The second device inputs the first resolution image, the image corresponding to the first resolution image, and the second resolution image into the neural network to perform super-resolution reconstruction on the second resolution image;

[0289] Wherein, the image corresponding to the first resolution image is the image in the second resolution image that corresponds to the first resolution image; the image in the second resolution image that corresponds to the first resolution image is the same image acquired by the first device as the first resolution image;

[0290] Alternatively, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image.

[0291] Because the super-resolution neural network is trained or inferred using fewer high-resolution images, the image processing method of the present application can achieve image super-resolution reconstruction faster than the process of training and inference using a large number of high-resolution images, reducing computation time and enabling real-time reconstruction of video.

[0292] The image processing method according to the embodiments of this application uses a high-resolution image of the current scene as a reference to perform inference on the super-resolution neural network. Therefore, a simple network structure can be used to achieve high-quality super-resolution reconstruction, which can further shorten the computation time while improving the quality of the reconstructed image.

[0293] Furthermore, the image processing method according to the above embodiments of this application does not require real-time training of the neural network, which helps to accelerate the inference process. Therefore, for specific scenarios, a trained neural network can be used for rapid deployment and reconstruction.

[0294] In the several embodiments of generating a first-resolution image frame and a second-resolution image frame provided in this application, the second-resolution image frame may contain a source image frame corresponding to the first-resolution image frame, or it may not contain a source image frame corresponding to the first-resolution image frame. Therefore, Figure 12 In the embodiment shown, the image corresponding to the first resolution image can be divided into two different cases.

[0295] The first type is: the image corresponding to the first resolution image is the image in the second resolution image that corresponds to the first resolution image, and the image in the second resolution image that corresponds to the first resolution image is the same image acquired by the first device that corresponds to the first resolution image. For example, such as Figure 10 As shown, the images acquired by the first device are a, b, c, d, e... in sequence. After low-resolution encoding, the second-resolution image frame obtained can be a. L b L c L d L e L ...The first resolution image frame obtained after high-resolution encoding and frame extraction can be b H and d H Frame extraction can be performed before or after high-resolution encoding; this application does not impose any limitation on this. Wherein, b H and b L Correspondingly, both are generated by encoding image b; d H and d L Correspondingly, both are generated by encoding image d. The second device, after receiving the data from the first device, processes image b... H and b L Decoding yields the first-resolution image b. h Second resolution image b l Second resolution image b l For the first resolution image b h The corresponding image.

[0296] The second method is that the image corresponding to the first resolution image can also be an image obtained by downsampling the first resolution image. Downsampling the first resolution image yields a low-resolution image. In other words, if there is no image corresponding to the first resolution image in the second resolution image, a low-resolution image can be obtained by downsampling the first resolution image and used as the image corresponding to the first resolution image.

[0297] When the second-resolution image frame generated by the first device contains a source image frame of the same origin as the first-resolution image frame, the backend does not need to downsample to obtain the image corresponding to the first-resolution image, which can reduce the processing pressure on the backend.

[0298] For step S5010, the second device can use relevant decoding technology to decode the first resolution image frame and the second resolution image frame respectively to obtain the corresponding first resolution image and second resolution image. The relevant decoding technology can correspond to the encoding technology used by the first device. This application does not limit the specific encoding and decoding technology.

[0299] In one possible implementation, the neural network used for super-resolution reconstruction of the second resolution image in step S5011 can be a convolutional neural network.

[0300] For step S5011, Figure 13 A flowchart of a reconstruction method according to an embodiment of this application is shown. Figure 14 This diagram illustrates a reconstruction process according to an embodiment of the present application, in conjunction with... Figure 13 and Figure 14 The super-resolution reconstruction method provided in this application is described.

[0301] like Figure 13 As shown, step S5011 may include the following steps:

[0302] Step S130: The second device performs convolution on the second resolution image to obtain a first feature map, and performs convolution on the image corresponding to the first resolution image to obtain a second feature map;

[0303] Step S131: The second device splices the first feature map and the second feature map to obtain a first spliced ​​feature map;

[0304] Step S132: The second device performs convolution on the first stitched feature map to obtain the third feature map;

[0305] Step S133: The second device performs convolution on the first resolution image to obtain a fourth feature map;

[0306] Step S134: The second device splices the third feature map and the fourth feature map to obtain a second spliced ​​feature map;

[0307] In step S135, the second device performs convolution on the second stitched feature map to obtain the reconstructed image of the second resolution image.

[0308] The image processing method according to the embodiments of this application uses a high-resolution image of the current scene as a reference to train and infer a super-resolution neural network. With a simple network structure, high-quality super-resolution reconstruction can be achieved, which can further shorten the computation time while improving the quality of the reconstructed image.

[0309] like Figure 14 As shown, H represents the first resolution image, L represents the image corresponding to the first resolution image, and X represents the second resolution image (i.e., the image that needs to be reconstructed using super-resolution). In step S130, the second device performs convolution on the second resolution image X to obtain the first feature. Figure X The second feature map L' is obtained by convolving the image L corresponding to the first resolution image. The convolution kernel used for the second resolution image and the image corresponding to the first resolution image can be the same, and the feature maps of the input images are extracted in the same way. Using the same convolution kernel means that the number of convolution kernels, their dimensions, and the values ​​of the elements in the convolution kernels are all the same.

[0310] In step S131, the second device targets the first feature. Figure X By concatenating the first feature map L' with the second feature map L', a first concatenated feature map L'X' can be obtained. The specific concatenation method can employ image feature-based concatenation algorithms from related technologies to concatenate the first and second feature maps; this application does not limit this approach.

[0311] In step S132, the second device performs convolution on the first spliced ​​feature map L'X' to obtain the third feature map H.

[0312] In step S133, the second device convolves the first resolution image to obtain the fourth feature map H'.

[0313] In step S134, the second device splices the third feature map H” and the fourth feature map H’ to obtain the second spliced ​​feature map H’H”.

[0314] In step S135, the second device convolves the second stitched feature map H'H” to obtain the reconstructed image XH of the second resolution image.

[0315] The convolution operations performed in steps S130, S132, S133, and S135 can be one or more convolutions. When performing multiple convolutions, different convolution kernels can be used, such as 1x1, 3x3, 5x5, 7x7, and other different convolution kernels. Residual networks can also be used for acceleration. For example, the final high-resolution output image XH can be directly added to the input high-resolution image before output.

[0316] like Figure 13 As shown, the process of convolving the first resolution image and the second resolution image in step S130 can be performed simultaneously or sequentially. Steps S130 and S133 can also be performed simultaneously or sequentially. Step S133 can also be performed simultaneously or sequentially with step S132. This application does not limit this.

[0317] In one possible implementation, when the second device performs super-resolution reconstruction on the second resolution image, the referenced first resolution image and the image corresponding to the first resolution image can be a pair of images. This pair of first resolution images and the image corresponding to the first resolution image can be the image that is closest in time to the second resolution image.

[0318] In another possible implementation, when the second device performs super-resolution reconstruction on the second resolution image, the referenced first resolution image and its corresponding image can be multiple pairs of images. These multiple pairs of first resolution images and their corresponding images can be multiple pairs of images preceding the second resolution image. If there are multiple pairs of first resolution images and their corresponding images, the second device can save them each time it calculates the fourth feature map H' of the first resolution image and the second feature map L' of the image corresponding to the first resolution image. This way, when performing super-resolution reconstruction on subsequent second resolution images, the second device can refer to all previous first resolution images and their corresponding images.

[0319] Figure 15 A flowchart illustrating a super-resolution reconstruction method according to another embodiment of this application is shown. Figure 15 As shown, step S501 may include:

[0320] In step S5010, the second device decodes the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image.

[0321] In step S5012, the second device inputs the first resolution image and the image corresponding to the first resolution image into the first neural network for training to obtain network parameters;

[0322] Wherein, the image corresponding to the first resolution image is the image in the second resolution image that corresponds to the first resolution image; the image in the second resolution image that corresponds to the first resolution image is the same image acquired by the first device as the first resolution image; or, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image.

[0323] In step S5013, the second device inputs the second resolution image into the second neural network for computation, and performs super-resolution reconstruction on the second resolution image;

[0324] The network parameters of the second neural network are obtained by real-time synchronization based on the network parameters of the first neural network.

[0325] The process of step S5010 can be referred to the description above, and will not be repeated here. The image corresponding to the first resolution image in step S5012 can be divided into two different cases, which are described in detail above, and will not be repeated here.

[0326] Steps S5012 and S5013 can be performed simultaneously. The second device trains the first neural network while simultaneously performing inference on the second neural network. At the same time, the second device can update the network parameters of the second neural network in real time based on the network parameters obtained from training the first neural network. In this way, the second device can perform super-resolution reconstruction of the second-resolution image by referring to the first-resolution image and its corresponding image.

[0327] Figure 16 A schematic diagram of a super-resolution reconstruction method according to an embodiment of this application is shown. Figure 16 As shown, the second device can include a first neural network and a second neural network. The network structures of the first and second neural networks are identical. Both the first and second neural networks can include an input layer, a hidden layer, and an output layer, and both can be convolutional neural networks. The network parameters of the second neural network are obtained in real time through synchronization with the network parameters of the first neural network.

[0328] Figure 16In this diagram, H represents the first-resolution image and L represents the image corresponding to the first-resolution image. Therefore, the first-resolution image H and its corresponding image L are used to train the first neural network to obtain network parameters. The first-resolution image H is the training target, and the corresponding image L is the input data for the first neural network. The second device can use methods such as loss functions and error back propagation (BP) algorithms to train the first neural network and obtain its parameters. The second device can perform a limited number of training iterations using a pair of images at a time. For example, it can use the pair of first-resolution images and their corresponding images that are most recent to the second-resolution image to train 10 times, obtaining network parameters which are then synchronized to the second neural network. Alternatively, the second device can perform continuous training. For instance, each time the first-resolution image and its corresponding image are decoded, they are input into the first neural network. Based on previous training, the network parameters are iteratively updated, and the updated parameters are synchronized to the second neural network. The synchronization frequency can be once every time the network parameters are updated, or once every few updates. For example, the network parameters are synchronized to the second neural network every three updates.

[0329] The second device can synchronize network parameters to the second neural network in real time. The input X to the second neural network represents a second-resolution image. The second neural network performs super-resolution reconstruction (forward inference) on the second-resolution image X to obtain a corresponding high-resolution image XH. The forward inference process may include: convolving the second-resolution image; after one or more convolutions, the high-resolution image XH can be obtained, and the high-resolution image XH is output. The second neural network can use different convolution kernels for one or more convolutions, such as 1x1, 3x3, 5x5, 7x7, etc.

[0330] The super-resolution reconstruction method described in the above embodiments of this application can use high-resolution images of the current scene as references to train and infer the super-resolution neural network. Therefore, a simple network structure can achieve high-quality super-resolution reconstruction, improving the quality of the reconstructed image while reducing computation time. Furthermore, a small number of reference image pairs allows for asynchronous parallel training and inference. The network parameters generated during online training can be updated in real time to guide the reconstruction process of subsequent frames. Compared to pre-trained neural networks (with unchanged parameters), this method is more adaptable to different application scenarios and ensures the quality of the reconstructed image.

[0331] For example, in related technologies, super-resolution reconstruction techniques involving training and inference without limiting the scene require pre-training with large amounts of various datasets. These neural networks have numerous layers, convolutional depths greater than 30, complex structures, and high computational costs, limiting them to offline operation. In contrast, the super-resolution reconstruction method of this application uses a small number of high-resolution images of the current scene as references, limiting the reference or training scene. It employs lightweight neural networks (convolutional depths can be less than 5) to achieve high-quality image reconstruction with low computational cost and supports multi-path real-time reconstruction.

[0332] Figure 17 A schematic diagram illustrating super-resolution reconstruction performed by a second device according to an embodiment of this application is shown. Figure 17 As shown, the second device unpacks the received bitstream, and can obtain the following: Figure 17 The reference high-resolution image frame, the corresponding low-resolution image frame (also called the reference low-resolution image frame), and the low-resolution image frame (e.g.) are shown. Figure 17 The P-frame obtained by unpacking is shown. Specifically, for the received bitstream, the second device can perform unpacking processing frame by frame. If the header of the NAL packet of the image frame in the first bitstream includes an extended field, the second device can process (unpack) the image frame to obtain a reference high-resolution image frame; if the header of the NAL packet of the image frame in the first bitstream does not include an extended field, the second device can process (unpack) the image frame to obtain a low-resolution image frame.

[0333] The second device can decode the reference high-resolution image frame, the reference low-resolution image frame, and the low-resolution image frame to obtain the reference high-resolution image, the reference low-resolution image, and the low-resolution image.

[0334] The second device can use a reference high-resolution image and a reference low-resolution image as references, and use a lightweight super-resolution reconstruction neural network to perform super-resolution reconstruction on the low-resolution image. The specific reconstruction process can be found in the implementation method provided above, and will not be repeated here.

[0335] In the embodiments of this application, the first device can employ various frame extraction and packet assembly methods, and the second device can employ various super-resolution reconstruction methods. Different implementation methods for the front-end and back-end can be combined with each other; this application does not limit this combination. The following examples illustrate various combinations of technical solutions.

[0336] Application Example 1

[0337] The first device simultaneously inputs the acquired images into a low-resolution encoder and a high-resolution encoder for low-resolution encoding and high-resolution encoding, generating low-resolution image frames and high-resolution image frames. The low-resolution image frame is the second-resolution image frame. The first device extracts frames from the high-resolution image frame to obtain a first-resolution image frame. The second-resolution image frame contains an image frame corresponding to the first-resolution image frame.

[0338] During packet assembly, the first device locates the corresponding source image frame for the first resolution image frame within the sequence of second resolution image frames, inserts the first resolution image frame after the corresponding source image frame, and assembles the packet into the first video stream. Figure 11a As shown.

[0339] The second device receives the first video stream sent by the first device. It first unpacks the image frames in the first video stream, identifying high-resolution image frames based on the Type field value of the NAL packet during unpacking. If the currently unpacked image frame is a low-resolution image frame, and a corresponding high-resolution image frame with the same content follows it, the second device can decode the low-resolution image frame and the corresponding high-resolution image frame separately to obtain the low-resolution image and the high-resolution image, and then cache them. If the currently unpacked image frame is a low-resolution image frame, and no corresponding high-resolution image frame follows it, the low-resolution image frame is decoded to obtain the low-resolution image (the low-resolution image to be reconstructed). The second device inputs the cached high-resolution image, the corresponding low-resolution image, and the low-resolution image to be reconstructed into a neural network for super-resolution reconstruction to obtain the high-resolution image.

[0340] like Figure 11a As shown, the second device... Figure 11a When the first L-frame is unpacked, a corresponding H-frame follows. The L-frame and H-frame are decoded to obtain a low-resolution image L1 and a high-resolution image H1, which are then buffered. When the second device unpacks the second L-frame, there is no corresponding H-frame following it. The second L-frame is decoded to obtain a low-resolution image L2. The second device can then input L1, H1, and L2 into a neural network for super-resolution reconstruction, obtaining the high-resolution image corresponding to L2. The second device can continue unpacking the third L-frame. There is no corresponding H-frame following the third L-frame, so the second device decodes the third L-frame to obtain a low-resolution image L3. The second device can then input L1, H1, and L3 into a neural network for super-resolution reconstruction, obtaining the high-resolution image corresponding to L3… This process is repeated sequentially. Figure 11aWhen the L-frame precedes the second H-frame, it is determined that a corresponding H-frame follows that L-frame. The L-frame and H-frame are decoded to obtain the low-resolution image Lm and the high-resolution image H2, which are then buffered. Here, m is a positive integer greater than 1. The second device unpacks the image. Figure 11a When the second H frame in the image is followed by an L frame, if it is determined that there is no corresponding H frame after the L frame, the second device can decode the L frame to obtain a low-resolution image frame Lm+1. The second device can then input Lm, H2, and Lm+1 into a neural network for super-resolution reconstruction to obtain a high-resolution image corresponding to Lm+1. Alternatively, the second device can input L1, H1, Lm, H2, and Lm+1 into the neural network to perform super-resolution reconstruction on Lm+1 to obtain a high-resolution image corresponding to Lm+1.

[0341] In this embodiment, the specific super-resolution reconstruction process can be found in [reference needed]. Figure 14 And the above about Figure 14 Some details will not be repeated here.

[0342] Application Example 2

[0343] The processing procedure for the first device in Application Example 2 is the same as that in Application Example 1.

[0344] The second device receives the first video stream sent by the first device. It first unpacks the image frames in the first video stream, identifying high-resolution image frames based on the Type field value of the NAL packet during unpacking. If the currently unpacked image frame is a low-resolution image frame, and a corresponding high-resolution image frame with the same content follows it, the second device can decode the low-resolution image frame and the corresponding high-resolution image frame separately to obtain a low-resolution image and a high-resolution image. The high-resolution image and the corresponding low-resolution image are then input into the first neural network for training to obtain network parameters, which are then synchronized to the second neural network in real time. If the currently unpacked image frame is a low-resolution image frame, and no corresponding high-resolution image frame follows it, the low-resolution image frame is decoded to obtain a low-resolution image (the low-resolution image to be reconstructed). The second device inputs the low-resolution image to be reconstructed into the second neural network for inference, enabling super-resolution reconstruction of the low-resolution image to obtain a high-resolution image.

[0345] Combination Figure 11a and Figure 16 This example is illustrated as follows: Figure 11a As shown, the second device... Figure 11a When the first L-frame is unpacked, there is a corresponding H-frame following the L-frame. Decoding the L-frame and H-frame yields a low-resolution image L1 and a high-resolution image H1. The second device then inputs the low-resolution image L1 and the high-resolution image H1 into... Figure 16 The first neural network in the first device is trained to obtain network parameters. The second device synchronizes these parameters to the second neural network in real time. This application does not limit the number of times the first neural network is trained. When the second device unpacks the second L-frame, there is no corresponding H-frame after it. The second L-frame is decoded to obtain a low-resolution image L2. The second device can input L2 into the second neural network for inference, enabling super-resolution reconstruction of L2 to obtain the high-resolution image corresponding to L2. The second device can continue to unpack the third L-frame. There is no corresponding H-frame after it. Therefore, the second device decodes the third L-frame to obtain a low-resolution image L3. The second device can input L3 into the second neural network for inference, enabling super-resolution reconstruction of L3 to obtain the high-resolution image corresponding to L3… This process is repeated sequentially. Figure 11a When the second H frame precedes an L frame, it is determined that a corresponding H frame follows that L frame. The L and H frames are decoded to obtain a low-resolution image Lm and a high-resolution image H2. These two images are then input into the first neural network for further training, where m is a positive integer greater than 1. The second device synchronizes the network parameters obtained from the training to the second neural network in real time. The second device unpacks the... Figure 11a When the second H frame is followed by an L frame, if it is determined that there is no corresponding H frame after the L frame, the second device can decode the L frame to obtain a low-resolution image Lm+1. The second device can then input Lm+1 into the second neural network for inference, which can achieve super-resolution reconstruction of Lm+1 to obtain the high-resolution image corresponding to Lm+1.

[0346] Application Example 3

[0347] The first device simultaneously inputs the acquired images into a low-resolution encoder and a high-resolution encoder to perform low-resolution encoding and high-resolution encoding, generating low-resolution image frames and high-resolution image frames. The low-resolution image frame is the second-resolution image frame. The first device extracts frames from the high-resolution image frame to obtain a first-resolution image frame. The second-resolution image frame contains an image frame that corresponds to the first-resolution image frame.

[0348] During packet assembly, the first device locates the corresponding source image frame for the first resolution image frame within the sequence of second resolution image frames, assembles the first resolution image frame and its corresponding source image frame into a combined frame, and then assembles the combined frame and the low-resolution image frame into a first video stream, such as... Figure 11b As shown.

[0349] The second device receives the first video stream sent by the first device. It first unpacks the image frames in the first video stream, identifying combined frames based on the value of the Type field during unpacking. If the currently unpacked image frame is a combined frame, the second device can unpack it to obtain a pair of image frames: a high-resolution image frame and its corresponding low-resolution image frame. The second device can then decode the high-resolution image frame and its corresponding low-resolution image frame separately to obtain a high-resolution image and a low-resolution image, which are then buffered. If the currently unpacked image frame is a low-resolution image frame, it is decoded to obtain a low-resolution image (the low-resolution image to be reconstructed). The second device then inputs the buffered high-resolution image, the corresponding low-resolution image, and the low-resolution image to be reconstructed into a neural network for super-resolution reconstruction to obtain a high-resolution image.

[0350] like Figure 11b As shown, the second device... Figure 11b The first combined frame is unpacked to obtain a high-resolution image frame H and its corresponding low-resolution image frame L. The L and H frames are then decoded to obtain low-resolution image L1 and high-resolution image H1, which are then buffered. The second device decodes the L frames following the combined frames to obtain low-resolution image L2. The second device can then input L1, H1, and L2 into a neural network for super-resolution reconstruction to obtain the high-resolution image corresponding to L2. The second device can continue decoding the next L frame to obtain low-resolution image L3. The second device can then input L1, H1, and L3 into a neural network for super-resolution reconstruction to obtain the high-resolution image corresponding to L3… and so on, repeating the above process. Figure 11b The second combined frame yields a high-resolution image frame and its corresponding low-resolution image frame. These are then decoded to obtain the low-resolution image Lm and the high-resolution image H2. Lm and H2 are buffered, where m is a positive integer greater than or equal to 1. The second device unpacks the image. Figure 11b When the second combined frame is followed by the L frame, the second device can decode the L frame to obtain the low-resolution image frame Lm+1. The second device can input Lm, H2 and Lm+1 into the neural network for super-resolution reconstruction to obtain the high-resolution image corresponding to Lm+1. Alternatively, the second device can input L1, H1, Lm, H2 and Lm+1 into the neural network to perform super-resolution reconstruction on Lm+1 to obtain the high-resolution image corresponding to Lm+1.

[0351] Application Example 4

[0352] The processing procedure for the first device in Application Example 4 is the same as that in Application Example 2.

[0353] The second device receives the first video stream sent by the first device. It first unpacks the image frames in the first video stream, identifying combined frames based on the Type field value of the NAL packet during unpacking. If the currently unpacked image frame is a combined frame, the second device can unpack it to obtain a pair of image frames: a high-resolution image frame and its corresponding low-resolution image frame. The second device can then decode the high-resolution image frame and its corresponding low-resolution image frame separately to obtain a high-resolution image and a low-resolution image. The second device can input the high-resolution image and its corresponding low-resolution image into a first neural network for training to obtain network parameters, and then synchronize the network parameters to the second neural network in real time. If the currently unpacked image frame is a low-resolution image frame, it decodes the low-resolution image frame to obtain a low-resolution image (the low-resolution image to be reconstructed). The second device inputs the low-resolution image to be reconstructed into the second neural network for inference, enabling super-resolution reconstruction of the low-resolution image to obtain a high-resolution image.

[0354] Combination Figure 11b and Figure 16 This example is illustrated as follows: Figure 11b As shown, the second device... Figure 11b The first combined frame in the image is unpacked to obtain a high-resolution image frame H and a corresponding low-resolution image frame L. The L and H frames are then decoded to obtain a low-resolution image L1 and a high-resolution image H1. The second device then inputs the low-resolution image L1 and the high-resolution image H1 into the image. Figure 16 The first neural network in the device is trained to obtain network parameters, and the second device synchronizes the network parameters to the second neural network in real time. This application does not limit the number of times the first neural network is trained.

[0355] The second device decodes the combined L-frames to obtain a low-resolution image L2. The second device can then input L2 into a second neural network for inference, enabling super-resolution reconstruction of L2 to obtain a corresponding high-resolution image. The second device can continue decoding the next L-frame to obtain a low-resolution image L3. The second device can then input L3 into the second neural network for inference, enabling super-resolution reconstruction of L3 to obtain a corresponding high-resolution image… and so on, repeating the above process. The second device unpacks… Figure 11b The second combined frame yields a high-resolution image frame and its corresponding low-resolution image frame, which are then decoded to obtain a high-resolution image H2 and a low-resolution image Lm, where m is a positive integer greater than or equal to 1. The second device inputs the high-resolution image H2 and the low-resolution image Lm into the first neural network for further training to obtain network parameters. The second device then synchronizes the trained network parameters to the second neural network in real time. The second device unpacks the... Figure 11b When the second combined frame is followed by the L frame, the second device can decode the L frame to obtain the low-resolution image frame Lm+1. The second device can then input Lm+1 into the second neural network for inference, which can achieve super-resolution reconstruction of Lm+1 to obtain the high-resolution image corresponding to Lm+1.

[0356] Application Example 5

[0357] The first device extracts and encodes frames from the acquired images, generating a second-resolution image frame that does not contain an image frame corresponding to the first-resolution image frame. The first device can use either one or two video streams for packet assembly. This example illustrates the use of one video stream.

[0358] When using a single video stream for transmission, the first device can arbitrarily insert first-resolution image frames into the queue of second-resolution image frames. Assume... Figure 11a There is no direct correspondence between L-frames and H-frames. Inserting the first-resolution image frame into the queue of the second-resolution image frames will yield the desired result. Figure 11a The first video stream shown.

[0359] The second device receives the first video stream sent by the first device. It first unpacks the image frames in the first video stream. During unpacking, it identifies high-resolution image frames based on the value of the Type field in the NAL packet. If the Type field value of the NAL packet of the currently unpacked image frame is the first extended value, it is a high-resolution image frame, and the second device can decode the high-resolution image frame to obtain a high-resolution image. The second device can also downsample the high-resolution image to obtain a corresponding low-resolution image, and then buffer both the high-resolution image and the corresponding low-resolution image.

[0360] If the Type field value of the NAL packet of the currently unpacked image frame is a non-extended value, it is a low-resolution image frame. The second device can decode the low-resolution image frame to obtain a low-resolution image (the low-resolution image to be reconstructed). The second device inputs the cached high-resolution image, the corresponding low-resolution image, and the low-resolution image to be reconstructed into the neural network for super-resolution reconstruction to obtain a high-resolution image.

[0361] like Figure 11a As shown, the second device... Figure 11aThe first L-frame is decoded to obtain a low-resolution image L1. L1, along with previously cached high-resolution images and their corresponding low-resolution images, are input into a neural network to perform super-resolution reconstruction on L1, resulting in a high-resolution image corresponding to L1. When the second device unpacks the first H-frame, the Type field value of the NAL packet in the first H-frame is the first extended value. Therefore, the second device decodes the H-frame to obtain a high-resolution image H1. H1 is downsampled to obtain a low-resolution image L' corresponding to the high-resolution image H1. The second device caches H1 and L'. The second device decodes the second L-frame to obtain a low-resolution image L2. The second device can input H1, L', and L2 into a neural network for super-resolution reconstruction, resulting in a high-resolution image corresponding to L2. The second device can continue decoding the third L-frame to obtain a low-resolution image L3. The second device can input H1, L', and L3 into a neural network for super-resolution reconstruction, resulting in a high-resolution image corresponding to L3... and so on. The process is repeated until the second device unpacks the first H-frame. Figure 11a When the second H frame is encountered, the Type field value of the NAL packet in the second H frame is the first extended value. The second device decodes the second H frame to obtain the high-resolution image H2, and downsamples H2 to obtain the corresponding low-resolution image L", which is then buffered. The second device unpacks the image. Figure 11a When the second H frame is followed by an L frame, the second device can decode the L frame to obtain a low-resolution image Lm. The second device can input H2, L”, and Lm into a neural network for super-resolution reconstruction to obtain a high-resolution image corresponding to Lm. Alternatively, the second device can input H1, L', H2, L”, and Lm into a neural network to perform super-resolution reconstruction on Lm to obtain a high-resolution image corresponding to Lm, where m is a positive integer.

[0362] Application Example 6

[0363] The processing procedure for the first device in Application Example 6 is the same as that in Application Example 5.

[0364] The second device receives the first video stream sent by the first device. It first unpacks the image frames in the first video stream, identifying high-resolution image frames based on the value of the Type field in the NAL packet during unpacking. If the Type field value of the NAL packet of the currently unpacked image frame is a first extended value, it is a high-resolution image frame, and the second device can decode it to obtain a high-resolution image. The second device can also downsample the high-resolution image to obtain a corresponding low-resolution image.

[0365] Combination Figure 11a and Figure 16 This example is illustrated as follows: Figure 11a As shown, the second device... Figure 11aThe first L-frame is decoded to obtain a low-resolution image L1. L1 is then input into the second neural network for inference, and super-resolution reconstruction is performed on L1 to obtain the corresponding high-resolution image. When the second device unpacks the first H-frame, the Type field value of the NAL packet in the first H-frame is the first extended value. Therefore, the second device decodes the H-frame to obtain a high-resolution image H1. H1 is downsampled to obtain a low-resolution image L' corresponding to the high-resolution image H1. The second device inputs H1 and L' into the first neural network for training to obtain new network parameters, which can then be synchronized to the second neural network. The second device decodes the second L-frame to obtain a low-resolution image L2. L2 can be input into the second neural network for super-resolution reconstruction to obtain the corresponding high-resolution image. The second device can continue to decode the third L-frame to obtain a low-resolution image L3. L3 can be input into the second neural network for super-resolution reconstruction to obtain the corresponding high-resolution image, and so on, repeating the above process. Figure 11a When the second H frame is encountered, the Type field value of the NAL packet in the second H frame is the first extended value. The second device decodes the second H frame to obtain the high-resolution image H2, and downsamples H2 to obtain the corresponding low-resolution image L". The second device can input H2 and L" into the first neural network for training, update the network parameters, and synchronize the updated network parameters to the second neural network. The second device then unpacks the second H frame. Figure 11a When the second H frame is followed by an L frame, the second device can decode the L frame to obtain a low-resolution image Lm. The second device can then input Lm into the second neural network for super-resolution reconstruction to obtain the high-resolution image corresponding to Lm.

[0366] Figure 18 and Figure 19 Schematic diagrams illustrating the effects of the image processing method of this application compared to related technologies are shown. For example... Figure 18 As shown, the IPC acquires images and encodes them to generate two bitstreams: a high-resolution main bitstream and a low-resolution sub-bitstream, which is as follows: Figure 18 The diagram shows a 12Mbps high-definition (HD) stream and a 4Mbps low-definition (LD) stream. The front-end processing device extracts frames from the high-resolution main stream (i.e., extracts frames from the HD stream), and then combines these extracted HD frames with frames from the low-resolution sub-stream to form a new stream. This new stream is then uploaded to the second device via the 5G wireless uplink network. Compared to transmitting 12Mbps HD and 4Mbps LDL streams, the image processing method described in this application saves bandwidth, allowing more video streams to be accessed with the same network bandwidth.

[0367] The second device can be an edge computing node in the network or a video server. The second device receives the bitstream, unpacks it, decodes it, and performs super-resolution reconstruction to obtain a high-definition bitstream, which is then displayed through a display device.

[0368] like Figure 19 As shown, in related technology (a), when using neural networks for super-resolution reconstruction, each frame requires more than 300ms to complete the reconstruction of a single 1080P resolution image to obtain a 4K high-definition image. The neural networks in related technologies are relatively complex, with more than 20 layers. Therefore, both the initial training and subsequent inference processes require a significant amount of time and need to be performed offline. Furthermore, the training and application scenarios in related technologies are not highly relevant, resulting in low-quality reconstructed images.

[0369] The implementation method of this application uses a high-resolution image of the current scene as a reference to train and infer the super-resolution neural network. Therefore, a simple network structure can achieve high-quality super-resolution reconstruction, improving the quality of the reconstructed image while reducing computation time. Figure 19 As shown, the neural network in embodiment (b) of this application, including the reference calibration part, has only 6 layers. It only takes 20ms to reconstruct a 4K high-definition image from a single frame of 1080P resolution image, which is more than 10 times faster than 300ms. Therefore, the image processing method of this application can train and infer simultaneously, and the training results are updated in real time as a reference for inference, further ensuring the correlation between training and inference and greatly improving the image reconstruction quality.

[0370] This application also provides an image processing apparatus that can be disposed in a first device. Figure 20 A block diagram of an image processing apparatus according to an embodiment of this application is shown. Figure 20 As shown, the image processing apparatus of this embodiment may include:

[0371] The encoding unit 2001 is used to encode the acquired image to generate a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of the first resolution image frames is less than the number of the second resolution image frames.

[0372] Packet assembly unit 2002 is used to package the first resolution image frame and the second resolution image frame into a video stream and send the video stream to the second device.

[0373] Compared to image processing in related technologies, the image processing apparatus of this application has a low frame rate for high-resolution image frames obtained by encoding the acquired images by the first device, and most of the encoded image frames are low-resolution image frames, meaning that the number of high-resolution image frames is relatively small. Therefore, the amount of data sent from the first device to the second device is small, and the bandwidth occupied by the video stream is narrow. Thus, with the same bandwidth, more video streams can be accessed.

[0374] In one possible implementation, the encoding unit 2001 includes:

[0375] The first encoding subunit is used to encode the acquired image to generate a high-resolution image frame and a low-resolution image frame; wherein the low-resolution image frame is a second-resolution frame.

[0376] A frame extraction unit is used to extract a portion of frames from a high-resolution image frame as a first-resolution image frame.

[0377] The frame extraction process described in the above embodiments of this application can reuse the high-resolution bitstream and low-resolution bitstream already generated by the camera. By extracting frames from the already generated high-resolution bitstream and low-resolution bitstream, a first resolution image frame and a second resolution image frame can be obtained. In other words, the image frames of the main bitstream and sub-bitstream in the dual bitstream technology can be reused without secondary encoding, without additional computing power, and without hardware dependence.

[0378] In one possible implementation, the encoding unit 2001 includes:

[0379] The second encoding subunit is used to extract all or part of the acquired images and encode them to generate a second resolution image frame, and to extract part of the acquired images and encode them to generate a first resolution image frame.

[0380] The frame-sampling method described in the above embodiments of this application performs frame-sampling on the acquired image before encoding, thus reducing the amount of image encoding required and improving encoding efficiency.

[0381] In one possible implementation, the packet assembly unit 2002 includes:

[0382] The first group of packet units is used to package the first resolution image frame and the second resolution image frame into a first video stream;

[0383] And / or,

[0384] The second group of sub-units is used to package the first resolution image frames into a second video stream, and to package the second resolution image frames into a third video stream.

[0385] In one possible implementation, the second resolution image frame contains a source image frame corresponding to the first resolution image frame. The first resolution image frame and the corresponding source image frame are generated by encoding the same image. In the first video stream, the first resolution image frame and the corresponding source image frame are arranged according to rules, or the first resolution image frame and the corresponding source image frame are combined into a combined frame. This packetization method maintains a consistent correspondence between the high-resolution and low-resolution image frames acquired by the first device and the video processor. Thus, the second device can directly use pairs of high-resolution and low-resolution image frames as a reference for reconstructing the low-resolution image, simplifying the processing of the second device.

[0386] In one possible implementation, in the first video stream, the first resolution image frame bit is adjacent to the corresponding source image frame.

[0387] In one possible implementation, the packet assembly unit 2002 includes:

[0388] The third group of sub-units is used to insert the Network Abstraction Layer (NAL) packet of the first resolution image frame into the NAL packet of the corresponding source image frame to form the combined frame.

[0389] In one possible implementation, in the first video stream, the value of the Type field in the NAL packet of the first resolution image frame is different from the value of the Type field in the NAL packet of the second resolution image frame.

[0390] This application also provides an image processing apparatus that can be disposed in a second device. Figure 21 A block diagram of an image processing apparatus according to an embodiment of this application is shown. Figure 21 As shown, the image processing apparatus of this embodiment may include:

[0391] The processing unit 2101 is used to process the received video stream to obtain a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of the first resolution image frames is less than the number of the second resolution image frames.

[0392] The reconstruction unit 2102 is used to reconstruct the second resolution image after decoding the second resolution image frame, using the first resolution image frame decoded as a reference.

[0393] Because fewer high-resolution images are used as references for super-resolution reconstruction of low-resolution images, the image processing apparatus of the present application can achieve image super-resolution reconstruction faster, reduce computation time, and achieve real-time reconstruction of video.

[0394] In related technologies, a large number of images are pre-collected to train a super-resolution reconstruction neural network. However, since the actual application scenario is often not necessarily related to the pre-collected scenario, the super-resolution reconstruction neural network structure is relatively complex, the training and inference process is time-consuming, and the image reconstruction quality cannot be guaranteed in order to improve the accuracy of super-resolution reconstruction. In contrast, the image processing apparatus according to the embodiments of this application uses a high-resolution image of the current scene as a reference to perform super-resolution reconstruction on a low-resolution image. Therefore, the scene correlation is relatively strong, and high-quality super-resolution reconstruction can be achieved. This further reduces the computation time while improving the quality of the reconstructed image.

[0395] In one possible implementation, the processing unit 2101 includes:

[0396] The first processing subunit is configured to, if the received video stream is a first video stream, process the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frame in the first video stream to obtain a first resolution image frame and a second resolution image frame.

[0397] In one possible implementation, the first processing subunit is further configured to: if the value of the Type field of the NAL packet of an image frame in the first video stream is a first extended value, then process the image frame to obtain a first resolution image frame; if the value of the Type field of the NAL packet of an image frame in the first video stream is a non-extended value, then process the image frame to obtain a second resolution image frame.

[0398] In one possible implementation, the first processing subunit is further configured to:

[0399] If the first video stream includes a composite frame, the composite frame is processed to obtain a first resolution image frame and a second resolution image frame corresponding to the first resolution image frame. The second resolution image frame corresponding to the first resolution image frame and the first resolution image frame are obtained by encoding the same image. The value of the Type field of the NAL packet of the composite frame is a second extended value.

[0400] In one possible implementation, the processing unit 2101 further includes:

[0401] The second processing subunit is configured to process the second video stream and the third video stream respectively if the received video stream includes a second video stream and a third video stream, to obtain a first resolution image frame corresponding to the second video stream and a second resolution image frame corresponding to the third video stream.

[0402] In one possible implementation, the reconstruction unit 2102 includes:

[0403] The decoding subunit is used to decode the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image;

[0404] The first reconstruction subunit is used to input the first resolution image, the image corresponding to the first resolution image, and the second resolution image into the neural network to perform super-resolution reconstruction on the second resolution image.

[0405] Wherein, the image corresponding to the first resolution image is the image in the second resolution image that corresponds to the first resolution image; the image in the second resolution image that corresponds to the first resolution image is the same image acquired by the first device as the first resolution image;

[0406] Alternatively, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image.

[0407] In one possible implementation, the first reconstruction subunit is further configured to:

[0408] The second resolution image is convolved to obtain the first feature map, and the image corresponding to the first resolution image is convolved to obtain the second feature map;

[0409] The first feature map and the second feature map are concatenated to obtain a first concatenated feature map;

[0410] Convolution is performed on the first spliced ​​feature map to obtain the third feature map;

[0411] Convolve the first resolution image to obtain the fourth feature map;

[0412] The third feature map and the fourth feature map are concatenated to obtain the second concatenated feature map;

[0413] The second stitched feature map is convolved to obtain the reconstructed image of the second resolution image.

[0414] The image processing apparatus according to the embodiments of this application uses a high-resolution image of the current scene as a reference to train and infer a super-resolution neural network. Employing a simple network structure, it achieves high-quality super-resolution reconstruction, further reducing computation time while improving the quality of the reconstructed image. Furthermore, the use of a small number of reference image pairs allows for asynchronous parallel training and inference. The network parameters generated during online training can be updated in real time to guide the reconstruction process of subsequent frames, ensuring the quality of the reconstructed image.

[0415] In one possible implementation, the reconstruction unit 2102 further includes:

[0416] The decoding subunit is used to decode the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image;

[0417] The training subunit is used to input the first resolution image and the image corresponding to the first resolution image into the first neural network for training to obtain network parameters; wherein, the image corresponding to the first resolution image is the image corresponding to the first resolution image in the second resolution image; the image corresponding to the first resolution image in the second resolution image is the same image acquired by the first device corresponding to the first resolution image; or, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image.

[0418] The inference subunit is used to input the second resolution image into the second neural network for computation and to perform super-resolution reconstruction of the second resolution image.

[0419] The network parameters of the second neural network are obtained by real-time synchronization based on the network parameters of the first neural network.

[0420] The super-resolution reconstruction process described in the above embodiments of this application can use a high-resolution image of the current scene as a reference for training and inference of the super-resolution neural network. Therefore, a simple network structure can achieve high-quality super-resolution reconstruction, improving the quality of the reconstructed image while reducing computation time. Furthermore, a small number of reference image pairs allows for asynchronous parallel training and inference. The network parameters generated during online training can be updated in real time to guide the reconstruction process of subsequent frames, ensuring the quality of the reconstructed image.

[0421] Figure 22 A block diagram of a first device according to an embodiment of this application is shown. Figure 22 As shown, the first device 2200 may include a processor 2201, and optionally, may also include a transceiver 2202 and a memory 2203.

[0422] In one possible implementation, the first device 2200 may further include a photosensor 2206 and a display 2204. The display 2204 can be used to display video or images.

[0423] Processor 2201 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as a central nervous system or command center. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0424] The processor 2201 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 2201 is a cache memory. This memory can store instructions or data that the processor 2201 has just used or is recurring. If the processor 2201 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 2201, and thus improves the efficiency of the system.

[0425] The processor 2201 can run the image processing method executed by the first device provided in the embodiments of this application, so as to send a smaller amount of data to the second device, so that the video stream sent to the second device occupies a narrower bandwidth, and more video streams can be accessed under the same bandwidth.

[0426] Memory 2203 may be volatile memory, such as random access memory; memory may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or memory 2203 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 2203 may be a combination of the above-described memories.

[0427] This application embodiment does not limit the specific connection medium between the processor 2201 and the memory 2203. In this application embodiment, the memory 2203 and the processor 2201 are connected via a bus 2205, which is represented by a thick line in the figure. The connection methods between other components are only for illustrative purposes and are not intended to be limiting. This bus 2205 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 22 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0428] Processor 2201 may have data transmission and reception functions, enabling it to communicate with other devices. For example, in this embodiment, processor 2201 may send a video stream (or bitstream) to the second device. Figure 22 In the device, an independent data transceiver module, such as transceiver 2202, can also be set up for sending and receiving data. When the processor 2201 communicates with other devices, it can transmit data through transceiver 2202. For example, in the embodiment of this application, the processor 2201 can send the video stream (or bitstream) to the second device through transceiver 2202.

[0429] The transceiver 2202 can have wired and wireless communication functions. The wireless communication function can be implemented through an antenna, a mobile communication module, a wireless communication module, a modem processor, and a baseband processor.

[0430] Antennas are used to transmit and receive electromagnetic wave signals. A first device may include one or more antennas, each of which can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0431] The mobile communication module can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, for use on the first device.

[0432] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs an audio signal through an audio device or displays an image or video through the display 2204. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 2201 and may be housed in the same device as the mobile communication module or other functional modules.

[0433] The wireless communication module can provide solutions for wireless communication applications on the first device, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module can be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to the processor 2201. The wireless communication module can also receive signals to be transmitted from the processor 2201, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via the antenna. In this embodiment, the wireless communication module is used to transmit data between other terminal devices under the control of the processor 2201.

[0434] The display 2204 is used to display images, videos, etc. The display 2204 can also display images and videos viewed through the viewfinder. The display 2204 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0435] The light sensor 2206 can be a photosensitive element of a camera, used to transmit the collected light signal to an image sensor. The image sensor generates a raw image of the object to be photographed based on the light signal.

[0436] When the first device adopts Figure 22 When in the form shown, Figure 22The processor 2201 can invoke computer execution instructions stored in the memory 2203, enabling the device to execute the method executed by the first device in any of the above method embodiments.

[0437] Specifically, memory 2203 stores information for implementing... Figure 4 , Figure 6 , Figure 8 The method or Figure 20 The computer-executable instructions for the method steps executed by the encoding unit 2001 and the packet assembly unit 2002 can be transmitted via... Figure 22 The processor 2201 in the memory calls the computer execution instructions stored in the memory 2203 to achieve this.

[0438] Figure 23 A block diagram of a second device according to an embodiment of this application is shown. Figure 23 As shown, the second device 2300 may include a processor 2301, and optionally, may also include a transceiver 2302 and a memory 2303.

[0439] In one possible implementation, the second device 2300 may further include a display 2304. The display 2304 may be used to display video or images.

[0440] For detailed descriptions of the processor 2301, transceiver 2302, memory 2303, and display 2304, please refer to [link to relevant documentation]. Figure 22 This section provides a partial description of the processor 2201, transceiver 2202, memory 223, and display 2204. (Compared to...) Figure 22 The difference lies in that the processor 2301 can run the image processing method executed by the second device provided in the embodiments of this application, so as to use the high-resolution image of the current scene as a reference to train and infer the super-resolution neural network, and achieve high-quality super-resolution reconstruction with a simple network structure. While shortening the computing time and realizing real-time image reconstruction, the quality of the reconstructed image is improved.

[0441] Specifically, memory 2303 stores information for implementing... Figure 4 , Figure 12 , Figure 13 and Figure 15 The method or Figure 21 The computer-executable instructions for the method steps executed by the processing unit 2101 and the reconstruction unit 2102 can be implemented through... Figure 23 The processor 2301 in the memory calls the computer execution instructions stored in the memory 2303 to achieve this.

[0442] The display 2304 can be used to display a high-resolution image obtained after super-resolution reconstruction of the input low-resolution image by a second device.

[0443] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0444] Embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0445] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0446] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0447] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0448] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0449] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0450] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0451] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0452] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that, The method includes: The first device encodes the acquired images to generate a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of the first resolution image frames is less than the number of the second resolution image frames. The first device packages the first resolution image frame and the second resolution image frame into a video stream and sends the video stream to the second device; wherein, the second resolution image frame contains a source image frame corresponding to the first resolution image frame, and the first resolution image frame and the corresponding source image frame are generated by encoding the same image; in the video stream, the first resolution image frame and the corresponding source image frame are arranged according to rules, or the first resolution image frame and the corresponding source image frame are combined into a combined frame.

2. The method according to claim 1, characterized in that, The first device encodes the image to generate a first-resolution image frame and a second-resolution image frame, including: The first device encodes the acquired images to generate high-resolution image frames and low-resolution image frames; wherein, the low-resolution image frames are second-resolution frames. The first device extracts a portion of frames from the high-resolution image frames to serve as the first-resolution image frames.

3. The method according to claim 1, characterized in that, The first device encodes the acquired images to generate first-resolution image frames and second-resolution image frames, including: The first device extracts all or part of the acquired images and encodes them to generate a second-resolution image frame, and extracts part of the acquired images and encodes them to generate a first-resolution image frame.

4. The method according to claim 1, characterized in that, In the video stream, the first resolution image frame is adjacent to the corresponding source image frame.

5. The method according to claim 1, characterized in that, The first device packages the first resolution image frame and the second resolution image frame into a video stream, including: The first device inserts the Network Abstraction Layer (NAL) packet of the first resolution image frame into the NAL packet of the corresponding source image frame to form the combined frame.

6. The method according to claim 1, characterized in that, In the video stream, the value of the Type field in the NAL packet of the first resolution image frame is different from the value of the Type field in the NAL packet of the second resolution image frame.

7. An image processing method, characterized in that, The method includes: The second device processes the received video stream to obtain a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of first resolution image frames is less than the number of second resolution image frames; wherein the second resolution image frame contains a source image frame corresponding to the first resolution image frame, and the first resolution image frame and the corresponding source image frame are generated by encoding the same image; in the video stream, the first resolution image frame and the corresponding source image frame are arranged according to rules, or the first resolution image frame and the corresponding source image frame are combined into a composite frame; The second device uses the first resolution image after decoding the first resolution image frame as a reference to reconstruct the second resolution image after decoding the second resolution image frame.

8. The method according to claim 7, characterized in that, The second device processes the received video stream to obtain a first-resolution image frame and a second-resolution image frame, including: If the received video stream is a first video stream, the second device processes the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frame in the first video stream to obtain a first resolution image frame and a second resolution image frame.

9. The method according to claim 8, characterized in that, The second device processes the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frames in the first video stream to obtain a first resolution image frame and a second resolution image frame, including: If the value of the Type field of the NAL packet of an image frame in the first video stream is a first extended value, then the second device processes the image frame to obtain a first resolution image frame. If the Type field of the NAL packet of an image frame in the first video stream has a non-extended value, the second device processes the image frame to obtain a second resolution image frame.

10. The method according to claim 8, characterized in that, The second device processes the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frames in the first video stream to obtain a first resolution image frame and a second resolution image frame, including: If the first video stream includes a composite frame, the second device processes the composite frame to obtain a first resolution image frame and a second resolution image frame corresponding to the first resolution image frame. The second resolution image frame corresponding to the first resolution image frame and the first resolution image frame are obtained by encoding the same image. In this case, the value of the Type field in the NAL packet of the combined frame is the second extended value.

11. The method according to any one of claims 7-10, characterized in that, The second device uses the first resolution image decoded from the first resolution image frame as a reference to reconstruct the second resolution image decoded from the second resolution image frame, including: The second device decodes the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image; The second device inputs the first resolution image, the image corresponding to the first resolution image, and the second resolution image into the neural network to perform super-resolution reconstruction on the second resolution image; Wherein, the image corresponding to the first resolution image is the image in the second resolution image that corresponds to the first resolution image; the image in the second resolution image that corresponds to the first resolution image is the same image acquired by the first device as the first resolution image; Alternatively, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image.

12. The method according to claim 11, characterized in that, The second device inputs the first resolution image, the image corresponding to the first resolution image, and the second resolution image into a neural network to perform super-resolution reconstruction on the second resolution image, including: The second device convolves the second resolution image to obtain a first feature map, and convolves the image corresponding to the first resolution image to obtain a second feature map; The second device stitches the first feature map and the second feature map together to obtain a first stitched feature map; The second device performs convolution on the first spliced ​​feature map to obtain the third feature map; The second device convolves the first resolution image to obtain a fourth feature map; The second device splices the third feature map and the fourth feature map to obtain a second spliced ​​feature map; The second device performs convolution on the second stitched feature map to obtain the reconstructed image of the second resolution image.

13. The method according to claim 7, characterized in that, The second device uses the first resolution image decoded from the first resolution image frame as a reference to reconstruct the second resolution image decoded from the second resolution image frame, including: The second device decodes the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image; The second device inputs the first resolution image and the image corresponding to the first resolution image into the first neural network for training to obtain network parameters; wherein, the image corresponding to the first resolution image is the image in the second resolution image that corresponds to the first resolution image; the image in the second resolution image that corresponds to the first resolution image is the same image acquired by the first device as the first resolution image; or, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image. The second device inputs the second resolution image into the second neural network for processing, and performs super-resolution reconstruction on the second resolution image; The network parameters of the second neural network are obtained by real-time synchronization based on the network parameters of the first neural network.

14. An image processing apparatus, characterized in that, The device includes: The encoding unit is used to encode the acquired image to generate a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of the first resolution image frames is less than the number of the second resolution image frames. A packet assembly unit is configured to assemble the first resolution image frame and the second resolution image frame into a video stream and send the video stream to a second device; wherein, the second resolution image frame contains a source image frame corresponding to the first resolution image frame, and the first resolution image frame and the corresponding source image frame are generated by encoding the same image; in the video stream, the first resolution image frame and the corresponding source image frame are arranged according to a rule, or the first resolution image frame and the corresponding source image frame are combined into a combined frame.

15. The apparatus according to claim 14, characterized in that, The encoding unit includes: The first encoding subunit is used to encode the acquired image to generate a high-resolution image frame and a low-resolution image frame; wherein the low-resolution image frame is a second-resolution frame. A frame extraction unit is used to extract a portion of frames from a high-resolution image frame as a first-resolution image frame.

16. The apparatus according to claim 14, characterized in that, The encoding unit includes: The second encoding subunit is used to extract all or part of the acquired images and encode them to generate a second resolution image frame, and to extract part of the acquired images and encode them to generate a first resolution image frame.

17. The apparatus according to claim 14, characterized in that, In the video stream, the first resolution image frame is adjacent to the corresponding source image frame.

18. The apparatus according to claim 14, characterized in that, The package assembly unit includes: The third group of sub-units is used to insert the Network Abstraction Layer (NAL) packet of the first resolution image frame into the NAL packet of the corresponding source image frame to form the combined frame.

19. The apparatus according to claim 14, characterized in that, In the video stream, the value of the Type field in the NAL packet of the first resolution image frame is different from the value of the Type field in the NAL packet of the second resolution image frame.

20. An image processing apparatus, characterized in that, The device includes: A processing unit is configured to process the received video stream to obtain a first resolution image frame and a second resolution image frame, wherein the resolution of the first resolution image frame is greater than the resolution of the second resolution image frame, and the number of first resolution image frames is less than the number of second resolution image frames; wherein the second resolution image frame contains a source image frame corresponding to the first resolution image frame, and the first resolution image frame and the corresponding source image frame are generated by encoding the same image; in the video stream, the first resolution image frame and the corresponding source image frame are arranged according to a rule, or the first resolution image frame and the corresponding source image frame are combined into a composite frame; The reconstruction unit is used to reconstruct the second resolution image after decoding the second resolution image frame, using the first resolution image frame decoded as a reference.

21. The apparatus according to claim 20, characterized in that, The processing unit includes: The first processing subunit is configured to, if the received video stream is a first video stream, process the image frames in the first video stream according to the value of the Type field of the NAL packet of the image frame in the first video stream to obtain a first resolution image frame and a second resolution image frame.

22. The apparatus according to claim 21, characterized in that, The first processing subunit is further configured to process the image frame to obtain a first resolution image frame if the value of the Type field of the NAL packet of the image frame in the first video stream is an extended value; and to process the image frame to obtain a second resolution image frame if the value of the Type field of the NAL packet of the image frame in the first video stream is not an extended value.

23. The apparatus according to claim 21, characterized in that, The first processing subunit is further configured to: If the first video stream includes a combined frame, the combined frame is processed to obtain a first resolution image frame and a second resolution image frame corresponding to the first resolution image frame. The second resolution image frame corresponding to the first resolution image frame and the first resolution image frame are obtained by encoding the same image. In this case, the value of the Type field in the NAL packet of the combined frame is the second extended value.

24. The apparatus according to any one of claims 20-23, characterized in that, The reconstruction unit includes: The decoding subunit is used to decode the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image; The first reconstruction subunit is used to input the first resolution image, the image corresponding to the first resolution image, and the second resolution image into the neural network to perform super-resolution reconstruction on the second resolution image. Wherein, the image corresponding to the first resolution image is the image in the second resolution image that corresponds to the first resolution image; the image in the second resolution image that corresponds to the first resolution image is the same image acquired by the first device as the first resolution image; Alternatively, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image.

25. The apparatus according to claim 24, characterized in that, The first reconstruction subunit is also used for: The second resolution image is convolved to obtain the first feature map, and the image corresponding to the first resolution image is convolved to obtain the second feature map; The first feature map and the second feature map are concatenated to obtain a first concatenated feature map; Convolution is performed on the first spliced ​​feature map to obtain the third feature map; Convolve the first resolution image to obtain the fourth feature map; The third feature map and the fourth feature map are concatenated to obtain the second concatenated feature map; The second stitched feature map is convolved to obtain the reconstructed image of the second resolution image.

26. The apparatus according to claim 20, characterized in that, The reconstruction unit also includes: The decoding subunit is used to decode the first resolution image frame and the second resolution image frame respectively to obtain the first resolution image and the second resolution image; The training subunit is used to input the first resolution image and the image corresponding to the first resolution image into the first neural network for training to obtain network parameters; wherein, the image corresponding to the first resolution image is the image corresponding to the first resolution image in the second resolution image; the image corresponding to the first resolution image in the second resolution image is the same image acquired by the first device corresponding to the first resolution image; or, the image corresponding to the first resolution image is an image obtained by downsampling the first resolution image. The inference subunit is used to input the second resolution image into the second neural network for computation and to perform super-resolution reconstruction of the second resolution image. The network parameters of the second neural network are obtained by real-time synchronization based on the network parameters of the first neural network.

27. A terminal device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1-6 or the method of any one of claims 7-13 when executing the instructions.

28. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-6, or the method described in any one of claims 7-13.

Images