Image processing apparatus and multiple frame processing method using the same

Abstract

An image processing apparatus is disclosed according to an embodiment, comprising: a memory storing one or more instructions; and a processor executing one or more instructions stored in the memory, wherein the processor performs the following operations: identifying a predicted sample point in a previous frame corresponding to a current sample point in the current frame; generating a predicted frame for the current frame by changing the sample point values ​​of co-located samples in the previous frame according to the sample point values ​​of the predicted sample points; deriving weights by comparing the sample point values ​​of the current sample point with the sample points of the predicted sample point; applying the weights to co-located samples of the predicted frame to obtain a weighted predicted frame; and obtaining a current output frame by processing the weighted predicted frame and the current frame via a neural network including convolutional layers, wherein the co-located samples of the previous frame and the co-located samples of the predicted frame are co-located samples for the current sample point.

Description

Technical Field

[0001] This disclosure relates to image processing or frame processing, and more specifically, to processing multiple images or multiple frames based on neural networks. Background Technology

[0002] There are various techniques for processing images before they are encoded or displayed. Image processing can be a technique for processing all types of information that are input / output to an image, and refers to techniques for processing and transforming images for better human understanding or secondary applications (such as image enhancement, emphasis, or compression).

[0003] Image processing algorithms have already been developed. With the latest advancements in artificial intelligence, most image processing is now performed using AI. Neural networks are a representative example of AI models.

[0004] Neural networks can be trained using training data. A neural network can then process an image using weights set during training to achieve the desired result. However, to date, neural network-based image processing has proven ineffective when processing multiple temporally related frames. Summary of the Invention

[0005] Technical issues

[0006] The technical goal is to process frames efficiently by taking into account the temporal correlation between multiple frames.

[0007] Technical solutions to the problem

[0008] According to an embodiment of this disclosure, an image processing apparatus includes: a memory configured to store one or more instructions; and a processor configured to: execute the one or more instructions stored in the memory to perform the following operations: identifying a predicted sample point in a previous frame corresponding to a current sample point in a current frame; generating a predicted frame of the current frame by changing the sample point values ​​of co-located samples in the previous frame according to the sample point values ​​of the predicted sample points; deriving weights by comparing the sample points of the current sample points with the sample points of the predicted sample points; applying the weights to co-located samples of the predicted frame to obtain a weighted predicted frame; and obtaining a current output frame by processing the current frame and the weighted predicted frame via a neural network including convolutional layers.

[0009] Advantages of this disclosure

[0010] The image processing apparatus and method for processing multiple frames according to embodiments of the present disclosure can improve the processing performance of the current frame by processing the current frame based on the temporal correlation between the current frame and the previous frame.

[0011] However, the technical effects of the image processing apparatus and the method of processing multiple frames using the image processing apparatus according to the embodiments of this disclosure are not limited thereto, and other unmentioned technical effects will be apparent to those skilled in the art from the following description. Attached Figure Description

[0012] Descriptions of the various figures are provided to provide a full understanding of the figures in this specification.

[0013] Figure 1 This is a diagram illustrating the configuration of an image processing apparatus according to an embodiment.

[0014] Figure 2 This is a diagram illustrating the processing of the current frame according to an embodiment.

[0015] Figure 3 This is a diagram illustrating a convolution operation according to an embodiment.

[0016] Figure 4 It is used to describe Figure 2 A diagram illustrating motion prediction processing.

[0017] Figure 5 This is a diagram illustrating a convolution operation applied to the current frame for motion prediction according to an embodiment.

[0018] Figure 6 This is a diagram illustrating a convolution operation applied to the previous frame for motion prediction according to an embodiment.

[0019] Figure 7 This is a diagram showing the predicted sample points corresponding to the sample points in the current frame according to an embodiment.

[0020] Figure 8 It is used to describe the embodiments. Figure 2 A diagram illustrating motion compensation processing.

[0021] Figure 9 This is a diagram illustrating the process of performing motion compensation on the previous frame using motion prediction results according to an embodiment.

[0022] Figure 10 This is a diagram illustrating the process of applying weights to a predicted frame as a result of motion compensation, according to an embodiment.

[0023] Figure 11 This is a diagram illustrating a method for increasing the number of motion vectors obtained for a downsampled frame according to an embodiment.

[0024] Figure 12 This is a diagram illustrating the structure of a first sub-neural network included in a neural network according to an embodiment.

[0025] Figure 13 This is a diagram illustrating the structure of the last sub-neural network included in the neural network according to an embodiment.

[0026] Figure 14 This is a diagram illustrating an application example of the image processing method according to an embodiment.

[0027] Figure 15 This is a diagram illustrating an application example of an image processing method according to another embodiment.

[0028] Figure 16 This is a diagram illustrating an application example of an image processing method according to another embodiment.

[0029] Figure 17 This is a flowchart illustrating a method for processing multiple frames according to an embodiment. Detailed Implementation

[0030] Best practice

[0031] According to one aspect of this disclosure, an image processing apparatus includes: a memory configured to store one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to perform the following operations: identifying a predicted sample point in a previous frame corresponding to a current sample point in a current frame; generating a predicted frame for the current frame by changing the sample point values ​​of co-located samples in the previous frame according to the sample point values ​​of the predicted sample points, wherein the co-located samples in the previous frame are in the same position as the current sample point; deriving weights by comparing the sample points of the current sample point with the sample points of the predicted sample points; applying the weights to the co-located samples of the predicted frame to obtain a weighted predicted frame, wherein the co-located samples of the predicted frame are in the same position as the current sample point; and obtaining a current output frame by processing the current frame and the weighted predicted frame via a neural network including convolutional layers.

[0032] The processor may be configured to execute one or more instructions to identify the predicted sample in the previous frame by: identifying the most similar sample in the previous frame as the predicted sample, wherein the most similar sample has the sample value most similar to the sample value of the current sample among the sample value of the co-sampling sample in the previous frame and the sample values ​​of the neighboring samples of the co-sampling sample in the previous frame.

[0033] The processor may be configured to execute one or more instructions to identify the most similar sample point of the previous frame as the predicted sample point by: performing convolution processing on the current sample point and the neighboring samples of the current sample point using a predetermined first filter kernel to obtain a first sample point value corresponding to the first filter kernel; performing convolution processing on the co-position sample point of the previous frame and the neighboring samples of the co-position sample point of the previous frame using a plurality of predetermined second filter kernels to obtain a plurality of second sample point values ​​corresponding to the plurality of second filter kernels; identifying the second sample point value most similar to the first sample point value from the plurality of second sample point values; and determining the corresponding sample point among the co-position sample point of the previous frame and the neighboring samples of the co-position sample point of the previous frame that corresponds to the identified second sample point value as the predicted sample point.

[0034] In the first filter core, the first filter sample corresponding to the current sample may have a preset first value, and the remaining first filter samples have a value of 0.

[0035] In the plurality of second filter cores, one second filter sample has a preset second value, and the remaining second filter samples have a value of 0, wherein the position of the one second filter sample is different for each of the plurality of second filter cores.

[0036] The sign of the preset first value can be opposite to the sign of the preset second value.

[0037] The processor may be configured to execute one or more instructions to change the sample value of the co-point of the previous frame by performing convolution processing on the co-point of the previous frame and the adjacent samples of the co-point of the previous frame using a predetermined third filter kernel, wherein, in the third filter kernel, the third filter sample corresponding to the predicted sample has a preset third value, and the remaining third filter samples have a value of 0.

[0038] The weight can be inversely proportional to the difference between the sample value of the current sample point and the sample value of the predicted sample point.

[0039] The processor may also be configured to execute the one or more instructions to perform the following operations: obtaining a previous output frame and a previous feature map as a result of processing the previous frame using the neural network; changing the sample values ​​of the co-position samples of the previous output frame and the previous feature map according to the positional relationship between the current sample and the predicted sample in the previous frame to generate a predicted output frame and a predicted feature map; and applying the weights to the co-position samples of the predicted output frame and the predicted feature map to obtain a weighted predicted output frame and a weighted predicted feature map, wherein the processor may be configured to execute the one or more instructions to process the current frame and the weighted frame by inputting the weighted predicted output frame, the weighted predicted feature map, the weighted predicted frame, and the current frame into the neural network.

[0040] The previous output frame may include a first previous output frame and a second previous output frame output from the neural network, wherein the second previous output frame is obtained by the neural network processing the first previous output frame reconstructed by encoding and decoding the first previous output frame.

[0041] The neural network may include multiple sub-neural networks, wherein the multiple sub-neural networks include a first convolutional layer, a second convolutional layer, and multiple third convolutional layers, wherein the first convolutional layer of the first sub-neural network is configured to perform convolution processing on the result of concatenating the weighted prediction output frame, the weighted prediction frame, and the current frame, wherein the second convolutional layer of the first sub-neural network is configured to perform convolution processing on the weighted prediction feature map, and wherein the multiple third convolutional layers of the first sub-neural network are configured to sequentially perform convolution processing on the result of concatenating the feature map output from the first convolutional layer of the first sub-neural network and the feature map output from the second convolutional layer of the first sub-neural network.

[0042] The first convolutional layer of the sub-neural network other than the first sub-neural network can be configured to perform convolution processing on the result of concatenating the weighted prediction frame, the current frame, and the intermediate output frame output from the previous sub-neural network, wherein the second convolutional layer of the sub-neural network other than the first sub-neural network is configured to perform convolution processing on the intermediate feature map output from the previous sub-neural network, and wherein a plurality of third convolutional layers of the sub-neural network other than the first sub-neural network are configured to sequentially perform convolution processing on the result of concatenating the feature map output from the first convolutional layer of the sub-neural network other than the first sub-neural network and the feature map output from the second convolutional layer of the sub-neural network other than the first sub-neural network.

[0043] The processor may also be configured to execute one or more instructions to send a bit stream generated by encoding the current output frame to the terminal device.

[0044] The processor may also be configured to execute one or more instructions to reproduce the current output frame via a display.

[0045] According to one aspect of this disclosure, a method for processing multiple frames performed by an image processing device includes: identifying a predicted sample point in a previous frame corresponding to a current sample point in the current frame; changing the sample point value of a co-located sample point in the previous frame based on the sample point value of the predicted sample point to generate a predicted frame for the current frame, wherein the co-located sample point in the previous frame is in the same position as the current sample point; deriving weights by comparing the sample point value of the current sample point with the sample point value of the predicted sample point; applying the weights to the co-located sample points of the predicted frame to obtain a weighted predicted frame, wherein the co-located sample points of the predicted frame are in the same position as the current sample point; and processing the current frame and the weighted predicted frame through a neural network including convolutional layers to obtain a current output frame.

[0046] According to one aspect of the invention, a non-transitory computer-readable recording media storage program, wherein, when executed by at least one processor, the program causes the at least one processor to perform the following operations: identifying a predicted sample point in a previous frame corresponding to a current sample point in the current frame; changing the sample point value of a co-located sample point in the previous frame according to the sample point value of the predicted sample point to generate a predicted frame for the current frame, wherein the co-located sample point in the previous frame is in the same position as the current sample point; deriving weights by comparing the sample point value of the current sample point with the sample point value of the predicted sample point; applying the weights to the co-located sample points of the predicted frame to obtain a weighted predicted frame, wherein the co-located sample points of the predicted frame are in the same position as the current sample point; and processing the current frame and the weighted predicted frame by a neural network including convolutional layers to obtain a current output frame.

[0047] Implementation of this disclosure

[0048] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0049] Because this disclosure allows for various modifications and numerous examples, specific embodiments of this disclosure will be shown in the accompanying drawings and described in detail in the written description. However, this is not intended to limit this disclosure to particular practices, and it will be understood that all modifications, equivalents, and alternatives without departing from the spirit and technical scope of the various embodiments of this disclosure are included herein.

[0050] In the description of embodiments of this disclosure, certain detailed explanations of the related technologies are omitted where it is believed that such detailed explanations might unnecessarily obscure the essence of this disclosure. Furthermore, the numbers used in the description of embodiments of this disclosure (e.g., first, second, etc.) are merely identifier codes used to distinguish one element from another.

[0051] Furthermore, it will be understood in this specification that, unless otherwise stated, when elements are “connected” or “combined” with each other, the elements may be directly connected or combined with each other, or indirectly connected or combined with each other, for example, in the presence of an intermediate element.

[0052] In this specification, regarding elements referred to as "units" or "modules," two or more elements may be combined into one element, or one element may be divided into two or more elements according to subdivision functions. Furthermore, each element described below, in addition to its own primary function, may additionally perform some or all of the functions performed by another element, and some of the primary functions of each element may be entirely performed by another component.

[0053] As is conventional in the art, embodiments can be described and illustrated with respect to blocks that perform one or more functions. These blocks (which may be referred to herein as units or modules, or by names such as drivers, controllers, devices, engines, etc.) may be physically implemented by analog or digital circuitry (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuitry, etc.) and may be driven by firmware and software. The circuitry may be embodied, for example, in one or more semiconductor chips or on a substrate support (such as a printed circuit board). The circuitry included in a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware performing some functions of the block and a processor performing other functions of the block. Each block of an embodiment may be physically separated into two or more interactive and discrete blocks. Similarly, the blocks of an embodiment may be physically combined into more complex blocks.

[0054] Furthermore, in this specification, "frame" can refer to a still image. For example, "frame" can include a still image that constitutes a moving image (or video).

[0055] Furthermore, in this specification, "sample point" refers to data assigned to a sampling location in a frame or feature map, i.e., the data to be processed. For example, a sample point can be a pixel value in a frame in the spatial domain. A unit comprising at least one sample point can be defined as a block.

[0056] Furthermore, in this specification, "current sample" may refer to a specific sample included in the current frame to be processed, or a sample to be processed that is included among the samples in the current frame. "Collocated sample" refers to a sample that is located at the same position as the current sample among the samples in frames other than the current frame (e.g., the previous frame, the next frame, the output frame, or the feature map).

[0057] Furthermore, in this specification, "neural network" may refer to an artificial intelligence model that simulates neurons in the brain, and is not limited to neural network models that use specific algorithms.

[0058] Furthermore, in this specification, "weight value" can refer to a value used in the computational processing of each layer of a neural network, and can be used, for example, to apply input values ​​to a specific formula. While weight value is generally used interchangeably with weight, in this disclosure, it is used in conjunction with weight derivation processing (e.g., as described below). Figure 1 The weights derived in the weight derivation process (230) are distinguished from those used in the computational processing of the neural network. The weights used in the computational processing of the neural network are referred to as weight values. The weight values ​​set as training results can be updated as needed using separate training data.

[0059] The embodiments of this disclosure will be described in detail below.

[0060] Figure 1 This is a diagram illustrating the configuration of an image processing apparatus 100 according to an embodiment of the present disclosure.

[0061] The image processing device 100 includes a memory 110 and a processor 130.

[0062] The image processing device 100 can be implemented as a device with image processing capabilities, such as a server, television, camera, mobile phone, tablet computer, or laptop computer.

[0063] Despite Figure 1 The memory 110 and processor 130 are shown separately, but the memory 110 and processor 130 can be implemented by a hardware module (e.g., a chip).

[0064] Processor 130 may be implemented as a dedicated processor for neural network-based image processing. In embodiments, processor 130 may be implemented through a combination of software and a general-purpose processor, such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). The dedicated processor may include memory for executing embodiments of this disclosure, or may include a memory processor for using external memory.

[0065] Processor 130 may include multiple processors. In this case, processor 130 may be implemented by a combination of dedicated processors, or by a combination of software and multiple general-purpose processors (such as an AP, CPU, or GPU).

[0066] Processor 130 may include at least one arithmetic logic unit (ALU) for the convolution operation described below. For the convolution operation, the ALU may include a multiplier that performs multiplication between sample values ​​and an adder that adds the results of the multiplication.

[0067] Memory 110 may store one or more instructions for processing consecutive frames. In embodiments of this disclosure, memory 110 may store a neural network for generating output frames. When the neural network is implemented as a dedicated hardware chip for artificial intelligence or as part of an existing general-purpose processor (e.g., CPU or AP) or a dedicated graphics processor (e.g., GPU), the neural network may not be stored in memory 110.

[0068] Processor 130 obtains consecutive output frames by sequentially processing consecutive frames according to instructions stored in memory 110. The term "consecutive frames" may refer to frames that representatively constitute a moving image. However, consecutive frames in this disclosure do not need to constitute a moving image. In other words, individually captured still images may be processed by image processing device 100 in a predetermined order, an arbitrary order, or an order set by the user.

[0069] As in Figure 1 As shown, the image processing device 100 can sequentially process the first frame X1 to the nth frame X... n Processing is performed to obtain the first output frame Y1 to the nth output frame Y. n In an embodiment, for example, in the description of X... t The index t shown indicates the order in which the image processing device 100 processes the frames.

[0070] Image processing device 100 can obtain the first output frame Y1 to the nth output frame Y through a pre-trained neural network. n Neural networks can be pre-trained to increase resolution, remove noise, increase dynamic range, or improve image quality.

[0071] For example, when a neural network is trained to increase the resolution of frames, the image processing device 100 can process the first frame X1 to the nth frame X based on the neural network. n Processing is performed to obtain a resolution higher than that of the first frame X1 to the nth frame X. n The first output frame Y1 to the nth output frame Y of the resolution nVarious methods exist for training neural networks to increase frame resolution. For example, a neural network can be trained by comparing a training output frame, obtained as a result of processing the training frames, with a labeled frame having a pre-increased resolution to calculate loss information, and minimizing the calculated loss information. As a result of training the neural network, the weight values ​​used in the layers of the neural network can be updated.

[0072] In another example, when training a neural network to remove noise from frames, the image processing device 100 can perform image processing on frames X1 through Xn based on the neural network. n Processing is performed to obtain noise levels less than those in frames X1 to Xn. n The noise from the first output frame Y1 to the nth output frame Y n There are various methods for training a neural network to remove noise from frames. For example, a neural network can be trained by the following steps: calculating loss information by comparing the output frames used for training, obtained as a result of processing the frames used for training, with pre-denoised labeled frames, and minimizing the calculated loss information.

[0073] Neural networks can be trained through supervised learning, unsupervised learning, or reinforcement learning.

[0074] According to embodiments of this disclosure, when processing the current frame, which is the current processing target, within a series of frames, the previous frame is also used. That is, as in... Figure 1 As shown, when the second frame X2 is processed, the first frame X1 is also input to the image processing device 100. As described below, the first output frame Y1 obtained as a result of processing the first frame X1, as well as the feature map obtained during the processing of the first frame X1, can also be input to the image processing device 100 together with the second frame X2.

[0075] The reason for inputting the previous frame into the image processing device 100 when processing the current frame is to take into account the temporal correlation between consecutive frames. When the processing of the current frame reflects the information of the previous frame (e.g., the sample values ​​of the previous frame), better results can be expected than when processing only the current frame based on a neural network.

[0076] However, when using the previous frame as is, the positions of objects included in the current output frame can be incorrect. This is because the positions of objects commonly included in the previous and current frames, captured at different times, may differ from each other. In other words, when shared objects are located at different points in the current and previous frames, the positions of objects included in the previous frame may be reflected during the processing of the current frame, thus the positions of objects included in the current output frame may differ from those in the current frame.

[0077] Furthermore, when an object moves, an object present in the previous frame may be occluded in the current frame. When an object is occluded, this can mean that all or part of an object present in the previous frame is not included in the current frame. For example, an object included in the previous frame might be hidden by another object in the current frame, or the object's image might not be captured by the camera when the current frame is captured. Objects from the previous frame that are occluded in the current frame may not be helpful when processing the current frame.

[0078] Therefore, although the image processing apparatus 100 according to the embodiments of the present disclosure also uses the previous frame during the processing of the current frame in order to take into account the temporal correlation between the current frame and the previous frame, the image processing apparatus 100 may not use the previous frame as is, but instead use a predicted frame generated from the previous frame to process the current frame.

[0079] Furthermore, the image processing device 100 can determine the extent to which samples of the predicted frame will be used in the processing of the current frame, and can gate the predicted frame accordingly.

[0080] In an embodiment, the image processing apparatus 100 may further include a display, or may be connected to a separate display device. At least one of the consecutive output frames generated by the image processing apparatus 100 may be reproduced on the display or display device. If necessary, at least one of the output frames may be post-processed, and then at least one of the post-processed output frames may be reproduced on the display or display device.

[0081] According to an implementation example, the image processing device 100 can encode at least one output frame in the output frames using an image compression method employing frequency conversion. The image compression method employing frequency conversion may include: processing to generate prediction data by predicting the output frame; processing to generate residual data corresponding to the difference between the output frame and the prediction data; processing to transform the residual data, which is a spatial domain component, into a frequency domain component; processing to quantize the residual data transformed into a frequency domain component; and processing to entropy encode the quantized residual data. The image compression method can be implemented using one of the following image processing methods employing frequency conversion: such as MPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, High Efficiency Video Coding (HEVC), VC-1, VP8, VP9, ​​and AOMedia Video 1 (AV1).

[0082] The encoded data or bitstream generated by encoding the output frames can be sent to an external device via a network, or can be stored in a data storage medium (such as a magnetic medium (e.g., a hard disk, floppy disk, or magnetic tape), an optical recording medium (e.g., an optical disc read-only memory (CD-ROM) or a digital versatile optical disc (DVD)) or a magneto-optical medium (e.g., a floppy disk).

[0083] Reference Figure 2 Detailed description of the processing of the current frame X by the image processing device 100 t An example of processing.

[0084] The image processing device 100 obtains data that will be input into the neural network 250 through motion prediction processing 210, motion compensation processing 220, weight derivation processing 230 and gating processing 240.

[0085] First, motion prediction processing 210 determines the current frame X. t The sample points and the previous frame X t-1 The processing of motion vectors between samples. Motion vector indicators exist in the previous frame X. t-1 and current frame X t The relative positional relationship of identical or similar samples in the same frame. For example, when a specific sample is in the previous frame X... t-1 The middle is located at coordinates (a,b) and in the current frame X t When the sample point is located at coordinates (c, d), the motion vector of that sample point can be represented as (ca, db). As described below, in embodiments of this disclosure, the motion vector can be represented as a filter kernel used for convolution operations.

[0086] Image processing device 100 performs motion prediction processing 210 on the previous frame X t-1 Identify and match the current frame X t The predicted sample points correspond to the sample points. Specifically, the image processing device 100 can search the previous frame X... t-1 Which samples are related to the current frame X? t The samples are similar, and the previous frame X can be used as a reference. t-1 The sample points found in the search are identified as the current frame X. t The predicted sample points in the sample points. For example, in the current frame X. t The current sample and the previous frame X t-1 When the sample point to the right of the sample point at the same position as the current sample point (i.e., the co-position sample point) is most similar, the image processing device 100 can identify the sample point to the right of the co-position sample point as the predicted sample point of the current sample point.

[0087] In embodiments of this disclosure, the image processing device 100 can perform motion prediction processing 210 via convolution operations, referring to... Figures 4 to 7 Describe an example.

[0088] In motion compensation processing 220, the image processing device 100 will use the current frame X t The sample points and the previous frame X t-1 In the current frame X t The relative positional relationship between the predicted samples corresponding to the sample points is used as the motion vector.

[0089] Motion compensation processing 220 is achieved by changing the X of the previous frame. t-1 The sample values ​​of the sample points are used to obtain the prediction frame X. t_pred The processing, in which the prediction frame X t_pred It is the current frame X t The predicted version.

[0090] Image processing device 100 can change the previous frame X based on the sample value of the predicted sample. t-1 The sample values ​​are used to obtain the predicted frame X. t_pred Specifically, in the previous frame X t-1 In the current frame X t The sample value of a co-located sample at the same position as the predicted sample can vary based on the sample value of the predicted sample. For example, when located in the current frame X... t The predicted sample of the current sample in the upper left corner is located in the previous frame X. t-1 When the same sample point is to the right of the previous frame X t-1 The sample value of the co-located sample can vary depending on the sample value located to the right (i.e., the sample value of the predicted sample).

[0091] Weight derivation processing 230 can determine the prediction frame X t_pred The samples help to analyze the current frame X. t The degree of processing or prediction frame X t_pred The samples in the current frame X t The degree of contribution in the processing is determined during the processing.

[0092] Through weight derivation, 230 is derived to be applied to the prediction frame X. t_pred The weights of the sample points. In the current frame X... t During processing, high weights are derived for samples with high contribution levels, and low weights are derived for samples with low contribution levels.

[0093] Weights can be based on the current frame X t The difference between the sample value of the current sample point and the sample value of the predicted sample point, where the sample value of the predicted sample point is different from the sample value of the current frame X. tThe sample values ​​correspond to the sample points. As the difference increases, the weight decreases; and as the difference decreases, the weight increases. This is because when the difference is large, it may mean that the predicted sample value corresponds to the current frame X. t When processing the sample points, they do not have a high contribution level, and a small weight is calculated.

[0094] Gating process 240 applies weights to the predicted frame X t_pred Processing of sample points. Predicting frame X t_pred The sample values ​​are based on the predicted frame X. t_pred The contribution of the sample points varies.

[0095] In embodiments of this disclosure, during gating process 240, the weights may be multiplied by the predicted frame X. t_pred The sample value of a sample point. The sample value of a sample point remains unchanged when multiplied by a weight of 1, while the sample value of a sample point decreases when multiplied by a weight less than 1.

[0096] For reference Figure 1 As stated, when for the current frame X t During processing, the image processing device 100 can also use the previous frame X t-1 The corresponding previous output frame Y t-1 And in the previous frame X t-1 The previous feature map S obtained during the processing is shown. t-1 .

[0097] Previous output frame Y t-1 and the previous feature map S t-1 The neural network 250 can be used to process the previous frame X t-1 The output is processed during the processing. The previous output frame Y can be output from the last layer of neural network 250. t-1 Furthermore, the previous feature map S can be output from the previous layer of the last layer of the neural network 250. t-1 The preceding layer of the last layer can refer to the layer directly connected to the last layer, or it can refer to the preceding layer that has one or more layers between the last layer and the preceding layer.

[0098] Because the previous output frame Y t-1 and the previous feature map S t-1 With the previous frame X t-1 Due to its characteristics, motion compensation processing 220 and gating processing 240 can be applied. That is, through motion compensation processing 220, the previous output frame Y can be... t-1 and the previous feature map S t-1 Obtain Y as the current output frame t The predicted output frame Y of the predicted version t_pred And as the current feature map S t The predicted feature map S of the predicted versiont_pred .

[0099] Applied to the previous output frame Y t-1 and the previous feature map S t-1 Motion compensation processing 220 can be applied to the previous frame X t-1 The motion compensation processing is the same as 220. In detail, it can be done based on the current frame X. t The current sample and the previous frame X t-1 The positional relationship (i.e., motion vector) between predicted samples in the previous output frame Y is used to change the previous output frame Y. t-1 and the previous feature map S t-1 The sample values ​​of the corresponding sample points are used to generate the predicted output frame Y. t_pred and predicted feature map S t_pred For example, in the current frame X t The predicted sample of the current sample is located in the previous frame X. t-1 When the same sample point is to the right of the previous output frame Y, t-1 and the previous feature map S t-1 The sample value of the corresponding sample point can vary depending on the sample value located to the right.

[0100] When the weights obtained through weight derivation process 230 are applied to the predicted output frame Y in gating process 240 t_pred and predicted feature map S t_pred When sampling points are selected, the weighted prediction output frame Y' can be obtained. t_pred Weighted prediction feature map S' t_pred .

[0101] Despite in processing Figure 2 The current frame X in t The processing uses the previous output frame Y t-1 and the previous feature map S t-1 Both, but this is just an example, and in the context of the current frame X t The processing can proceed without using the previous output frame Y. t-1 and the previous feature map S t-1 In other words, for the current frame X t In the processing, only the previous frame X can be considered. t-1 In another example, in the current frame X t In the processing, only the previous output frame Y can be used. t-1 and the previous feature map S t-1 one of the.

[0102] The weighted prediction frame X' derived through gating processing 240 t_pred Weighted prediction output frame Y' t_pred Weighted prediction feature map S' t_pred and the current frame Xt It is input into neural network 250. As the result of processing by neural network 250, it obtains the current frame X. t The corresponding current output frame Y t .

[0103] The neural network 250 according to embodiments of the present disclosure may include convolutional layers. In the convolutional layers, convolution processing is performed on the input data using filter kernels. (See also...) Figure 3 An example describing the convolution process of a convolutional layer.

[0104] Neural network 250 may include one or more sub-neural networks, for example, first sub-neural network 260-1, ..., up to nth sub-neural network 260-n. Although in Figure 2 The neural network 250 includes a first sub-neural network 260-1, ..., up to the nth sub-neural network 260-n, but this is just an example, and the neural network 250 may include only one sub-neural network, for example, the first sub-neural network 260-1. When a sub-neural network is included in the neural network 250, it may mean that the neural network 250 includes a fusion layer 262 and multiple convolutional layers.

[0105] The fusion layer is used to combine the current frame X t Compared with the data output from the gated process 240 (i.e., the weighted prediction frame X') t_pred Weighted prediction output frame Y' t_pred Weighted prediction feature map S' t_pred Fusion. Different types of data can be combined through fusion processing.

[0106] The current frame X is then processed by subsequent convolutional layers (CNN) 264 pairs. t With weighted prediction frame X' t_pred Weighted prediction output frame Y' t_pred Weighted prediction feature map S' t_pred The combined result is then subjected to convolution. In an embodiment, when an element is convolved, this may mean performing convolution on the element.

[0107] As the processing result of the first sub-neural network 260-1, the intermediate output frame Y is obtained. t_int and intermediate feature map S t_int Intermediate output frame Y t_int The output of the last layer included in the first sub-neural network 260-1, and the intermediate feature map S t_int The output is from the previous layer, which is included in the last layer of the first sub-neural network 260-1.

[0108] Current frame X t Weighted prediction frame X' t_predAnd the intermediate output frame Y from the first sub-neural network 260-1 t_int and intermediate feature map S t_int It is input into the second sub-neural network 260-2. As in the first sub-neural network 260-1, in the fusion layer 262 of the second sub-neural network 260-2, the current frame X... t Weighted prediction frame X' t_pred Intermediate output frame Y t_int and intermediate feature map S t_int They are combined and then convolved. The intermediate output frame Y... t_int and intermediate feature map S t_int The second sub-neural network 260-2 is used as the processing result output, and the intermediate output frame Y is output. t_int and the intermediate feature map S of the output t_int It is input into the third sub-neural network 260-3. As in the second sub-neural network 260-2, the current frame X... t and weighted prediction frame X' t_pred It can then be further fed into the third sub-neural network 260-3. The final sub-neural network 260-n obtains the current frame X. t The corresponding current output frame Y t As a result of the processing.

[0109] Current frame X t And the current output frame Y output from the last sub-neural network 260-n t and the current feature map S t It can be used for the next frame X t+1 The process is in progress.

[0110] In the current frame X t When it is the first frame of a series of frames, the previous frame X t-1 Previous output frame Y t-1 and the previous feature map S t-1 They can be set to each have a predetermined sample value (e.g., 0).

[0111] Before describing the motion prediction process 210 and the motion compensation process 220 in detail below, an example of the convolution operation will be described.

[0112] Figure 3 It is a diagram used to describe the convolution operation.

[0113] Feature map 350 is generated by multiplication and addition operations between the weight values ​​of filter kernel 330 used in the convolutional layer and the sample values ​​in the corresponding frame 310 that correspond to the weight values ​​of filter kernel 330. Filter kernel 330 has a predetermined size ( Figure 3 (3×3 in the middle).

[0114] The number of feature maps 350 varies depending on the number of filter kernels 330. The number of filter kernels 330 and the number of feature maps 350 can be the same. That is, when one filter kernel 330 is used in a convolutional layer, one feature map 350 is generated, and when two filter kernels 330 are used, two feature maps 350 are generated.

[0115] exist Figure 3 In frame 310, I1 to I49 indicate the samples of frame 310, and F1 to F9 in filter kernel 330 indicate the weight values ​​of filter kernel 330. Furthermore, M1 to M9 in feature map 350 indicate the samples of feature map 350.

[0116] Despite Figure 3 Frame 310 in the example includes 49 samples, but this is just an example. For instance, when Frame 310 has a 4K resolution, it could include 3840 × 2160 samples.

[0117] In the convolution operation, a multiplication operation can be performed between each of the sample values ​​of samples I1, I2, I3, I8, I9, I10, I15, I16, and I17 of frame 310 and each of the weight values ​​F1, F2, F3, F4, F5, F6, F7, F8, and F9 of filter kernel 330, and the value obtained by combining (e.g., adding) the results of the multiplication operation can be assigned as the value of sample M1 of feature map 350. When the stride of the convolution operation is 2, a multiplication operation can be performed between each of the sample values ​​of I3, I4, I5, I10, I11, I12, I17, I18 and I19 of frame 310 and each of the weight values ​​F1, F2, F3, F4, F5, F6, F7, F8 and F9 of filter kernel 330, and the value obtained by combining the results of the multiplication operation can be assigned as the value of sample M2 of feature map 350.

[0118] Because while the filter kernel 330 moves along the stride until it reaches the last sample in frame 310, a convolution operation is performed between the sample value in frame 310 and the weight value of the filter kernel 330, a feature map 350 of a specific size can be obtained.

[0119] Figure 4 It is used to describe Figure 2 The motion prediction processing diagram for 210 is shown.

[0120] In embodiments of this disclosure, the image processing device 100 can process the current frame X based on convolution operations 410 and 420. t And the previous frame X t-1 Perform motion prediction.

[0121] Image processing device 100 can process the current frame X using a predetermined first filter core 415. t Convolution operation 410 is performed to obtain the first feature map 417, and the previous frame X can be processed by using a predetermined number of second filter kernels 425. t-1 Perform convolution operation 420 to obtain multiple second feature maps 427.

[0122] Can be used on the current frame X t The samples are sequentially subjected to convolution operations 410 based on the first filter kernel 415, and the previous frame X can be processed. t-1 The samples are sequentially subjected to convolution operations 420 based on multiple second filter kernels 425.

[0123] When the current frame X t When the current sample and its neighboring samples are subjected to a convolution operation 410 based on the first filter kernel 415, the sample values ​​of the co-position samples of the first feature map 417 can be obtained. Furthermore, when the previous frame X is processed... t-1 When performing convolution operation 420 based on multiple second filter kernels 425 on the co-position sample points and the adjacent sample points of the co-position sample points, the sample point values ​​of the co-position sample points of multiple second feature maps 427 can be obtained.

[0124] Each of the first filter core 415 and the plurality of second filter cores 425 may have a specific size. For example, although each of the first filter core 415 and the plurality of second filter cores 425 may have a size as shown in Figure 4 The size shown is 3×3, but this disclosure is not limited thereto. According to implementation examples, each of the first filter core 415 and the plurality of second filter cores 425 may have a size of 4×4 or 5×5.

[0125] As based on the first filter core 415 for the current frame X t The result of the convolution operation 410 can obtain a value with the same characteristics as the current frame X. t The first feature map 417 is of the same size. The current frame X can be processed by convolution operation 410. t Perform padding to obtain a value similar to the current frame X. t The first feature map 417 is of the same size. Padding refers to the padding in the current frame X. t Processing of assigning samples with predetermined sample values ​​(e.g., 0) to at least one of the left, top, right, or bottom boundaries. Current frame X t The number of sample points is increased by filling.

[0126] In the first filter core 415, the sample corresponding to the current sample may have a preset first value, and the remaining samples may have a value of 0. Although in Figure 4The first filter core 415 has values ​​1 and 0, but 1 is merely an example of the preset first value.

[0127] The first filter core 415 can be applied to the current frame X. t The current sample and its neighboring samples. The sample corresponding to the current sample refers to the sample among the samples of the first filter core 415 that has been multiplied by the current sample. Figure 4 The center sample of the first filter kernel 415 is shown to have a weight value of 1, which takes into account the current frame X. t The case where it is filled in the direction to the left of the left boundary and in the direction above the top boundary. For example, when considering the current frame X... t When performing convolution operation 410 on the upper left sample points, it can be done only in the current frame X. t Multiplication is performed between the upper left sample and the center sample of the first filter kernel 415 only when the area is filled in the left or upper direction. Therefore, in the current frame X t When there is no filling in the left and top directions, the top left sample of the first filter core 415 has a value of 1.

[0128] When the sample corresponding to the current sample in the first filter core 415 has a preset first value, the current frame X is passed. t The sample value of the sample points is multiplied by a first value to calculate the sample points of the first feature map 417. Therefore, when the first value is 1, the sample value of the first feature map 417 is multiplied by the sample value of the current frame X. t The sample point values ​​are the same.

[0129] Based on the implementation example, in motion prediction processing, the step for the current frame X can be omitted. t The convolution operation 410. This is because when the sample corresponding to the current sample in the first filter kernel 415 has a value of 1 and other samples have a value of 0, the first feature map 417 obtained as a result of the convolution operation 410 is compared with the current frame X. t Same. Therefore, it can be accessed via the current frame X. t With regard to the previous frame X t-1 The second feature map 427 obtained by the convolution operation 420 is compared to identify the predicted sample point 430.

[0130] In response to the previous frame X t-1 The convolution operation 420 uses multiple second filter kernels 425, each having a value of 0 and a preset second value. The preset second value can be the same as a preset first value. For example, both the first and second values ​​can be 1. When the first and second values ​​are the same, the first filter kernel 415 can correspond to any one of the multiple second filter kernels 425.

[0131] According to the implementation example, the preset second value can be a value with a different sign than the preset first value. For example, when the first value is 1, the second value can be -1.

[0132] Any one of the samples from the plurality of second filter cores 425 may have a preset second value, and the remaining samples may have a value of 0. The location of the sample with the second value may be different for each of the plurality of second filter cores 425. For example, in Figure 4 As shown, in any one of the second filter cores 425, the upper left sample may have a second value, and in the other second filter core 425, the sample to the right of the upper left sample may have a second value.

[0133] The number of second filter cores 425 can vary depending on their size. When the size of the second filter core 425 is 3×3, the number of second filter cores 425 can be 9. This is because the location of the sample with the second value is different for each second filter core 425.

[0134] This can be achieved by using the second filter core 425 for the previous frame X. t-1 The convolution operation 420 yields a second feature map 427. The number of second feature maps 427 is the same as the number of second filter kernels 425.

[0135] The second filter core 425 can be used to extract the previous frame X. t-1 The sample value is any one of the co-located samples and the neighboring samples of the co-located sample. For example, a second filter kernel with a second value for the upper left sample can be used to extract the sample value located in the previous frame X. t-1 The sample value of the sample point located to the upper left of the corresponding sample point, and the second filter kernel with the second value of the sample point located to the upper right of the corresponding sample point, can be used to extract the sample value located in the previous frame X. t-1 The sample value of the sample point located to the upper right of the corresponding sample point.

[0136] The image processing device 100 can identify the current frame X by comparing the sample values ​​of the first feature map 417 with the sample values ​​of a plurality of second feature maps 427. t The image processing device 100 can identify the sample point 430 most similar to the sample point value at a specific location in the first feature map 417 from multiple sample point values ​​at a specific location in the multiple second feature map 427, and can also use the previous frame X t-1 The sample point corresponding to the identified sample point is identified as the predicted sample point 430 of the current sample point at a specific location.

[0137] For example, when the current sample point is the current frame X tWhen selecting the center sample point in the first feature map 417, the sample value most similar to the center sample point of the first feature map 417 is identified from among the multiple sample values ​​of the center sample points of the multiple second feature maps 427. Previous frame X t-1 The sample point corresponding to the identified sample point value can be identified as the predicted sample point 430 of the current sample point. When the sample point value of the center sample point of the second feature map 427 generated by the second filter kernel 425 based on the sample point with the second value in the upper right corner is most similar to the sample point value of the center sample point of the first feature map 417, it is located in the previous frame X. t-1 The sample point to the upper right of the center sample point can be identified as the predicted sample point 430 of the current sample point.

[0138] Reference Figures 5 to 7 An example describing motion prediction processing.

[0139] Figure 5 This is a diagram showing the convolution operation applied to the current frame 510 for motion prediction.

[0140] In the first filter core 415, the center sample corresponding to the current sample has a value of 1, and the remaining samples have a value of 0.

[0141] The current frame 510 may have sample points a1, b1, c1, d1, e1, f1, g1, h1, and i1. Although in Figure 5 The current frame 510 is 3×3 in size, but this is for ease of interpretation and the size of the current frame 510 can be changed in various ways.

[0142] The current frame 510 can be filled in the left, top, right, and bottom directions to generate a first feature map 417 with the same size as the current frame 510. By filling, samples p0 to p15 with predetermined sample values ​​in the left, top, right, and bottom directions of the current frame 510 can be added to the current frame 510.

[0143] The stride of the convolution operation can be set to 1 so that the convolution operation can be performed sequentially on the samples of the current frame 510.

[0144] First, the sample value of the first sample point (i.e., the upper left sample point) of the first feature map 417 is derived by performing a convolution operation between the weight value of the first filter kernel 415 and the samples p0, p1, p2, p5, a1, b1, p7, d1, and e1 of the current frame 510. Because the center sample point of the first filter kernel 415 has a value of 1 and the remaining samples have a value of 0, the sample value of the upper left sample point of the first feature map 417 is derived as a1.

[0145] The sample value of the second sample point of the first feature map 417 (i.e., the sample point to the right of the sample point located in the upper left corner) is derived by performing a convolution operation between the weight value of the first filter kernel 415 and the samples p1, p2, p3, a1, b1, c1, d1, e1, and f1 of the current frame 510. The sample value of the second sample point of the first feature map 417 is derived as b1 through the convolution operation.

[0146] Perform a convolution operation based on the samples of the current frame 510 and the first filter kernel 415 until the first filter kernel 415 reaches the last sample of the current frame 510 (i.e., sample i1). When the first filter kernel 415 reaches sample i1, the sample value of the last sample of the first feature map 417 is derived as i1 by performing a convolution operation between the weight value of the first filter kernel 415 and the samples e1, f1, p8, h1, i1, p10, p13, p14 and p15 of the current frame 510.

[0147] Reference Figure 5 It was found that when the center sample of the first filter kernel 415 has a value of 1, the sample value of the current frame 510 is the same as the sample value of the first feature map 417. That is, the first filter kernel 415 with a sample value of 1 corresponding to the current sample is used to extract the sample value of the current frame 510.

[0148] Figure 6 This is a diagram showing the convolution operation applied to the previous frame for motion prediction.

[0149] The second filter core 425 may include one sample with a value of 1 and other samples with a value of 0. As described above, the location of the sample with a weight value of 1 may be different for each second filter core 425. The second filter core 425 may contain one sample with a value of -1 and other samples with a value of 0.

[0150] The previous frame 530 may include sample point a2, sample point b2, sample point c2, sample point d2, sample point e2, sample point f2, sample point g2, sample point h2, and sample point i2. Although in Figure 6 The previous frame 530 has a size of 3×3, but this is for ease of explanation, and the size of the previous frame 530 can be changed in various ways.

[0151] The previous frame 530 can be padded in the left, top, right, and bottom directions to generate a second feature map 427 with the same size as the previous frame 530. By padding, samples with predetermined sample values ​​in the left, top, right, and bottom directions of the previous frame 530 can be added to the previous frame 530.

[0152] The second feature map 427 corresponding to the second filter kernel 425 can be obtained by convolution operation based on the previous frame 530 and the second filter kernel 425.

[0153] Based on the location of the sample points with a value of 1, the second filter core 425 can be referred to as second filter core A 425-1, second filter core B 425-2, to second filter core I 425-9 to distinguish the second filter core 425, and the second feature map 427 can be referred to as second feature map A 427-1, second feature map B 427-2, to second feature map I 427-9 to distinguish the second feature map 427.

[0154] The stride of the convolution operation can be set to 1 so that the convolution operation is performed sequentially on the samples of the previous frame 530.

[0155] First, a second feature map A427-1 can be obtained by convolving a second filter kernel A425-1 (based on a sample point with a value of 1 in the upper left corner) with the previous frame 530. As described above, the second filter kernel A425-1 can be convolved with the previous frame 530 while moving according to a stride of 1. The second filter kernel A425-1 is used to extract the sample value located in the upper left corner of the sample point in the previous frame 530. Therefore, the second feature map A427-1 has a value obtained by multiplying the sample value of the sample point located in the upper left corner of the sample point in the previous frame 530 by 1. For example, when the current sample point is a1, the corresponding sample point in the previous frame 530 is a2, and the sample value of the corresponding sample point in the second feature map A427-1 is derived as the sample value p0 located in the upper left corner of sample point a2.

[0156] Next, a second feature map B427-2 is obtained through a convolution operation between the previous frame 530 and a second filter kernel B425-2, which has a value of 1, on the right side of the sample located in the upper left corner. The second filter kernel B425-2 is used to extract the sample values ​​located above the sample in the previous frame 530. Therefore, the second feature map B427-2 has a value obtained by multiplying the sample value of the sample located above the sample in the previous frame 530 by 1. For example, when the current sample is a1, the corresponding sample in the previous frame 530 is a2, and the sample value of the corresponding sample in the second feature map B427-2 is derived as the sample value of p1 located above sample a2.

[0157] In this way, the second feature maps A427-1 to I427-9 can be obtained by convolution operation between the previous frame 530 and the second filter kernels A425-1 to I425-9.

[0158] Figure 7 This is a diagram showing the predicted sample 430 corresponding to the sample in the current frame.

[0159] Image processing device 100 identifies which samples in the second feature map 427 are most similar to samples in the first feature map 417. In this case, samples in the second feature map 427 and samples in the first feature map 417 that are located at the same position are compared. Specifically, image processing device 100 can calculate the absolute value of the difference between the sample value of a sample at a specific position in the first feature map 417 and the sample value of a sample at a specific position in the second feature map 427, and can identify the sample value with the smallest absolute value. Image processing device 100 can determine the sample in the previous frame 530 corresponding to the sample value with the smallest absolute difference as the predicted sample.

[0160] As described above, when the sign of the first value of a sample point in the first filter core 415 is the same as the sign of the second value of a sample point in the second filter core 425, the difference between the sample value of the sample point in the first feature map 417 and the sample value of the sample point in the second feature map 427 can be calculated by difference operation. Conversely, when the signs of the first value and the second value are different from each other, the difference between the sample value of the sample point in the first feature map 417 and the sample value of the sample point in the second feature map 427 can be calculated by summation operation.

[0161] Reference Figure 5 The image processing device 100 can calculate the absolute value of the difference between the sample value of the upper left sample point a1 of the first feature map 417 and the sample value of the upper left sample point of the second feature map 427. For example, it can calculate |a1-p0| between the sample value of the upper left sample point a1 of the first feature map 417 and the sample value of the upper left sample point p0 of the second feature map A427-1, and it can calculate |a1-p1| between the sample value of the upper left sample point a1 of the first feature map 417 and the sample value of the upper left sample point p1 of the second feature map B427-2. When the second feature map 417 is in the upper left corner, the image processing device 100 can calculate the absolute value of the difference between the sample value of the upper left sample point a1 of the first feature map 417 and the sample value of the upper left sample point p1 of the second feature map B427-2. Figure 1 When the absolute value of the difference between the sample value of the upper left sample point e2 of 427-9 and the sample value of the upper left sample point a1 of the first feature map 417 is minimized, the sample value of the second feature map in the previous frame 530 is minimized. Figure 1 The sample point e2 corresponding to the upper left sample point e2 of 427-9 can be determined as the predicted sample point of the upper left sample point a1 of the current frame 510.

[0162] In this way, the predicted sample point 430 corresponding to each sample point of the current frame 510 can be identified by comparing the sample point values ​​of the first feature map 417 with the sample point values ​​of the second feature map 427.

[0163] exist Figure 7 In the process, the predicted sample points corresponding to the sample points of the current frame 510 were identified as b2, e2, f2, e2, f2, i2, h2, e2, and i2.

[0164] Although it has been referenced Figures 4 to 7 The process of determining the predicted sample points describes motion prediction processing, but motion prediction processing can be a process of searching for motion vectors. Each of the second filter kernels 425 can be a positional relationship between the current sample point and the co-located sample point of the previous frame 530, as well as between the current sample point and the neighboring samples of the co-located sample point of the previous frame 530 (i.e., motion vector candidates), that is, motion vector candidates. In other words, motion prediction processing can be a process of determining, from multiple motion vector candidates (multiple second filter kernels), a motion vector candidate (any one of the second filter kernels) that indicates the sample point most similar to the current sample point as the motion vector of the current sample point.

[0165] Figure 8 It is used to describe Figure 2 The motion compensation processing diagram for 220 is shown.

[0166] Motion compensation processing modifies the X-ray image of the previous frame based on the sample values ​​of the predicted samples. t-1 The location of the current frame X t The processing of sample values ​​at the same locations as the sample points. The predicted frame X can be obtained through motion compensation processing. t_pred .

[0167] The image processing device 100 can perform motion compensation processing through convolution operations, such as those in motion prediction processing.

[0168] Image processing device 100 can select from a predetermined plurality of third filter cores 815 which will be used in the previous frame X t-1 The third filter kernel is used for motion compensation of each sample point, and the convolution operation 810 based on the selected third filter kernel can be applied to the previous frame X. t-1 Each sample point. In this case, it is possible to select the previous frame X. t-1 The third filter kernel corresponding to each sample point.

[0169] Multiple third filter cores 815 may include samples with a predetermined third value and samples with a value of 0, and the positions of the samples with the third value may be different for each third filter core 815. The third value may be, for example, 1. According to an implementation example, multiple second filter cores 425 used in motion prediction processing may also be used in motion compensation processing.

[0170] Figure 9 This is a diagram illustrating the process of performing motion compensation on the previous frame using motion prediction results.

[0171] The third filter core 815 is named according to the position of the sample point with the third value, from third filter core A815-1 to third filter core I815-9, in order to distinguish the third filter core 815.

[0172] The image processing device 100 can select a third filter kernel with a third value at the position corresponding to the predicted sample for each sample in the previous frame 530 located at the same position as the sample in the current frame 510.

[0173] First, sample point a2 located in the upper left of the previous frame 530 will be described. When it is determined that the predicted sample point of sample point a1 located in the upper left of the current frame 510 is sample point b2, the image processing device 100 can select a third filter kernel F815-6 for sample point a2, wherein the third filter kernel F815-6 includes a sample point with a value of 1 located to the right of the center sample point and the remaining sample points with a value of 0. In this case, the image processing device 100 can deduce the upper left sample point b2 of the predicted frame 900 by multiplication and summation operations based on the values ​​0, 0, 0, 0, 0, 1, 0, 0 and 0 of the third filter kernel F815-6 and the samples p0, p1, p2, p5, a2, b2, p7, d2 and e2 of the previous frame 530. In other words, it was found that by performing a convolution operation on sample a2 of the previous frame 530 and the adjacent sample a2 of the previous frame 530, sample a2 was replaced by sample b2 in the prediction frame 900.

[0174] Next, when the predicted sample point e2 is determined to be sample point b1 located above the center sample point of the current frame 510, a third filter kernel H 815-8 can be selected for sample point b2. The third filter kernel H 815-8 has samples with a value of 1 located below the center sample point and remaining samples with a value of 0. The image processing device 100 can deduce sample point e2 located above the center sample point of the prediction frame 900 by multiplication and summation operations based on the values ​​0, 0, 0, 0, 0, 0, 0, 1, and 0 of the third filter kernel H 815-8 and the samples p1, p2, p3, a2, b2, c2, d2, e2, and f2 of the previous frame 530. In other words, it was found that by performing a convolution operation on sample b2 of the previous frame 530 and its adjacent samples, sample b2 of the previous frame 530 was replaced by sample e2 in the prediction frame 900.

[0175] Next, when the predicted sample i1 located in the lower right of the current frame 510 is determined to be sample i2, a third filter kernel E 815-5 can be selected for sample i2. The third filter kernel E 815-5 includes a sample with a value of 1 located at the center sample and the remaining samples with a value of 0. The image processing device 100 can deduce the sample i2 located in the lower right of the prediction frame 900 by multiplication and summation operations based on the values ​​0, 0, 0, 0, 1, 0, 0, 0 and 0 of the third filter kernel E815-5 and the samples e2, f2, p8, h2, i2, p10, p13, p14 and p15 of the previous frame 530. In other words, it was found that by performing a convolution operation on sample i2 of the previous frame 530 and its adjacent samples, sample i2 of the previous frame 530 is preserved as sample i2 in the prediction frame 900.

[0176] When a convolution operation is performed on the third filter kernel 815 corresponding to each sample from the first sample to the last sample of the previous frame 530, a prediction frame 900 can be generated as a prediction version of the current frame 510.

[0177] Figure 10 This is a diagram illustrating the process of applying weight 950 to the predicted frame 900 obtained as a result of motion compensation.

[0178] The image processing device 100 can calculate weights 950 based on the difference between the current sample in the current frame 510 and the predicted sample in the previous frame 530 (or the co-position sample in the predicted frame 900). The image processing device 100 can calculate weights 950 for each sample in the current frame 510.

[0179] As described above, weight 950 indicates the degree to which the samples of predicted frame 900 contribute to the processing of the current frame 510.

[0180] The weight 950 can be derived based on Equation 1.

[0181] [Equation 1]

[0182]

[0183] In Equation 1, the predetermined constant σ can be, for example, 16. Referring to Equation 1, it is found that when the sample value of the current sample point is the same as the sample value of the predicted sample point, the weight 950 can be calculated as 1, and as the difference between the sample value of the current sample point and the sample value of the predicted sample point increases, the weight 950 decreases.

[0184] The image processing device 100 can obtain a weighted prediction frame 1000 by multiplying each sample point of the prediction frame 900 by the weight 950 corresponding to that sample point.

[0185] As described above, the image processing device 100 can obtain a weighted prediction output frame and a weighted prediction feature map by applying the corresponding weights 950 to each sample point of the prediction output frame and the prediction feature map.

[0186] As described above, motion prediction and motion compensation processing according to embodiments of this disclosure can be performed based on convolution operations. Because it can be achieved through methods such as... Figure 4 Motion prediction processing can be performed using one convolution operation (when convolution operation 410 for the current frame is omitted) or two convolution operations, as shown, and can be performed via... Figure 8 The convolution operation shown is used to perform motion compensation processing, thus significantly reducing computational complexity.

[0187] Motion prediction processing can also be applied to the downsampled current frame and the downsampled previous frame. This reduces the load and complexity of motion prediction processing. Here, the term "downsampling" refers to the process of reducing the number of samples in a frame. Frame downsampling can be performed using any of a variety of methods. For example, the number of samples in the current and previous frames can be reduced by pooling. Pooling can include max pooling or average pooling. Pooling can be related to pooling layers used in related fields of artificial neural networks. Depending on the implementation example, downsampling of the current and previous frames can be performed using any of the downsampling algorithms of various related techniques.

[0188] When motion prediction processing is performed on the downsampled current frame and the downsampled previous frame, the same number of motion vectors as the samples included in the downsampled current frame are derived. Because the number of motion vectors required in motion compensation processing is greater than the number of motion vectors obtained through motion prediction processing based on the downsampled frame, the number of motion vectors obtained in motion prediction processing can be increased.

[0189] Reference Figure 11 Examples of methods to increase the number of motion vectors obtained in motion prediction processing.

[0190] Figure 11 This is a diagram illustrating a method for increasing the number of motion vectors obtained for a downsampled frame.

[0191] Reference Figure 11 The size of the downsampled frame 1110 is 2×2, and the size of the frame 1130 before downsampling is 4×4. The size of the downsampled frame 1110 can be changed in various ways depending on the downsampling ratio.

[0192] When motion prediction processing is applied to the downsampled frame 1110, the four motion vectors (i.e., filter kernels) corresponding to the four samples included in the downsampled frame 1110 are derived. Because the size of the frame 1130 before downsampling is 4×4, 16 motion vectors are required in the motion compensation processing.

[0193] For example, the image processing device 100 can group the samples of the frame 1130 before downsampling according to the number of samples in the downsampled frame 1110. The image processing device 100 can assign each motion vector derived in the motion prediction processing to each sample group of the frame 1130 before downsampling. In this case, the positions of the sample groups of the frame 1130 before downsampling and the positions of the samples in the downsampled frame 1110 can be taken into account.

[0194] Specifically, the motion vector mv1 derived from the upper-left sample 1112 among samples 1112, 1114, 1116, and 1118 in the downsampled frame 1110 can be assigned to the upper-left sample group 1132 among sample groups 1132, 1134, 1136, and 1138 in the frame 1130 before downsampling. Therefore, motion compensation can be performed on the samples included in the sample group 1132 located in the upper-left of frame 1130 based on the motion vector mv1. Note that motion compensation is performed on the previous frame before downsampling.

[0195] The motion vector mv2 derived from the upper-right sample 1114 among samples 1112, 1114, 1116, and 1118 in the downsampled frame 1110 can be assigned to the upper-right sample group 1134 among sample groups 1132, 1134, 1136, and 1138 in the frame 1130 before downsampling. Therefore, motion compensation can be performed on the samples included in the upper-right sample group 1134 of frame 1130 based on the motion vector mv2.

[0196] If the number of samples included in the sample group is large, applying the same motion vector to all samples included in the sample group can reduce the accuracy of motion compensation.

[0197] In another example, the image processing device 100 may apply a motion vector obtained by combining motion vectors assigned to a group of sample points and motion vectors assigned to adjacent groups of sample points to sample points included in the sample point group that are adjacent to the boundary of the adjacent group of sample points.

[0198] In another example, image processing device 100 can obtain motion vectors for motion compensation of frame 1130 before downsampling by interpolating the motion vectors obtained for downsampled frame 1110. Examples of interpolation may include bilinear interpolation, bicubic interpolation, and nearest neighbor interpolation.

[0199] When motion prediction is performed on the downsampled frame 1110, the number of weights derived in the weight derivation process is less than the number of weights required in the gating process. Therefore, the image processing device 100 increases the number of weights obtained through the weight derivation process. In an embodiment, the gating process can be applied to the prediction frame generated from the previous frame before downsampling through motion compensation processing.

[0200] For example, the image processing device 100 can group the samples of the frame 1130 before downsampling according to the number of samples in the downsampled frame 1110. The image processing device 100 can assign each weight derived in the weight derivation process to each sample group of the frame 1130 before downsampling. In this case, the positions of the sample groups of the frame 1130 before downsampling and the positions of the samples in the downsampled frame 1110 can be taken into account.

[0201] Specifically, a first weight derived from the top-left sample 1112 among samples 1112, 1114, 1116, and 1118 in the downsampled frame 1110 can be applied to the top-left sample group 1132 among sample groups 1132, 1134, 1136, and 1138 in the frame 1130 before downsampling. Therefore, gating processing can be performed on the samples included in the top-left sample group 1132 in frame 1130 based on the first weight. Furthermore, a second weight derived from the upper-right sample 1114 among samples 1112, 1114, 1116, and 1118 in the downsampled frame 1110 can be assigned to the upper-right sample group 1134 among sample groups 1132, 1134, 1136, and 1138 in the frame 1130 before downsampling. Therefore, gating processing can be performed on the samples included in the upper-right sample group 1134 in frame 1130 based on the second weight.

[0202] In another example, the image processing device 100 may apply weights obtained by combining weights assigned to a sample group and weights assigned to an adjacent sample group to the sample points included in the sample group that are adjacent to the boundary of the adjacent sample group.

[0203] In another example, image processing device 100 can obtain weights for gating processing of frame 1130 before downsampling by interpolating the weights obtained for downsampled frame 1110. Examples of interpolation may include bilinear interpolation, bicubic interpolation, and nearest neighbor interpolation.

[0204] Although motion prediction and motion compensation are performed based on convolution operations, this is merely an example. Motion prediction and motion compensation can be performed using algorithms derived from related techniques employed in inter-frame prediction within video codecs.

[0205] For example, motion prediction processing can be performed based on block matching algorithms or optical flow algorithms. Block matching and optical flow are algorithms used to search for the most similar sample or block in the previous frame to the sample or block in the current frame. Motion vectors between the sample or block in the current frame and similar sample or block in the previous frame can be obtained using block matching and optical flow algorithms, and the predicted frame can be obtained by performing motion compensation on the previous frame based on the obtained motion vectors.

[0206] Reference Figure 12 and 13 Describe an example of a neural network used for processing frames.

[0207] For reference Figure 2 As described above, when using neural network 250 to process the current frame X t Weighted prediction frame X' t_pred Weighted prediction output frame Y' t_pred Weighted prediction feature map S' t_pred During processing, the current frame X can be obtained. t The corresponding current output frame Y t .

[0208] The neural network 250 may include one or more sub-neural networks, and each sub-neural network may include a fusion layer and multiple convolutional layers.

[0209] exist Figure 12 The diagram shows an example of the structure of the first sub-neural network 1200 in one or more sub-neural networks.

[0210] Reference Figure 12 The first sub-neural network 1200 may include a fusion layer 1210 and multiple third convolutional layers 1230, wherein the fusion layer 1210 includes a first convolutional layer 1214 and a second convolutional layer 1216. In the convolutional layers, convolution operations are performed on the input data based on filter kernels determined through training.

[0211] Fusion layer 1210 for the current frame X t and the data output through gating processing (i.e., weighted prediction frame X') t_predWeighted prediction output frame Y' t_pred Weighted prediction feature map S' t_pred (to be integrated)

[0212] First, (for example, as shown in cascade 1212) the current frame X is... t Weighted prediction frame X' t_pred and weighted prediction output frame Y' t_pred Cascade, then X the current frame of the cascade t Weighted prediction frame X' t_pred and weighted prediction output frame Y' t_pred Input is fed into the first convolutional layer 1214. Concatenation can be applied to the current frame X along the channel direction. t Weighted prediction frame X' t_pred and weighted prediction output frame Y' t_pred Perform combination processing.

[0213] The data obtained as a result of cascading layer 1212 is convolved in the first convolutional layer 1214. The 3×3×1 in the first convolutional layer 1214 indicates that the input data is convolved using a filter kernel of size 3×3. As a result of the convolutional processing, a feature map is generated by the filter kernel.

[0214] Besides the current frame X t Weighted prediction frame X' t_pred and weighted prediction output frame Y' t_pred The cascaded result is fed into the first convolutional layer 1214, where the weighted predicted feature map S' is used. t_pred It is also fed into the second convolutional layer 1216. In the second convolutional layer 1216, the weighted prediction feature map S' is processed. t_pred Convolution processing is performed. The 3×3×1 in the second convolutional layer 1216 indicates that the input data is convolved using a filter kernel of size 3×3. As a result of the convolution processing, a feature map is generated by the filter kernel.

[0215] (For example, as shown in cascade 1218) data output from the first convolutional layer 1214 and data output from the second convolutional layer 1216 are concatenated, and then the concatenated data output from the first convolutional layer 1214 and data output from the second convolutional layer 1216 are processed sequentially by a plurality of third convolutional layers 1230.

[0216] In this embodiment, in the fusion layer 1210, the current frame X is... t Weighted prediction frame X' t_pred and weighted prediction output frame Y' t_pred The difference lies in the weighted prediction of the feature map S'. t_predIt can be distinguished and fed into the second convolutional layer 1216, for example because of the weighted prediction feature map S' t_pred The domain and the current frame X t Weighted prediction frame X' t_pred and weighted prediction output frame Y' t_pred The domains are different. Because the weighted prediction feature map S' t_pred It is the feature domain data obtained during the processing of frames, and the current frame X t Weighted prediction frame X' t_pred and weighted prediction output frame Y' t_pred It is the image data corresponding to the processing target or the image data obtained as the processing result, so the current frame X t Weighted prediction frame X' t_pred and weighted prediction output frame Y' t_pred The data is differentiated, convolved, and then concatenated. That is, the first convolutional layer 1214 and the second convolutional layer 1216 can be used to process the current frame X. t Weighted prediction frame X' t_pred Weighted prediction output frame Y' t_pred Weighted prediction feature map S' t_pred Domain matching.

[0217] When (for example, as shown in cascade 1218) data output from the first convolutional layer 1214 and data output from the second convolutional layer 1216 are concatenated and then processed sequentially by multiple third convolutional layers 1230, an intermediate output frame Y can be obtained. t_int For example, in Figure 12 As shown, the intermediate output frame Y is output from the last layer 1234 of multiple third convolutional layers 1230. t_int And output the intermediate feature map S from the previous layer 1232 of the last layer 1234. t_int Despite Figure 12 The last layer 1234 is located adjacent to the previous layer 1232, but one or more convolutional layers may be located between the previous layer 1232 and the last layer 1234.

[0218] The 3×3×1 in the third convolutional layer 1230 indicates that the input data is convolved using a filter kernel of size 3×3. As a result of the convolution process, a feature map or an output frame can be generated by the filter kernel.

[0219] Intermediate feature maps S output from multiple third convolutional layers 1230 t_int and intermediate output frame Y t_int It is then fed into the next sub-neural network.

[0220] When the neural network consists of only one sub-neural network, the current output frame Y is output from the last layer 1234 of multiple third convolutional layers 1230. t And output the current feature map S from the previous layer 1232 of the last layer 1234. t .

[0221] Current output frame Y t and the current feature map S t It can be used in the processing of the next frame.

[0222] Figure 12 The structure of the first sub-neural network 1200 is merely an example, and the number of convolutional layers, the size of the filter kernels, and the number of filter kernels included in the first sub-neural network 1200 can be varied in various ways depending on the implementation method.

[0223] Figure 13 This is a diagram showing the structure of the last sub-neural network 1300 included in the neural network.

[0224] Similar to the first sub-neural network 1200, the last sub-neural network 1300 may include a fusion layer 1310 and multiple third convolutional layers 1330, wherein the fusion layer 1310 includes a first convolutional layer 1314 and a second convolutional layer 1316. In the convolutional layers, convolution operations are performed on the input data based on filter kernels determined through training.

[0225] Fusion layer 1310 for the current frame X t Weighted prediction frame X' t_pred The intermediate output frame Y from the previous sub-neural network t_int and the intermediate feature map S output from the previous sub-neural network t_int To integrate.

[0226] First, (for example, as shown in cascade 1312) the current frame X is... t Weighted prediction frame X' t_pred and intermediate output frame Y t_int Cascade, then X the current frame of the cascade t Weighted prediction frame X' t_pred and intermediate output frame Y t_int Input to the first convolutional layer 1314.

[0227] The output data, as a result of cascade 1312, is convolved in the first convolutional layer 1314. The 3×3×1 in the first convolutional layer 1314 indicates that the input data is convolved using a filter kernel of size 3×3. As a result of the convolutional processing, a feature map is generated by the filter kernel.

[0228] Besides the current frame X t Weighted prediction frame X' t_pred and intermediate output frame Y t_int The cascaded result is fed into the first convolutional layer 1314, and the intermediate feature map S t_int It is also input into the second convolutional layer 1316. As described above, the intermediate feature map S is processed in the second convolutional layer 1316. t_int Convolution processing is performed. The 3×3×1 in the second convolutional layer 1316 indicates that the input data is convolved using a filter kernel of size 3×3. As a result of the convolution processing, a feature map is generated by the filter kernel.

[0229] As described above, the first convolutional layer 1314 and the second convolutional layer 1316 in the fusion layer 1310 can be used to process the current frame X t Weighted prediction frame X' t_pred Intermediate output frame Y t_int and intermediate feature map S t_int Domain matching.

[0230] (For example, as shown in cascade 1318) data output from the first convolutional layer 1314 and data output from the second convolutional layer 1316 are concatenated, and then the concatenated data output from the first convolutional layer 1314 and data output from the second convolutional layer 1316 are processed sequentially by multiple third convolutional layers 1330.

[0231] When the data output from the first convolutional layer 1314 and the data output from the second convolutional layer 1316 are concatenated and then processed sequentially by multiple third convolutional layers 1330, the current output frame Y can be obtained. t .

[0232] As in Figure 13 As shown, the current output frame Y is output from the last layer 1334 of multiple third convolutional layers 1330. t And output the current feature map S from the previous layer 1332 of the last layer 1334. t Despite Figure 13 The last layer 1334 is located adjacent to the previous layer 1332, but one or more convolutional layers may be located between the previous layer 1332 and the last layer 1334.

[0233] when Figure 13 When sub-neural network 1300 is not the last sub-neural network, intermediate output frames Y can be output from the last layer 1334 of multiple third convolutional layers 1330. t_int And intermediate feature maps S can be output from the previous layer 1332 of the last layer 1334. t_intThe intermediate output frame Y t_int and the intermediate feature map S of the output t_int It can be input into the next sub-neural network.

[0234] The 3×3×1 in the third convolutional layer 1330 indicates that the input data is convolved using a filter kernel of size 3×3. As a result of the convolution process, a feature map or an output frame can be generated by the filter kernel.

[0235] Figure 13 The structure of the sub-neural network is merely an example, and the number of convolutional layers, the size of the filter kernels, and the number of filter kernels included in the sub-neural network 1300 can be varied in various ways depending on the implementation method.

[0236] Figure 14 This is a diagram illustrating an application example of an image processing method according to an embodiment of the present disclosure.

[0237] Figure 14 The application example shows the process of obtaining an output frame with a higher resolution than the input frame through image processing of the input frame.

[0238] Figure 14 Video Neural Network (VNN) 1400 and Figure 2 The corresponding neural network 250. Assume that motion prediction processing, motion compensation processing, weight derivation processing, and gating processing are performed before the frame is input into the VNN1400.

[0239] When the first frame 1412 is processed by VNN 1400, a first output frame 1432 with a resolution higher than that of the first frame 1412 is obtained. The first frame 1412 and the first output frame 1432 are input to VNN 1400 together with the second frame 1414, and a second output frame 1434 with a resolution higher than that of the second frame 1414 is obtained as the processing result of VNN 1400. The second frame 1414 and the second output frame 143 are input to VNN 1400 together with the third frame 1416, and a third output frame 1436 with a resolution higher than that of the third frame 1416 is obtained as the processing result of VNN 1400.

[0240] When the resolution of frames received from servers, etc., is to be increased, Figure 14 Application examples may be useful. The server sends a low bit rate to the image processing device 100 by encoding low-resolution frames, and the image processing device 100 can obtain a higher-resolution output frame by decoding and processing the low-resolution frames.

[0241] Figure 15This is a diagram illustrating an application example of an image processing method according to another embodiment of the present disclosure.

[0242] Figure 15 The application example illustrates the process of obtaining an output frame by combining the characteristics of the input frame with the image processing of the input frame.

[0243] As described above, it is assumed that motion prediction processing, motion compensation processing, weight derivation processing, and gating processing are performed before the frame is input into the VNN 1500.

[0244] The first frame 1512 is input to VNN 1500, and the result of VNN 1500 processing the first frame 1512, along with the first frame 1512 and the second frame 1514, is input to VNN 1500 again. The processing result of VNN 1500, along with the second frame 1514 and the third frame 1516, is input to VNN 1500 again. As the processing result of VNN 1500, an output frame 1530 reflecting all the characteristics of the first frame 1512, the second frame 1514, and the third frame 1516 is obtained.

[0245] When the dynamic range of frames is to be increased, Figure 15 Application examples can be useful. For instance, when capturing one frame with a long exposure time and another frame with a short exposure time, the feature of including both frames can be used to obtain an output frame with high dynamic range.

[0246] Figure 16 This is a diagram illustrating an application example of an image processing method according to another embodiment of the present disclosure.

[0247] Figure 16 The application example considers the scenario where the image processing device 100 operates as a server or image provider. Typically, the server encodes the image and sends it to a terminal device, and the terminal device decodes the bitstream received from the server and reconstructs the image. When processing frames to compensate for losses that occur during the encoding / decoding process, the image processing device 100 may also use frames encoded by the encoder 120 and then decoded by the decoder 140.

[0248] In detail, the image processing device 100 obtains a first output frame A by processing the first frame 1612 based on a VNN 1600. A first bitstream is generated by encoding the first output frame A, and the first output frame A is reconstructed by decoding the first bitstream. The image processing device 100 obtains a first output frame B by processing the first output frame A using the VNN 1600.

[0249] The first frame 1612, the first output frame A, and the first output frame B are input to the VNN 1600 together with the second frame 1614. Although in the above embodiments of this disclosure, an output frame is input to the VNN 1600 together with the next frame, Figure 16 In the application example, two output frames are input into the VNN 1600 along with the next frame. Motion compensation and gating processes can be applied to the two output frames before they are input into the VNN 1600.

[0250] The VNN 1600 processes the second frame 1614, the first frame 1612, the first output frame A, and the first output frame B to obtain the second output frame A. The second output frame A is encoded to generate a second bitstream, and the second output frame A is reconstructed by decoding the second bitstream. The image processing device 100 obtains the second output frame B by processing the second output frame A and the first output frame B using the VNN 1600. In an embodiment, the image processing device 100 can obtain the second output frame B by processing the first output frame A reconstructed through decoding, as well as the second output frame A and the first output frame B, using the VNN 1600.

[0251] The second frame 1614, the second output frame A, and the second output frame B are input to the VNN 1600 along with the third frame 1616. The VNN 1600 can process the third frame 1616, the second frame 1614, the second output frame A, and the second output frame B, and obtain the third output frame A.

[0252] Figure 17 This is a flowchart illustrating a method for processing multiple frames according to an embodiment of the present disclosure.

[0253] In operation S1710, the image processing device 100 identifies a predicted sample point in the previous frame that corresponds to the current sample point in the current frame. To identify the predicted sample point, motion prediction can be performed on both the current frame and the previous frame. As described above, to identify the predicted sample point, a convolution operation can be performed on both the current frame and the previous frame.

[0254] In operation S1720, the image processing device 100 generates a predicted frame for the current frame by changing the sample values ​​of the corresponding samples in the previous frame based on the sample values ​​of the predicted samples. The predicted frame can be generated by performing a convolution operation on the previous frame based on a filter kernel corresponding to the motion vector.

[0255] In operation S1730, the image processing device 100 derives weights by comparing the sample value of the current sample point with the sample value of the predicted sample point. The image processing device 100 can determine that the weight decreases as the difference between the sample value of the current sample point and the sample value of the predicted sample point increases, and can determine that the weight increases as the difference between the sample value of the current sample point and the sample value of the predicted sample point decreases.

[0256] In operation S1740, the image processing device 100 applies weights to the co-sampling points of the predicted frame. The image processing device 100 can multiply the co-sampling points of the predicted frame by the weights.

[0257] In operation S1750, the image processing device 100 obtains the current output frame by processing the current frame and the weighted prediction frame via a neural network including convolutional layers.

[0258] Embodiments of this disclosure can be implemented as computer-executable programs, and the programs can be stored in machine-readable storage media.

[0259] Machine-readable storage media may be provided as non-transitory storage media. Here, "non-transitory storage media" means that the storage media does not include signals and is tangible, but does not distinguish whether the data is stored in the storage media semi-permanently or temporarily. For example, "non-transitory storage media" may include buffers for temporarily storing data.

[0260] According to embodiments of this disclosure, methods according to various embodiments of this disclosure can be provided in a computer program product. The computer program product is a product that can be purchased between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., CD-ROM) or may be distributed via an app store (e.g., Play Store). TM Online distribution (e.g., download or upload) or direct distribution (e.g., download or upload) between two user devices (e.g., smartphones). When distributed online, at least a portion of the computer program product (e.g., a downloadable application) may be temporarily generated or at least temporarily stored in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a relay server).

[0261] While one or more embodiments of this disclosure have been described with reference to the accompanying drawings, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope defined by the appended claims.

Claims

1. An image processing apparatus, comprising: The memory is configured to store one or more instructions; as well as The processor is configured to execute one or more instructions stored in memory to cause the image processing device to perform the following operations: Identify the predicted sample point in the previous frame that corresponds to the current sample point in the current frame. A predicted frame for the current frame is generated by changing the sample value of the corresponding sample in the previous frame based on the sample value of the predicted sample, wherein the corresponding sample in the previous frame is in the same position as the current sample. Based on the positional relationship between the current sample point and the predicted sample point, a predicted output frame is generated from the previous output frame corresponding to the previous frame. Weights are derived by comparing the sample value of the current sample point with the sample value of the predicted sample point. The weights are applied to the co-location samples of the predicted frame and the predicted output frame to obtain a weighted predicted frame and a weighted predicted output frame. The current output frame is obtained by processing the current frame, the weighted prediction frame, and the weighted prediction output frame through a neural network including convolutional layers. Wherein, the co-position sample points of the prediction frame and the prediction output frame are in the same position as the current sample point.

2. The image processing device according to claim 1, wherein The processor is configured to cause the image processing device to identify the predicted sample point in the previous frame by recognizing the most similar sample point in the previous frame as the predicted sample point, wherein the most similar sample point has the sample value most similar to the sample value of the current sample point among the sample value of the co-located sample point in the previous frame and the sample values ​​of the neighboring sample points of the co-located sample point in the previous frame.

3. The image processing device according to claim 2, wherein The processor is configured to cause the image processing device to identify the most similar sample point of the previous frame as the predicted sample point by performing the following operations: By performing convolution processing on the current sample and its neighboring samples using a predetermined first filter kernel, a first sample value corresponding to the first filter kernel is obtained. By performing convolution processing on the co-position samples of the previous frame and the adjacent samples of the co-position samples of the previous frame using multiple predetermined second filter kernels, multiple second sample values ​​corresponding to the multiple second filter kernels are obtained. Identify the second sample point value from the plurality of second sample point values ​​that is most similar to the first sample point value, and The corresponding sample point that corresponds to the identified second sample point value among the co-position sample points of the previous frame and the adjacent sample points of the co-position sample points of the previous frame is determined as the predicted sample point.

4. The image processing device according to claim 3, wherein In the first filter core, the first filter sample corresponding to the current sample has a preset first value, and the remaining first filter samples have a value of 0.

5. The image processing device according to claim 4, wherein In the plurality of second filter cores, one second filter sample has a preset second value, and the remaining second filter samples have a value of 0. In this context, for each of the plurality of second filter cores, the position of the second filter sample point is different.

6. The image processing device according to claim 5, wherein The sign of the preset first value is opposite to the sign of the preset second value.

7. The image processing apparatus according to claim 3, wherein, The processor is configured to cause the image processing device to change the sample value of the co-point of the previous frame by performing convolution processing on the co-point of the previous frame and its adjacent samples using a predetermined third filter to change the sample value of the co-point of the previous frame. In the third filter kernel, the third filter sample corresponding to the predicted sample has a preset third value, and the remaining third filter samples have a value of 0.

8. The image processing device according to claim 1, wherein The weight is inversely proportional to the difference between the sample value of the current sample point and the sample value of the predicted sample point.

9. The image processing device according to claim 1, wherein The processor is also configured to cause the image processing device to perform the following operations: Obtain the previous feature map as a result of processing the previous frame using the neural network. A predicted feature map is generated by changing the sample values ​​of co-located samples in the previous feature map according to the positional relationship between the current sample and the predicted sample in the previous frame. The weights are applied to the co-location samples of the predicted feature map to obtain a weighted predicted feature map. The processor is configured to cause the image processing device to input the weighted prediction output frame, the weighted prediction feature map, the weighted prediction frame, and the current frame into the neural network.

10. The image processing device according to claim 1, wherein The previous output frame includes a first previous output frame and a second previous output frame output from the neural network, wherein the second previous output frame is obtained by the neural network processing the first previous output frame reconstructed by encoding and decoding the first previous output frame.

11. The image processing device according to claim 9, wherein The neural network includes multiple sub-neural networks, wherein the multiple sub-neural networks include a first convolutional layer, a second convolutional layer, and multiple third convolutional layers. The first convolutional layer of the first sub-neural network is configured to perform convolution processing on the result of concatenating the weighted prediction output frame, the weighted prediction frame, and the current frame. The second convolutional layer of the first sub-neural network is configured to perform convolution processing on the weighted prediction feature map, and The plurality of third convolutional layers of the first sub-neural network are configured to sequentially perform convolution processing on the result of concatenating the feature map output from the first convolutional layer of the first sub-neural network and the feature map output from the second convolutional layer of the first sub-neural network.

12. The image processing device according to claim 11, wherein The first convolutional layer of the sub-neural network other than the first sub-neural network is configured to perform convolution processing on the result of concatenating the weighted prediction frame, the current frame, and the intermediate output frame output from the previous sub-neural network. The second convolutional layer of the sub-neural network, excluding the first sub-neural network, is configured to perform convolution processing on the intermediate feature map output from the previous sub-neural network, and In this configuration, the multiple third convolutional layers of the sub-neural networks other than the first sub-neural network are configured to sequentially perform convolution processing on the result of concatenating the feature maps output from the first convolutional layer of the sub-neural networks other than the first sub-neural network and the feature maps output from the second convolutional layer of the sub-neural networks other than the first sub-neural network.

13. The image processing device according to claim 1, wherein The processor is also configured to cause the image processing device to send a bit stream generated by encoding the current output frame to the terminal device.

14. A method of processing a plurality of frames performed by an image processing device, wherein, The method includes: Identify the predicted sample point in the previous frame that corresponds to the current sample point in the current frame; A predicted frame for the current frame is generated by changing the sample value of the co-position sample of the previous frame according to the sample value of the predicted sample, wherein the co-position sample of the previous frame is in the same position as the current sample. Based on the positional relationship between the current sample point and the predicted sample point, a predicted output frame is generated from the previous output frame corresponding to the previous frame. The weights are derived by comparing the sample value of the current sample point with the sample value of the predicted sample point. The weights are applied to the co-location samples of the predicted frame and the predicted output frame to obtain a weighted predicted frame and a weighted predicted output frame; and The current frame, the weighted prediction frame, and the weighted prediction output frame are processed by a neural network including convolutional layers to obtain the current output frame. Wherein, the co-position sample points of the prediction frame and the prediction output frame are in the same position as the current sample point. 15.A non-transitory computer-readable recording medium storing a program, wherein, When executed by at least one processor, the program causes the image processing device to perform the following operations: Identify the predicted sample point in the previous frame that corresponds to the current sample point in the current frame; A predicted frame for the current frame is generated by changing the sample value of the co-position sample of the previous frame according to the sample value of the predicted sample, wherein the co-position sample of the previous frame is in the same position as the current sample. Based on the positional relationship between the current sample point and the predicted sample point, a predicted output frame is generated from the previous output frame corresponding to the previous frame. The weights are derived by comparing the sample value of the current sample point with the sample value of the predicted sample point. The weights are applied to the co-location samples of the predicted frame and the predicted output frame to obtain a weighted predicted frame and a weighted predicted output frame; and The current frame, the weighted prediction frame, and the weighted prediction output frame are processed by a neural network including convolutional layers to obtain the current output frame. Wherein, the co-position sample points of the prediction frame and the prediction output frame are in the same position as the current sample point.

Images