Image processing method, image processing apparatus, image processing system, and storage medium

Abstract

The invention provides an image processing method, an image processing device, an image processing system and a storage medium. The image processing method comprises the following steps: acquiring a first input frame group and a second input frame group from a moving image; obtaining a first output frame group and a second output frame group through a machine learning model; and acquiring an output moving image frame based on the first output frame and the second output frame. Each of the first input frame and the second input frame includes a first frame and a second frame. The timing of each first frame included in one of the first input frame and the second input frame is different from any timing included in the other of the first input frame and the second input frame. The timing of each second frame included in the one of the first input frame and the second input frame overlaps the timing of one second frame included in the other of the first input frame and the second input frame.

Description

Technical Field

[0001] This disclosure relates to image processing methods, image processing apparatus, image processing systems, and storage media. Background Technology

[0002] Traditional machine learning models can be used to perform image recognition or regression tasks with high accuracy. These models can be applied not only to still images but also to moving images with multiple frames. In moving images, when acquiring the output frame, frame information from moments before and after the output frame can be utilized, allowing for even more precise processing. U.S. Patent Application Publication No. 2023 / 019679 discloses a processing method that uses a machine learning model to upgrade the input frame by propagating feature maps of the input frame from moments before and after each output frame. Summary of the Invention

[0003] One aspect of this disclosure provides an image processing method, the method comprising: acquiring a first input frame group comprising a plurality of consecutive first input frames from a moving image; acquiring a first output frame group comprising a plurality of first output frames, the first output frame group being output by a machine learning model that has received and processed the first input frame group; acquiring a second input frame group comprising a plurality of consecutive second input frames from the moving image; acquiring a second output frame group comprising a plurality of second output frames, the second output frame group being output by a machine learning model that has received and processed the second input frame group; and acquiring an output moving image frame based on the plurality of first output frames and the plurality of second output frames. Each input frame in the plurality of first input frames and the plurality of second input frames comprises one or more first frames and one or more second frames, wherein the time of each first frame contained in one of the plurality of first input frames and the plurality of second input frames is different from any time contained in another frame in the plurality of first input frames and the plurality of second input frames, and the time of each second frame contained in the one frame in the plurality of first input frames and the plurality of second input frames overlaps with the time of a second frame contained in the other frame in the plurality of first input frames and the plurality of second input frames. Image processing systems and image processing apparatuses that utilize the above-described image processing methods also constitute another aspect of this disclosure.

[0004] Further features of various embodiments of this disclosure will become clear from the following description of embodiments with reference to the accompanying drawings. Attached Figure Description

[0005] Figure 1 This is a block diagram of an image processing system according to the first embodiment.

[0006] Figure 2 This is an external view of the image processing system according to the first embodiment.

[0007] Figure 3 The training process according to the first embodiment is shown.

[0008] Figure 4 The training process of the neural network according to the first embodiment is shown.

[0009] Figure 5 The training process of the neural network according to the first embodiment is shown.

[0010] Figure 6 The training process of the neural network according to the first embodiment is shown.

[0011] Figure 7 An example of acquiring an upscaled motion image is shown in the first embodiment.

[0012] Figure 8 This is a flowchart illustrating the generation of the upgraded motion picture in the first embodiment.

[0013] Figure 9 An example of the first input frame group and the second input frame group in the first embodiment is shown.

[0014] Figure 10 An example of acquiring the output motion image frame is shown in the first embodiment.

[0015] Figure 11 This is a block diagram of an image processing system according to the second embodiment.

[0016] Figure 12 This is an external view of the image processing system according to the second embodiment.

[0017] Figure 13 This is a flowchart illustrating the generation of the upgraded motion picture in the second embodiment.

[0018] Figure 14 An example of acquiring the output motion image frame is shown in the second embodiment.

[0019] Figure 15 This is a block diagram of an image processing system according to a third embodiment.

[0020] Figure 16 This is a flowchart illustrating the generation of an upgraded motion picture according to a third embodiment. Detailed Implementation

[0021] In the following text, the term "unit" can refer to a software environment, a hardware environment, or a combination of both. In a software environment, the term "unit" refers to a function, application, software module, function, routine, instruction set, or program that can be executed by a programmable processor, such as a microprocessor, central processing unit (CPU), or a specially designed programmable device or controller. Memory contains instructions or programs that, when executed by the CPU, cause the CPU to perform operations corresponding to the unit or function. In a hardware environment, the term "unit" refers to a hardware element, circuit, component, physical structure, system, module, or subsystem. According to a specific embodiment, the term "unit" can include mechanical, optical, or electronic components, or any combination thereof. The term "unit" can include active components (e.g., transistors) or passive components (e.g., capacitors). The term "unit" can include a semiconductor device having a substrate and other layers of material with varying concentrations of conductivity. It can include a CPU or programmable processor capable of executing a program stored in memory to perform a specified function. The term "unit" can include logic elements (e.g., "AND" or "OR") implemented by transistor circuitry or any other switching circuitry. In the combination of software and hardware environments, the term "unit" or "circuit" refers to any combination of the aforementioned software and hardware environments. Additionally, the terms "component," "assembly," "part," or "device" can also refer to a "circuit" with or without integrated packaging materials.

[0022] Embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Corresponding elements in the various figures will be designated by the same reference numerals, and repeated descriptions thereof will be omitted.

[0023] First, an overview of the various embodiments will be described. Each embodiment uses a machine learning model to generate an upgraded motion picture with multiple upgraded output frames from a motion picture having multiple consecutive input frames.

[0024] Machine learning models include, for example, neural networks, genetic programming, and Bayesian networks. Neural networks include convolutional neural networks (CNNs), generative adversarial networks (GANs), and recurrent neural networks (RNNs).

[0025] Upgrading is an image upscaling process that generates a sharp, high-resolution image by estimating high-frequency components that cannot be expressed in low-resolution images with few pixels.

[0026] Although upgrades have been given as examples, the image processing according to the following embodiments is also applicable to image processing such as sharpening and noise reduction.

[0027] Each embodiment is able to provide high-quality motion pictures while suppressing the impact of output frames that result in image quality degradation due to the inability to fully utilize information about the previous (or last) and next (or next) frames.

[0028] In the following description, the phase of learning (or training) the weights of the machine learning model will be referred to as the training (learning) phase, while the phase of upgrading using the machine learning model and the trained weights will be referred to as the estimation phase.

[0029] The image processing apparatus according to the various embodiments can be any apparatus as long as it has the image processing function of this disclosure, and can be implemented in the form of a camera device (e.g., a camera) or a PC.

[0030] First Embodiment

[0031] This embodiment will discuss a method for upgrading captured images using machine learning models.

[0032] Figure 1 This is a block diagram of the image processing system 100 according to this embodiment. Figure 2 This is an external view of the image processing system 100. The image processing system 100 includes a training (learning) device 101, a camera device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, and a network 107.

[0033] The training device 101 is an image processing apparatus that performs training processing, and includes a memory 101a, an acquisition unit 101b, a generator 101c, and an updater 101d. The acquisition unit 101b acquires a series of training images and a series of corresponding ground truth images. The generator 101c inputs the training images into a multi-layer neural network to generate a series of output images. The updater 101d updates the network parameters of the neural network based on the error between the output images calculated by the generator 101c and the ground truth images. Details of the training process will be described later using flowcharts. The trained network parameters are stored in the memory 101a.

[0034] The imaging device 102 includes an optical system 102a and an image sensor 102b. The optical system 102a converges light rays incident from the subject space onto the imaging device 102. The image sensor 102b receives (photoelectrically converts) the optical image (subject image) formed by the optical system 102a to obtain a captured image. The image sensor 102b is, for example, a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor. The captured image and the captured motion image acquired by the imaging device 102 contain blurring due to aberrations and diffraction of the optical system 102a, as well as noise generated by the image sensor 102b.

[0035] Image estimation device 103 is a device for performing estimation processing and includes a memory 103a, an acquisition unit 103b, and a corrector 103c. Image estimation device 103 may include at least one processor for executing instructions. Image estimation device 103 performs upgrade processing on a captured motion image including multiple acquired images to generate an output motion image. A multi-layer neural network is used for upgrade, and network parameter information is read from memory 103a. The network parameters are trained by training device 101, and image estimation device 103 reads the network parameters from memory 101a beforehand via network 107 and then stores them in memory 103a. The stored network parameters themselves can be in numerical form or encoded format. Details regarding the training of network parameters and the upgrade using network parameters will be described later.

[0036] The output motion image is sent to at least one of a display device 104, a recording medium 105, and an output device 106. The display device 104 is, for example, a liquid crystal display or a projector. The user can view the motion image while performing editing tasks via the display device 104 during processing. The recording medium 105 is, for example, a semiconductor memory, a hard disk drive, or a server on a network. The output device 106 is, for example, a printer. The image estimation device 103 has the function of performing development processing and other image processing as needed.

[0037] Now refer to Figure 3 and Figure 4 The weight (weight information) training method (method for generating a trained model) executed by the training device 101 according to this embodiment will be described. Figure 3 This illustrates the process of training weights. Figure 3 The steps in the process are mainly executed by the acquisition unit 101b, generator 101c or updater 101d in the training device 101. Figure 4 This illustrates the process of training the weights of a neural network (machine learning model).

[0038] In step S101, the acquisition unit 101b acquires an original moving image containing multiple original still images (subject images). In this embodiment, the original moving image is a moving image containing high-resolution (high-quality) original still images with almost no blurring due to aberrations or diffraction of the optical system 102a. Multiple original moving images are acquired. The acquired moving image has images containing various subjects (i.e., edges, textures, gradients, flat parts, etc., of various intensities and directions). Various motions caused by the movement of the viewpoint and the subject are contained between the multiple original still images in the original moving image. The original still images and the original moving images can be real-life images or images generated by computer graphics (CG).

[0039] The signal values ​​of the original still image and the original moving image can be higher than the brightness saturation value of the image sensor 102b. This is because, even in actual subjects, some subjects may exceed this brightness saturation value when the imaging device 102 captures images under specific exposure conditions. The original still image and the original moving image are generated by downscaling the original still image and clipping the signal using the brightness saturation value of the image sensor 102b. In particular, when using an actual image as the original still image, blurring has already occurred due to aberrations and diffraction; therefore, by downscaling the image, the effect of blurring can be reduced, and a high-resolution (high-quality) image can be obtained. If the original still image contains sufficient high-frequency components, downscaling is unnecessary. The original still image may also contain noise components. In this case, the noise contained in the original still image can be considered as a subject, so the noise in the original still image is not particularly problematic.

[0040] In step S102, generator 101c generates a real block (real data) containing multiple consecutive images and a training block (training data) containing multiple consecutive images corresponding to the real block. Multiple real blocks and training blocks are generated, and one or more blocks are generated corresponding to an original moving image. In this embodiment, the real block and training block are multiple consecutive images reflecting the same subject. This embodiment uses multiple combinations, each combination containing a set of real blocks and training blocks as training data. A block refers to multiple images having a predetermined number of pixels (e.g., 64×64 pixels, etc.) and a predetermined number of frames (e.g., 10 frames, etc.).

[0041] This embodiment uses mini-batch training to train the weights of a multi-layer neural network. Therefore, in step S102, multiple sets of real blocks and training blocks are generated. However, this disclosure is not limited to this example, and online training or batch training can be used. In this embodiment, the original still image, the original moving image, the real blocks, and the training blocks can be undeveloped images (original images) or developed images. However, when training using original images, the original images are also input during estimation, and when training using developed images, the developed images are also input during estimation.

[0042] In step S103, generator 101c will Figure 4 Training blocks (training data) 212 containing multiple consecutive images are input into a multi-layer neural network to generate estimation blocks (estimation data) 213 containing multiple consecutive images. For mini-batch training, estimation blocks 213 corresponding to the multiple training blocks 212 are generated. Figure 4 The flow from step S103 to step S104 is shown. The estimated block 213 has a larger number of pixels (higher sharpness) than the training block 212, and ideally coincides with the real block (real data) 211. This embodiment uses... Figure 4 The neural network structure shown. Figure 4 In this diagram, CN represents a convolutional layer, which computes the convolution of the input with the filter and the sum of the input and the bias, and performs a nonlinear transformation on the result using an activation function. The initial values ​​of the filter components and the bias are arbitrary and, in this embodiment, determined by random numbers. The activation function can be, for example, a rectified linear unit (ReLU) or a sigmoid function. Although convolutional layers are used in the construction of neural networks, this disclosure is not limited to this example, and residual blocks, etc., can be used instead of convolutional layers.

[0043] Except for the final layer, the output of each layer is called a feature map. For each training block in the multiple training blocks 212, an estimation block 213 is generated. The propagation 222 of the previous time step and the propagation 223 of the next time step combine the feature maps output from the intermediate layers in the previous or next time step. These feature maps are combined by connecting them in the channel direction, but they can also be added or weighted by addition. Figure 4 The example illustrates propagation at times t-1 and t+1 before and after time t. However, this embodiment is not limited to this example, and propagation at more distant times, such as t-2 and t+2 before and after time t, can also be added. Propagation at the later time (t+1) occurs after the propagation at the previous time (t-1). However, this embodiment is not limited to this example, and propagation at the later time can occur first, or propagation at the previous and later times can occur simultaneously. Propagation at the previous and later times each occurs once, but multiple times are also possible.

[0044] Here, due to the motion of the viewpoint and the subject, the feature maps will shift at one time step or the next. Therefore, the feature maps can be aligned and combined at one time step or the next. Various alignment methods can be used, such as those using optical flow and deformable convolution. Alignment using optical flow can include separate steps of obtaining the optical flow between time steps from multiple consecutive images in the training block.

[0045] This embodiment uses Figure 4 The construction of a neural network was discussed using an example, but it is not limited to this example. Various variations can be applied as long as a training block with multiple consecutive images is input and an estimated block with corresponding consecutive images is output. Information from at least one time step before and after each image's time step can be used to obtain the images of the estimated block. This information can be the feature map described above, a feature map obtained using the input image, or an image (frame). When using images as before and after time step information, the image obtained by combining (concatenating or adding) the before and after images can be input into the convolutional layer.

[0046] When training a machine learning model that performs upscaling, the sizes of the output and ground truth blocks change according to the upscaling ratio (magnification). The upscaling ratio refers to the vertical and horizontal magnification when the image is enlarged. At an upscaling ratio of 2, the sizes of the output and ground truth blocks are twice the size of the training blocks (twice the vertical and horizontal dimensions, and four times the number of pixels).

[0047] like Figure 5 As shown, the neural network can be constructed to output an estimated block of the same size, in which the degradation caused by interpolation is corrected, by processing an image obtained by upscaling the training block by an upscaling factor through interpolation, and setting this image as the training block. Skip connection 221 calculates the sum of the residuals estimated from the training block 212 and the ground truth block 211 with the training block 212 to generate the estimated block 213. During skip connection 221, the feature maps are combined and the element-wise sum is calculated. Figure 6 As shown, a neural network can be constructed to process an image obtained by upscaling the training block by an upgrade factor through interpolation in a convolutional layer, and output an estimated block of size equal to the upgrade factor of the training block.

[0048] In step S104, updater 101d updates the weights (weight information) of the neural network based on the error between the estimated block 213 and the real block (real data) 211. Here, the weights include filter components and the bias of each layer. Backpropagation is used to update the weights, but this embodiment is not limited to this. For mini-batch training, the error between multiple real blocks 211 and their corresponding estimated blocks 213 is obtained, and the weights are updated. For the loss function, for example, the L2 norm or L1 norm can be used.

[0049] In step S105, the updater 101d determines whether weight training is complete. This can be determined based on whether the number of training (weight update) iterations has reached a specified value, or whether the weight change during the update period is less than a specified value. If training is not yet complete, the process returns to step S102 and acquires multiple new real blocks and training blocks. If training is complete, the training device 101 (updater 101d) ends training and stores the weight information in memory 101a.

[0050] The generation (upgrade process) of the upgraded motion picture performed by the image estimation device 103 according to this embodiment will now be described.

[0051] Before proceeding with a detailed description of the upgrade process, the effects of this disclosure will now be described. Figure 7 An example is shown of using a machine learning model to upgrade a motion picture to obtain an upgraded motion picture. When upgrading the motion picture, each input frame group 231, which has multiple input frames as part of the motion picture, is upgraded. By repeating this process, an output motion picture frame 233 (the upgraded motion picture) containing an output frame group 232 has multiple upgraded output frames.

[0052] In this embodiment, the input frame group 231 has 10 frames, and when acquiring each output frame in the output frame group 232, information from four time points—the first two time points and the next two time points—is used. Now, let's focus on the output frame group 232. Output frames located at the edges of time points where information from preceding and following time points is not fully utilized (shown in shaded boxes) have lower image quality than other output frames. When output motion picture frames are acquired by concatenating output frame groups containing output frames that do not fully utilize information from preceding and following time points, image quality differences arise between these frames, and the image quality of the motion picture is also reduced. Therefore, this embodiment performs an upgrade process when some frames from each input frame group overlap, and constructs output motion picture frames. Thus, while suppressing the impact of reduced output frame image quality due to insufficient utilization of preceding and following frames, a high-quality upgraded motion picture is acquired.

[0053] Now refer to Figure 8 This describes the generation (upgrade process) of an upgraded motion image performed by the image estimation device 103 according to this embodiment. Figure 8 This is a flowchart illustrating the generation of the upgraded motion picture. Figure 8 The steps in the process are mainly performed by the acquisition unit 103b and the corrector 103c in the image estimation device 103.

[0054] In step S111, the acquisition unit 103b acquires the captured motion image and weight information. Similar to training, the captured motion image is a motion image containing either an undeveloped original image or a developed image, and in this embodiment, the captured motion image is sent from the camera device 102. The weight information is the weights of the machine learning model sent from the training device 101 and stored in the memory 103a.

[0055] In step S112, the acquisition unit 103b acquires an input frame group (first input frame group) containing multiple consecutive input frames from the captured motion image. In this embodiment, an input frame group containing 10 frames is acquired.

[0056] In step S113, the corrector 103c performs upgrade processing on the first input frame group based on the weights of the acquired machine learning model, and obtains an output frame group (first output frame group) containing multiple upgraded output frames. In this embodiment, the input frame group containing 10 frames is upgraded, and an output frame group containing 10 upgraded frames is obtained.

[0057] In step S114, the acquisition unit 103b acquires an input frame group (second input frame group) containing multiple consecutive frames from the captured motion image. Here, the first input frame group and the second input frame group contain at least one frame at a different time, and contain at least one frame at an overlapping time (such that the times overlap).

[0058] Here, the number of overlapping frames between the first and second input frame groups is set to be equal to or greater than the number of consecutive moments used in acquiring each output frame in the machine learning model. The number of overlapping moments between the first and second input frame groups can be more than twice the number of consecutive moments used in acquiring each output frame in the machine learning model. In acquiring the output frame, this embodiment uses information from four moments: the first two moments and the last two moments, and the number of overlapping moments is four, that is, twice the number of either the first or last two moments. Due to this configuration, subsequent steps can provide output motion picture frames solely from high-quality output frames acquired by fully utilizing the preceding and following frames.

[0059] Figure 9Examples of a first input frame group and a second input frame group are shown. This embodiment illustrates an example where the first input frame group 234 and the second input frame group 235 have 10 frames and the overlapping frame 236 has 4 frames. Figure 9 In this embodiment, the values ​​written in each block corresponding to each input frame in each input frame group represent the number of preceding and following time points used to acquire the corresponding output frame. When acquiring the output frame, this embodiment uses information from four time points: the first two and the last two. This allows for the acquisition of the most accurate output frame in one go, utilizing information from all four time points. Compared to other time points, the output frames corresponding to the first two and last two frames in the first input frame group 234 and the second input frame group 235 cannot fully utilize the information from preceding and following time points, resulting in lower image quality compared to output frames from other time points.

[0060] In step S115, the corrector 103c performs an upgrade process based on the weights of the machine learning model obtained in step S111, and obtains an output frame group (second output frame group) containing multiple upgraded output frames.

[0061] In step S116, the corrector 103c connects the first input frame group and the second input frame group and acquires an output motion picture frame. As described above, the first input frame group and the second input frame group include frames with overlapping times. Therefore, for each output frame corresponding to a frame with an overlapping time, the output frame with that overlapping time is excluded from one of the first output frame group and the second output frame group and connected to acquire an output motion picture frame. Here, among the output frames corresponding to the overlapping times in the first input frame group and the second input frame group, output frames with a small number of preceding and following times used when acquiring each output frame are excluded.

[0062] Figure 10 An example is shown of obtaining output motion picture frames 239 from the first output frame group 237 and the second output frame group 238. Similarly, in... Figure 10 In this embodiment, the values ​​in each block corresponding to each output frame in each output frame group represent the number of preceding and following moments used when acquiring each output frame. In order to exclude output frames with a small number of preceding and following moments from the output frames at overlapping moments, output frames with 2 and 3 preceding and following moments respectively are excluded and concatenated to obtain output motion picture frame 239. Thus, high-quality output motion picture frames can be constructed without using output frames that degrade image quality due to insufficient utilization of preceding and following moment information.

[0063] Although this embodiment excludes the output frames at overlapping times from the first output frame group and the second output frame group, the machine learning model can still output the output frame group in which the output frames at overlapping times are excluded.

[0064] In step S117, if there are unprocessed frames in the frames of the captured motion image or in the frames to be processed that are part of the captured motion image, the process returns to step 114 to obtain and upgrade the subsequent unprocessed input frame group. In this case, the output motion image frame obtained in step S116 is set as the first output frame group, and the output frame group obtained through the new upgrade process is set as the second output frame group. Similarly, in step S116, output motion image frames are obtained through connection.

[0065] In step S117, if there are no unprocessed frames in the frames of the captured motion image or in the frames to be processed that are part of the captured motion image, the process ends and the output motion image frame is obtained as the output motion image (upgraded motion image).

[0066] The above processing can provide upgraded motion pictures with high quality while suppressing the impact of output frames that degrade image quality due to the inability to fully utilize information from previous and subsequent moments.

[0067] Second Embodiment

[0068] This embodiment discusses a construction for generating upgraded moving images using an image estimator in a camera device. The difference between this embodiment and the first embodiment lies in the upgraded moving image generation process. This embodiment obtains the output moving image frames by calculating the weighted average of the output frames at overlapping times in each output frame group. This embodiment only discusses the construction different from the first embodiment and omits descriptions of similar constructions.

[0069] Figure 11 This is a block diagram of the image processing system 300 according to this embodiment. Figure 12 This is an external view of the image processing system 300. The image processing system 300 includes a training device (image processing device) 301 and a camera device 302 connected via a network 303.

[0070] The training device 301 includes a memory 311, an acquisition unit 312, a generator 313, and an updater (training unit) 314, and uses a neural network to train weights (weight information) for upgrading.

[0071] The camera device 302 captures images of the subject space, obtains moving images, and generates an upgraded moving image from the captured moving images using read weight information. The camera device 302 includes an optical system 321 and an image sensor 322. The image estimator 323 has an acquisition unit 323a and a corrector 323b, and performs upgrade processing on the captured moving images using weight information stored in a memory 324.

[0072] The weight information has been trained by the training device 301 and stored in the memory 311. The camera device 302 reads the weight information from the memory 311 via the network 303 and stores it in the memory 324. The captured motion image and the upgraded motion image are stored in the recording medium 325. When the user issues a command to display the upgraded motion image, the stored upgraded motion image is read out and displayed on the display unit 326. The image estimator 323 can read out and upgrade the captured motion image stored in the recording medium 325. The above series of controls are performed by the system controller 327.

[0073] The training method of the machine learning model executed by the training device 301 according to this embodiment is similar to that in the first embodiment, so its description will be omitted.

[0074] Now refer to Figure 13 This describes the enhancement processing of the moving image performed by the image estimator 323 according to this embodiment. Figure 13 This is a flowchart illustrating the generation of the upgraded motion picture. Figure 13 The steps in this process are mainly performed by the acquisition unit 323a and the corrector 323b in the image estimator 323. The steps other than step S216 are similar to the steps other than step S216 performed by the acquisition unit 103b and the corrector 103c in the first embodiment, and therefore their description will be omitted.

[0075] In step S216, the corrector 323b connects the upgraded first output frame group and the second output frame group, and acquires an output motion picture frame. As described in the first embodiment, since the first input frame group and the second input frame group contain frames with overlapping times, the output motion picture frame is acquired by connecting the output frames at the overlapping times using a weighted average. Here, at the overlapping times between the first input frame group and the second input frame group, the weighted average can be performed by reducing the weight of the output frames that have less information about the preceding and following times when acquiring each output frame.

[0076] Figure 14 An example is shown of obtaining an output motion picture frame 339 from a first output frame group 237 and a second output frame group 238. This embodiment performs a weighted average by reducing the weight of output frames with less prior and subsequent time information that were already used when acquiring each output frame at the overlapping time. For example, the weights of the first output frame group 237 are shown as reference numeral 337, and the weights of the second output frame group 238 are shown as reference numeral 338. The output frames at the overlapping time are weighted, averaged, and concatenated to obtain the output motion picture frame 339.

[0077] Here, the weights of the weighted average can be determined based on the number of preceding and following time points used when acquiring each output frame. Similarly, in Figure 14 In this context, the values ​​written in the blocks corresponding to each output frame in each output frame group represent the number of time points before and after each output frame has been acquired. For example, such as... Figure 14 As shown, when the amount of information between two time steps is 2, the weight is set to 0.2, and when the amount of information between three time steps is 3, the weight is set to 0.4. However, this embodiment is not limited to this example. Therefore, it is possible to obtain high-quality upgraded motion images while suppressing the impact of reduced output frame image quality caused by incomplete utilization of preceding and following frames.

[0078] Third Embodiment

[0079] The image processing system according to this embodiment differs from the first and second embodiments in that it has a processing device (computer) configured to send a captured motion image, which is the target of image processing, to an image estimation device, and receive a processed output image (upgraded image) from the image estimation device. This embodiment only discusses configurations different from the first embodiment and omits descriptions of similar configurations.

[0080] Figure 15 This is a block diagram of an image processing system 600 according to this embodiment. The image processing system 600 includes a training device 601, a camera device 602, an image estimation device 603, and a computer (processing device) 604. The training device 601 and the image estimation device 603 are, for example, servers. The computer 604 is, for example, a user terminal (personal computer or smartphone). The computer 604 is connected to the image estimation device 603 via a network 605. The image estimation device 603 is connected to the training device 601 via a network 606. That is, the computer 604 and the image estimation device 603 can communicate with each other, and the image estimation device 603 and the training device 601 can also communicate with each other.

[0081] The construction of training device 601 is similar to that of training device 101 according to the first embodiment, therefore its description is omitted. The construction of camera device 602 is similar to that of camera device 102 according to the first embodiment, therefore its description is omitted.

[0082] The image estimation apparatus 603 includes a memory 603a, an acquisition unit 603b, a corrector 603c, and a communication unit (receiver) 603d. The memory 603a, acquisition unit 603b, and corrector 603c are similar to those of the image estimation apparatus 103 according to the first embodiment. The communication unit 603d has the function of receiving requests sent from the computer 604 and sending the output motion image generated by the image estimation apparatus 603 to the computer 604.

[0083] Computer 604 includes a communication unit (transmitter) 604a, a display unit 604b, an image processing unit 604c, and a recorder 604d. Communication unit 604a has the function of sending a request to image estimation device 603 to cause image estimation device 603 to perform processing on captured moving images, and the function of receiving the output image processed by image estimation device 603. Display unit 604b has the function of displaying various information. The information displayed by display unit 604b includes, for example, captured moving images to be sent to image estimation device 603 and output moving images received from image estimation device 603. Image processing unit 604c has the function of performing further image processing on the output moving images received from image estimation device 603. Recorder 604d records captured moving images acquired from camera device 602, output moving images received from image estimation device 603, etc.

[0084] The image processing according to this embodiment will now be described. The image processing according to this embodiment is equivalent to the upgraded processing of moving images described in the first embodiment. Figure 8 ).

[0085] Figure 16 This is a flowchart illustrating the generation of the upgraded motion picture in this embodiment. Figure 16 The process begins when the user issues a command to start image processing via computer 604. The operation of computer 604 will now be described.

[0086] In step S701, the computer 604 sends a request to the image estimation device 603 to process the captured moving image. The captured moving image to be processed can be sent to the image estimation device 603 using any method. For example, the captured moving image can be uploaded to the image estimation device 603 simultaneously with the processing in step S701, or it can be uploaded to the image estimation device 603 before the processing in step S701. The captured moving image can be an image stored on a server different from the image estimation device 603. In step S701, the computer 604 may send user ID information, etc., along with the request to process the captured moving image.

[0087] In step S702, computer 604 receives the output motion image generated within image estimation device 603. As in the first embodiment, the output image is an upgraded motion image.

[0088] The operation of the image estimation device 603 will now be described.

[0089] In step S801, the image estimation device 603 receives a request from the computer 604 to process the captured moving image. The image estimation device 603 determines that it has been instructed to perform upgrade processing on the captured moving image and executes step S802 and subsequent steps.

[0090] In step S802, the image estimation device 603 acquires the captured motion image and weight information. The weight information is in the same manner as in the first embodiment ( Figure 3 Information trained in a similar manner (trained or learned model). The image estimation device 603 can obtain weight information from the training device 601, or it can obtain weight information previously obtained from the training device 601 and stored in the memory 603a.

[0091] The processes of steps S802 to S808 are similar to those of steps S111 to S117 in the first embodiment, so their descriptions will be omitted.

[0092] In step S809, the image estimation device 603 sends the output motion image to the computer 604.

[0093] This embodiment follows the upgrade process of the first embodiment, but the upgrade process can be performed according to the second embodiment. This embodiment can achieve similar processing and obtain effects on tasks other than the upgrade process.

[0094] When the correction process is performed within the image estimation device 603 as in this embodiment, the processing load caused by the upgrade process can be handled within the image estimation device 603, thereby reducing the processing power required by the computer 604.

[0095] As described above, as in this embodiment, the image estimation device 603 can be configured to be controlled by a computer 604 that is communicatively connected to the image estimation device 603.

[0096] Other embodiments

[0097] Embodiments of this disclosure can also be implemented by a computer in a system or apparatus that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more embodiments described above and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) to perform the functions of one or more embodiments described above, and by methods executed by the computer in the system or apparatus, such as reading and executing computer-executable instructions from the storage medium to perform the functions of one or more embodiments described above and / or controlling one or more circuits to perform the functions of one or more embodiments described above. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include separate computers or a network of separate processors to read and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), the memory of a distributed computing system, an optical disk (e.g., an optical disc (CD), a digital versatile optical disc (DVD), or a Blu-ray disc (BD)). TM One or more of the following: flash memory devices, memory cards, etc.

[0098] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0099] While this disclosure describes exemplary embodiments, it should be understood that this disclosure is not limited to these exemplary embodiments. The scope of the appended claims should be interpreted in the broadest possible sense to cover all such variations and equivalent structures and functions.

[0100] Each embodiment provides an image processing method capable of acquiring high-quality processed motion images.

Claims

1. An image processing method, comprising: Obtain a first input frame group containing multiple consecutive first input frames from the motion image; Obtain a first output frame group containing multiple first output frames, the first output frame group being output by a machine learning model that has received and processed the first input frame group; Obtain a second input frame group containing multiple consecutive second input frames from the motion image; Obtain a second output frame group containing multiple second output frames, the second output frame group being output by the machine learning model that has received and processed the second input frame group; as well as The output motion image frame is obtained based on the plurality of first output frames and the plurality of second output frames. The feature is that each of the plurality of first input frames and the plurality of second input frames includes one or more first frames and one or more second frames, the time of each first frame included in one of the plurality of first input frames and the plurality of second input frames is different from any time included in the other of the plurality of first input frames and the plurality of second input frames, and the time of each second frame included in one of the plurality of first input frames and the plurality of second input frames overlaps with the time of a second frame included in the other of the plurality of first input frames and the plurality of second input frames.

2. The image processing method according to claim 1, characterized in that, The machine learning model uses information from at least one time before and after the time of each output frame when acquiring the first output frame group.

3. The image processing method according to claim 2, characterized in that, The machine learning model upgrades the first input frame group and outputs the first output frame group.

4. The image processing method according to claim 1, characterized in that, The number of second frames contained in each of the plurality of first input frames and the plurality of second input frames is equal to or greater than the amount of information about moments before or after the time of each output frame used in the machine learning model when acquiring each output frame.

5. The image processing method according to claim 1, characterized in that, The number of second frames included in each of the plurality of first input frames and the plurality of second input frames is more than twice the amount of information about moments before or after the time of each output frame used in the machine learning model when acquiring each output frame.

6. The image processing method according to claim 1, characterized in that, The output motion picture frame is obtained by using a plurality of output frames obtained by excluding the output frames corresponding to each of the plurality of first output frames and the plurality of second output frames.

7. The image processing method according to claim 6, characterized in that, The output frames to be excluded corresponding to each second frame are those that have a small amount of information at least before and after the time when each output frame is used to acquire each output frame.

8. The image processing method according to claim 1, characterized in that, The output motion image frame is obtained based on the output frame obtained by weighted averaging of each second frame contained in the plurality of first output frames and each second frame contained in the plurality of second output frames.

9. The image processing method according to claim 8, characterized in that, The weights used for the weighted average are small and are assigned to output frames that have less information at least before and after the time when each output frame is acquired.

10. The image processing method according to claim 1, characterized in that, The machine learning model uses feature maps from at least one time before and after the time of the output frame when acquiring the output frame.

11. The image processing method according to any one of claims 1 to 10, characterized in that, The machine learning model uses feature maps obtained by using input images at least one time before and after the time of the output frame when acquiring the output frame.

12. An image processing apparatus comprising: At least one memory that stores instructions; as well as At least one processor is configured to execute instructions as follows: Obtain a first input frame group containing multiple consecutive first input frames from the motion image; Obtain a first output frame group containing multiple first output frames, the first output frame group being output by a machine learning model that has received and processed the first input frame group; Obtain a second input frame group containing multiple consecutive second input frames from the motion image; Obtain a second output frame group containing multiple second output frames, the second output frame group being output by the machine learning model that has received and processed the second input frame group; as well as The output motion image frame is obtained based on the plurality of first output frames and the plurality of second output frames. The feature is that each of the plurality of first input frames and the plurality of second input frames includes one or more first frames and one or more second frames, the time of each first frame included in one of the plurality of first input frames and the plurality of second input frames is different from any time included in the other of the plurality of first input frames and the plurality of second input frames, and the time of each second frame included in one of the plurality of first input frames and the plurality of second input frames overlaps with the time of a second frame included in the other of the plurality of first input frames and the plurality of second input frames.

13. An image processing system comprising: First device; as well as The second device communicates with the first device. The first device is characterized by including a transmitter configured to send a request to the second device for performing processing on the moving image. The second device includes at least one memory for storing instructions and at least one processor, wherein the at least one processor is configured to: Receive request, Acquire moving images, Obtain a first input frame group containing multiple consecutive first input frames from the motion image; Obtain a first output frame group containing multiple first output frames, the first output frame group being output by a machine learning model that has received and processed the first input frame group; Obtain a second input frame group containing multiple consecutive second input frames from the motion image; Obtain a second output frame group containing multiple second output frames, the second output frame group being output by a machine learning model that has received and processed the second input frame group; and The output motion image frame is obtained based on the plurality of first output frames and the plurality of second output frames. Each of the plurality of first input frames and the plurality of second input frames includes one or more first frames and one or more second frames. The time of each first frame included in one of the plurality of first input frames and the plurality of second input frames is different from any time included in the other of the plurality of first input frames and the plurality of second input frames. Furthermore, the time of each second frame included in one of the plurality of first input frames and the plurality of second input frames overlaps with the time of a second frame included in the other of the plurality of first input frames and the plurality of second input frames.

14. A non-transitory computer-readable storage medium storing a program that causes a computer to perform the image processing method according to any one of claims 1 to 11.

Images