Image processing device, learning device, image processing system, image processing method, and program

Abstract

To provide an image processing system improving reasoning ability for an image including a motion image subject.SOLUTION: An image processing system reasons motion information of an image subject in captured images. And it carries out classification processing for classifying captured images by a reasoning result of the motion information of the image subject, and selects a learning model from a plurality of learning models to be used for the reasoning processing based on a result of the classification processing. Then, it carries out the reasoning processing through applying the learning model selected for the captured images.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to machine learning. [Background technology]

[0002] In recent years, a technology has emerged that applies a learning model obtained by learning parameters of a neural network such as CNN to captured images in real time. In general, the inference performance of a model learned by supervised learning depends on the quantity and quality of training data. In particular, the inference performance of the learning model for inputs not included in a data set that includes the training data is lower than the inference performance for inputs included in the data set. For example, when the conditions of an imaging device are different between learning and inference, the inference performance of the learning model may be lower. Patent Document 1 discloses a technology for switching the parameters of a learning model depending on the imaging element, optical system, or shooting mode of an imaging device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-186918 A Summary of the Invention [Problem to be solved by the invention]

[0004] The characteristics of the images included in the training data and the captured images to be inferred depend not only on conditions that are determined before the start of shooting, such as the imaging element and optical system of the imaging device, or the shooting mode, but also on conditions that change after the start of shooting. For example, the speed and direction of the subject's movement are not determined before the start of shooting, and change from moment to moment during shooting, so the motion blur of the subject changes. However, Patent Document 1 cannot handle differences in the movement of the subject within the image. Therefore, there is a possibility that the inference performance of the learning model will decrease for captured images that contain subject movements that are not included in the training data.

[0005] Therefore, an object of the present invention is to improve inference performance for images that include moving subjects. [Means for solving the problem]

[0006] The present invention is characterized by having an inference means for performing inference processing by applying a learning model to an image, an estimation means for estimating movement information of a subject in the image, and a selection means for selecting a learning model to be used in the inference processing from a plurality of learning models based on the estimation result by the estimation means. Effect of the Invention

[0007] According to the present invention, it is possible to improve inference performance for images that include moving subjects. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of each device. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of each device. [Diagram 3] 13 is a flowchart showing a process executed by the learning device. [Figure 4] 4 is a flowchart showing a process executed by the image processing apparatus. [Diagram 5] FIG. 11 is an explanatory diagram of group classification based on subject movement information. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of each system. [Figure 7] 13 is a flowchart showing a process executed by the learning system. [Figure 8] 13 is a flowchart showing the processing executed by the inference system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, with reference to the attached drawings, embodiments 1 to 3 will be described. In each of the following embodiments, during learning, motion information of a subject in an image of learning data is estimated, and the learning data is classified into groups based on the estimated motion information. Then, a learning model is trained based on the learning data group for each group. In this way, multiple learning models are generated. During inference, motion information of a subject in a captured image to be inferred is estimated, and the captured image is classified into groups based on the estimated motion information. Then, a learning model is selected based on the classified group, and inference is performed using the selected learning model.

[0010] The learning content of the learning model is not particularly limited. For example, a plurality of pairs of an image and an image with noise added to the image are prepared, and the learning model is trained using the learning data with the latter as input image data and the former as output image data, so that noise removal is performed in inference using the learning model. In addition, a plurality of pairs of a high-resolution image and a low-resolution image are prepared, and the learning model is trained using the learning data with the latter as input image data and the former as output image data, so that super-resolution processing is performed in inference using the learning model to make the resolution of the input image higher than the resolution. Below, an example in which the learning content is super-resolution will be described, but the present invention is not limited to this example.

[0011] [Embodiment 1] The image processing system according to this embodiment includes a learning device 1 that performs learning processing, and an image processing device 2 that performs inference processing. The learning device 1 and the image processing device 2 are connected to each other via a communication I / F (interface) 37 (FIG. 2) or the like. Note that the image processing system may be configured such that the learning device 1 and the image processing device 2 are integrated.

[0012] <System configuration> 1(a) shows the functional configuration of a learning device 1 according to this embodiment. The learning device 1 includes a pre-classification learning data storage unit 11, a data classification unit 12, a post-classification learning data storage unit 13, a pre-learning model storage unit 14, a learning unit 15, and a post-learning model storage unit 16.

[0013] The pre-classification learning data storage unit 11 is a storage area on the HDD 34 (FIG. 2) of the learning device 1, and stores pre-classification learning data. In this embodiment, the learning data is a set of high-resolution images and low-resolution images. Note that the pre-classification learning data storage unit 11 may store learning data in an external data server or the like and acquire learning data from an external source.

[0014] The data classifying unit 12 estimates motion information of subjects within images of the learning data stored in the pre-classification learning data storage unit 11, and classifies the learning data into a plurality of groups based on the estimation results. The classified learning data storage unit 13 is a storage area on the HDD 34 (FIG. 2) of the learning device 1, and stores the learning data stored in the pre-classified learning data storage unit 11 for each group (groups 1, 2, and 3 in the example of FIG. 1(a)) classified by the data classifying unit 12. The classified learning data storage unit 13 may classify learning data acquired from outside by the data classifying unit 12, assign a classified group number to the data, and store it again externally.

[0015] The pre-learned model storage unit 14 is a storage area on the HDD 34 (FIG. 2) of the learning device 1, and stores a pre-learned model in which the graph structure of the neural network is defined and the model parameters have not yet been learned. The learning unit 15 performs learning using the learning data groups for each group stored in the classified learning data storage unit 13 to cause the pre-learning model to learn mapping from low-resolution images to high-resolution images. The trained model storage unit 16 is a storage area on the HDD 34 (FIG. 2) of the learning device 1, and stores the pre-trained model trained by the learning unit 15. In the example of FIG. 1(a), trained models 1, 2, and 3 are stored in association with the groups 1, 2, and 3 classified by the data classification unit 12.

[0016] 1(b) shows the functional configuration of the image processing device 2 according to this embodiment. The image processing device 2 has the functions of an image acquisition unit 21, a data classification unit 22, an inference unit 23, a learned model storage unit 24, and an image output unit 25.

[0017] The image acquisition unit 21 continuously acquires images captured by an imaging device 39 (FIG. 2). The data classification unit 22 estimates motion information of a subject in a captured image, and classifies the captured image into a plurality of groups based on the estimation result. The function of the data classification unit 22 is similar to that of the data classification unit 12 of the learning device 1. The inference unit 23 selects a post-learning model to be applied to the captured image from among the multiple post-learning models stored in the post-learning model memory unit 24 based on the classification result of the data classification unit 22, and performs super-resolution processing by inputting the captured image to the selected post-learning model. The trained model storage unit 24 is a storage area on the HDD 34 (FIG. 2) of the image processing device 2, and stores trained models in a configuration similar to that of the trained model storage unit 16 of the learning device 1. In the example of FIG. 1(b), trained models 1, 2, and 3 are stored therein, similar to the trained model storage unit 16 of the learning device 1. The image output unit 25 outputs the image obtained by the super-resolution processing of the inference unit 23 to a display device 40 (FIG. 2), a storage device, or the like.

[0018] 2 is a diagram showing the hardware configuration of the learning device 1. Since the hardware configuration of the image processing device 2 is similar to that of the learning device 1, only the learning device 1 will be described and a description of the image processing device 2 will be omitted.

[0019] The learning device 1 includes a CPU 31, a RAM 32, a ROM 33, a HDD 34, an input / output I / F 35, an input device 36, and a communication I / F 37. These components are connected to each other via a bus 38. A CPU (Central Processing Unit) 31 controls the entire learning device 1. A GPU (Graphics Processing Unit) may be used instead of or together with the CPU. In the learning device 1, the CPU 31 executes a program stored in the ROM 33 or the like to realize the functions of the data classification unit 12 and the learning unit 15 in FIG. 1(a). In the image processing device 2, the CPU 31 executes a program stored in the ROM 33 or the like to realize the functions of the image acquisition unit 21, the data classification unit 22, the inference unit 23, and the image output unit 25 in FIG. 1(b).

[0020] The RAM 32 is a volatile memory and temporarily stores images and execution results of processing. The ROM 33 is a non-volatile memory and stores programs and various data. The HDD (hard disk drive) 34 is a rewritable secondary storage device and stores programs, model parameters, execution results of processing, etc. The learning device 1 further stores learning data. The input / output I / F 35 is an interface for inputting and outputting data between peripheral devices. In the image processing device 2, an imaging device 39 and a display device 40 are connected via the input / output I / F 35. For example, an image input / output board is used as the input / output I / F 35. The imaging device 39 continuously captures images at a set frame rate and continuously outputs the captured images to the image processing device 2. Note that an imaging device such as a Web camera may be connected to the image processing device 2 via the communication I / F 37. The display device 40 is a liquid crystal display or the like and displays execution results of processing, images, etc. The input device 36 is a keyboard, a mouse, a touch panel device, etc., and inputs various user instructions. The communication I / F 37 is an interface for transmitting and receiving data to and from an external device via a LAN or the like.

[0021] <Processing flow> Fig. 3 is a flowchart showing the processing executed by the learning device 1 according to this embodiment. The processing of each flowchart shown in Fig. 3 is realized by the CPU 31 of the learning device 1 executing a program stored in the ROM 33 or the like. Hereinafter, each step (process) is represented by adding an S before the reference symbol.

[0022] FIG. 3(a) is a flowchart showing the process of classifying training data. In S11, the data classification unit 12 acquires pre-classification learning data from the pre-classification learning data storage unit 11. The learning data here is a set of a high-resolution image and a low-resolution image, but is not particularly limited as long as it includes an image for learning. In S12, the data classification unit 12 estimates motion information of the subject in the image of the learning data acquired in S11 (hereinafter, referred to as target learning data). In this embodiment, the motion information of the subject is a scalar value. Examples of the scalar value include the magnitude of the motion blur of the subject and the magnitude of the movement amount of the subject in the image. For example, the data classification unit 12 estimates the magnitude of the motion blur of the subject using an image processing method such as block matching.

[0023] In S13, the data classification unit 12 classifies the target learning data into one of group 1, group 2, ..., group N based on the estimation result of the subject motion information obtained in S12. Fig. 5 is an explanatory diagram of group classification based on the subject motion information. For example, as shown in Fig. 5(a), a monotonically increasing sequence {m n}={m1=0, m2, . . . , m N} and m n more than m n+1 For an area less than 1, group n is assigned, where n is an arbitrary integer between 1 and N-1. Here, the data classification unit 12 assigns m n ≦M <m n+1 If this holds, then the target training data belongs to group n and m NIf M is satisfied, the target training data is treated as belonging to group N. In S14, the data classification unit 12 assigns a group number to the target learning data based on the classification result in S13, and stores the group in the classified learning data storage unit 13. This completes the series of processes in this flowchart. The data classification unit 12 executes the process in this flowchart for all of the pre-classification learning data stored in the pre-classification learning data storage unit 11. As a result, a group of learning data divided into N groups, group 1, group 2, ..., group N, is generated.

[0024] FIG. 3(b) is a flowchart showing the learning process using the classification learning data. In S15, the learning unit 15 acquires a pre-learning model from the pre-learning model storage unit 14, and acquires a group of learning data of a specific group number (here, group n) from the classified learning data storage unit 13. In S16, the learning unit 15 uses the group of learning data acquired in S15 to cause the pre-learning model to learn mapping from low-resolution images to high-resolution images. In S17, the learning unit 15 stores the model learned in S16 in the trained model storage unit 16 as trained model n for group n. This completes the series of processes in this flowchart. The learning unit 15 executes the process in this flowchart for each of the N training data groups. This generates N trained models (model parameters) of model 1, model 2, ..., model N. The model number corresponds to the group number. The learning device 1 transmits the generated N trained models to the image processing device 2 via the communication I / F 35 of the learning device 1. The image processing device 2 stores the N trained models received from the learning device 1 in the trained model storage unit 24.

[0025] Here, a description will be given of a Convolutional Neural Network (CNN) applied to the learning model of this embodiment.

[0026] CNN is a technology that repeatedly performs nonlinear calculations after convolving a filter generated by training with image data. The filter is also called a local receptive field. The image data obtained by convolving the filter with image data and performing nonlinear calculations is called a feature map. Furthermore, learning is performed using training data (or data sets) consisting of pairs of input image data and output image data. Simply put, learning is the generation of filter values ​​from training data that can convert input image data into the corresponding output image data with high accuracy. More details on this will be given later.

[0027] When image data has RGB color channels or when a feature map is composed of multiple pieces of image data, the filter used for convolution also has multiple channels accordingly. In other words, a convolution filter is expressed as a four-dimensional array that includes the number of channels in addition to the vertical and horizontal sizes and number of sheets. The process of performing nonlinear operations after convolving a filter with image data (or a feature map) is expressed in units of layers, such as the nth layer feature map or the nth layer filter. For example, a CNN that repeats filter convolution and nonlinear operations three times has a three-layer model structure.

[0028] The above-mentioned nonlinear calculation process is expressed by the following equation (1).

number

[0029]

number

[0030] Learning models using CNN include ResNet in the field of image recognition and DCGAN in the field of image generation. ResNet is characterized by a model structure that provides paths to shortcut processing in parallel to the convolutional layer, which realizes a multi-layer model and achieves highly accurate recognition that exceeds the recognition rate of humans. The reason that the processing becomes more accurate by making the model multi-layered is, simply put, because it is possible to express nonlinear relationships between input and output by repeating nonlinear calculations many times. DCGAN is a learning model based on Generative Adversarial Networks (GAN), which has a structure that alternately learns a generative model (Generator) and a discriminator model (Discriminator).

[0031] Next, we will explain CNN learning. CNN learning is performed by minimizing the objective function expressed by the following formula (3) for learning data consisting of a set of input learning image (student image) data and corresponding output learning image (teacher image) data.

[0032]

number

[0033] The Adam method is expressed by the following formula (4).

number

[0034] 4 is a flowchart showing the processing executed by the image processing device 2 according to this embodiment. The CPU 31 of the image processing device 2 executes a program stored in the ROM 33 or the like to realize the processing of the flowchart shown in FIG.

[0035] In S21, the inference unit 23 acquires all of the N post-learning models (model parameters) stored in the post-learning model storage unit 24, and expands them in the RAM 32 or the like of the image processing device 2. It is assumed that the post-learning model storage unit 24 stores the N post-learning models received from the learning device 1. In S22, the image acquisition unit 21 acquires a captured image from the imaging device 39. In S23, the data classification unit 22 estimates information about the movement of the subject in the captured image acquired in S22. The process of this step is similar to that of S12. In S24, the data classification unit 22 performs a classification process similar to that in S13 on the captured image acquired in S22 based on the estimation result of the motion information of the subject obtained in S23. For example, if the integer n' is greater than or equal to m for the estimated value M' of the motion information of the subject obtained in S23, n' ≦M' <m n'+1 If the above condition is satisfied, the captured image acquired in S22 is processed as belonging to group n'. In S25, the inference unit 23 selects a trained model corresponding to the classification result in S24 from among the N trained models acquired in S21. Specifically, the inference unit 23 selects a trained model having a model number that is the same as the group number to which the captured image belongs.

[0036] In S26, the inference unit 23 performs inference processing (here, super-resolution processing) by inputting the captured image acquired in S22 to the post-learning model selected in S25. In S27, the image output unit 25 outputs the image obtained by the inference process in S26 to the display device 40. Note that the image may also be output to an external terminal device, HDD 34, or the like. In S28, the image acquisition unit 21 judges whether or not the image capture by the imaging device 39 has ended. If it is judged that the image capture is continuing, the process returns to S22, and the image acquisition unit 21 acquires the captured image of the next frame. This makes it possible to switch the model to be applied to the captured images that are continuously acquired according to the motion information of the subject in the captured images. While the image capture continues, the processes of S22 to S27 are repeatedly executed. If it is judged that the image capture has ended, the series of processes in this flowchart ends.

[0037] According to the present embodiment as described above, the inference performance for images containing moving objects can be improved by switching the learning model to be applied according to the motion information of the object in the image. Furthermore, compared to a method of applying a learning model obtained by learning various images with different motion information of the object, the network size of the learning model can be reduced, and therefore a decrease in the inference speed can be suppressed.

[0038] [Embodiment 2] In the above description of the first embodiment, a method for classifying images contained in the learning data under the condition that the images belong to one of groups 1 to N and do not belong to any other groups has been described. In this method, since there is no common learning data between different groups, there is a possibility that differences in inference results will occur between different groups, and the change in the image output before and after the applied model is switched will not be smooth. In addition, for example, when the estimated value M' of the motion information of the subject in the captured image is a boundary value m between group p and group p+1, p In the case where the model p is close to the model p, the model applied to the successive captured images may frequently switch between model p and model p+1. Such frequent model switching may impair the visibility of the images output continuously. In the second embodiment, a method for smoothing the change in the images output before and after the selected model is switched and for avoiding frequent model switching will be described. Note that the description overlapping with the first embodiment will be omitted.

[0039] The flowcharts of this embodiment are shown in Figures 3 and 4, which are the same as those of embodiment 1. The following will mainly explain steps that differ from embodiment 1. First, the process executed by the learning device 1 of this embodiment will be described. In S13, the data classification unit 12 classifies the learning data into one or two consecutive groups of group 1, group 2, ..., and group N based on the estimation result of the motion information of the subject. For example, as shown in FIG. 5(b), for the possible range of the estimated value M of the motion information of the subject, a monotonically increasing sequence {m n L}={m1 L =0, m2 L , , m L N}, {m n H}={m1 H , m2 H , , m H N = +∞}. Then, m n L more than m n H For areas less than m, group n is assigned, where n is an integer between 1 and N-1. n-1 H <m n+1 L (n ≠ 1) and m n+1 L <m n H In other words, the boundary value of a certain region is included in the region adjacent to the region. Here, the data classification unit 12 classifies m n L ≦M <m n H If this holds, then the target training data belongs to group n and m N L If M is satisfied, the target training data is treated as belonging to group N.

[0040] In S14, the data classification unit 12 assigns a group number to the target learning data based on the classification result of the learning data in S13, and stores the group number in the classified learning data storage unit 13. Note that the learning data may belong to a maximum of two consecutive groups. In the above example, n+1 L ≦M <m n H If this holds, then the target training data belongs to two groups, group n and group n+1. By sharing one training data between consecutive groups in this way, it is expected that the change in the output image before and after the selected model is switched will be smoother.

[0041] Next, the process executed by the image processing device 2 of this embodiment will be described. In S24, the data classification unit 22 performs a classification process similar to that of S13 on the captured image acquired in S22 based on the subject motion information obtained as the estimation result in S23. For example, if the integer n' is greater than or equal to m for the estimated value M' of the subject motion information obtained in S23, n'-1 ≦M' <m n' If the above condition is satisfied, the captured image acquired in S22 is processed as belonging to group n'.

[0042] Here, the integer n' is an integer, and two consecutive integer values ​​(q, q+1) are both m n'-1 ≦M' <m n' In this case, the data classification unit 22 adopts a value that is the same as or closer to the group number of the previous frame image (for example, the captured image one frame before). Specifically, if the captured image one frame before belongs to group q or a group with a number lower than q, n'=q, and if it belongs to group q+1 or a group with a number higher than q+1, n'=q+1. By this process, an arbitrary group lower limit value m n L For example, the estimated value M of the motion information of the object in the captured image is m n L Even if the number of groups changes, it will continue to belong to group n-1. Similarly, any group upper limit value m nH For example, the estimated value M of the motion information of the subject in the captured image is m n H Even if the model fluctuates, it will continue to belong to group n+1. This makes it possible to avoid frequent switching of the model selected in S25.

[0043] As a method for smoothing the output image before and after switching the selected model, in addition to the process of sharing learning data between consecutive groups, the following process may be performed. When the estimated value M' of the motion information of the subject in the captured image is equal to or greater than the lower limit value and less than the upper limit value for multiple groups, the inference unit 23 may perform inference processing using the trained models corresponding to all of the groups. In this case, the weighted arithmetic average of the inference results may be the final output.

[0044] For example, the inference unit 23 may calculate the final output value using the following formula (5). The calculated output value is provided to the image output unit 25.

number

[0045] In FIG. 5(b), the boundary values ​​of each group are set so that the learning data is shared between two consecutive groups, but as a method of sharing learning data between multiple consecutive groups, boundary values ​​may be set so that the learning data is shared between three or more consecutive groups. For example, as shown in FIG. 5(c), for the possible estimated values ​​M of the motion information of the subject in the captured image, a monotonically increasing sequence {m n}={m1=0, m2, …, m N+2} and define m n more than mn+1 For an area less than 1, group n is assigned, where n is an arbitrary integer between 1 and N-1. Here, the data classification unit 12 assigns m n ≦M <m n+3 If this holds, then the target training data belongs to group n and m N+2 If M is satisfied, the target learning data is treated as belonging to group N. Here, the lower limit m of any group is n Whereas, m n-1 ≦m n <m n+2 And, m n-2 ≦m n <m n+1 also holds at the same time. In other words, the boundary value of the area to which group n is assigned is also included in the areas of group n-1 and group n-2. By sharing one learning data among three consecutive groups in this way, it is expected that the change in the output image before and after switching the selected model will be smooth.

[0046] According to the present embodiment as described above, it is possible to reduce the difference in inference results due to models of adjacent groups. It is also possible to avoid frequent switching of models between adjacent groups. This makes it possible to smooth the change in the output image between consecutive frames.

[0047] [Embodiment 3] In the image processing system according to each of the above-mentioned embodiments, the learning device 1 and the image processing device 2 execute the processing, but there are cases where the scale of the learning model or the amount of calculation is large, and only the learning processing or the inference processing is to be executed on a separate device. In this embodiment, the configuration of an image processing system in which only the learning and inference are executed on an external calculation device, and the flow of processing using the image processing system are explained. Note that the explanation overlapping with the first embodiment is omitted.

[0048] <System configuration> The image processing system according to this embodiment includes a learning system made up of a learning device 1 that performs group classification of learning data and a calculation device 3 that performs learning processing, and an inference system made up of an image processing device 2 that performs group classification of captured images and a calculation device 4 that performs inference processing. Note that the hardware configurations of the learning devices 1 and 2 and the calculation devices 3 and 4 are similar to the hardware configuration of the learning device 1 shown in Fig. 2, and therefore description thereof will be omitted.

[0049] 6(a) shows the functional configuration of the learning system according to this embodiment. The learning device 1 has a pre-classification learning data storage unit 11, a data classification unit 12, a classified learning data storage unit 13, a pre-learning model storage unit 14, a learned model storage unit 16, and a communication unit 17. Among these components, the pre-classification learning data storage unit 11, the data classification unit 12, the classified learning data storage unit 13, the pre-learning model storage unit 14, and the learned model storage unit 16 are also included in the system configuration of the first embodiment and have the same functions. The communication unit 17 transmits a pre-learning model and a group of learning data for each group to the calculation device 3 via the communication I / F 35 of the learning device 1, and receives a learned model for each group from the calculation device 3.

[0050] The arithmetic device 3 has a learning unit 61 and a communication unit 62. The functions of the learning unit 61 and the communication unit 62 are realized by the CPU 31 of the arithmetic device 3 executing a program stored in the ROM 33 or the like. The learning unit 61 performs learning for each group using a group of learning data for each group to cause the pre-learning model to learn mapping from a low-resolution image to a high-resolution image. The communication unit 62 receives the pre-learning model and the group of learning data for each group from the learning device 1 via the communication I / F 35 of the arithmetic device 3, and transmits the post-learning model for each group to the learning device 1.

[0051] 6(b) shows the functional configuration of the inference system according to this embodiment. The image processing device 2 has an image acquisition unit 21, a data classification unit 22, a learned model storage unit 24, an image output unit 25, and a communication unit 26. Among these components, the image acquisition unit 21, the data classification unit 22, the learned model storage unit 24, and the image output unit 25 are also included in the system configuration of the first embodiment and have the same functions. The communication unit 26 transmits the captured image acquired by the image acquisition unit 21, the model selection information (e.g., model number), and the learned model group to the calculation device 4 via the communication I / F 35 of the image processing device 2, and receives the image obtained by the super-resolution processing from the calculation device 4.

[0052] The arithmetic device 4 has an inference unit 65 and a communication unit 66. The functions of the inference unit 65 and the communication unit 66 are realized by the CPU 31 of the arithmetic device 4 executing a program stored in the ROM 33 or the like. The inference unit 65 selects a learned model corresponding to the model selection information from the learned model group, and performs super-resolution processing by applying the selected learned model to the captured image. The communication unit 66 receives the captured image, the model selection information, and the learned model group from the image processing device 2 via the communication I / F 35 of the arithmetic device 4, and transmits the image obtained by the super-resolution processing to the image processing device 2.

[0053] <Processing flow> Fig. 7 is a flowchart showing the process executed by the learning system according to this embodiment. The process of classifying learning data is the same as that shown in the flowchart of Fig. 3(a), and therefore the description will be omitted. Fig. 7 is a flowchart showing the learning process performed by the learning system. First, in S71, the communication unit 17 acquires a pre-learning model from the pre-learning model storage unit 14, and acquires from the classified learning data storage unit 13 a group of learning data of a specific group number (here, group n). In S72, the communication unit 17 transmits to the calculation device 3 the unlearned model and the learning data group of the specific group number acquired in S71.

[0054] In S73, the communication unit 62 receives from the learning device 1 the unlearned model and the learning data set of the specific group number transmitted in S72. In S74, the learning unit 61 uses the group of learning data received in S73 to cause the pre-learning model to learn mapping from low-resolution images to high-resolution images. In S75, the communication unit 62 transmits the model trained in S74 to the learning device 1 as a trained model n for the group n.

[0055] In S76, the communication unit 17 receives from the arithmetic device 3 the trained model n for the group n transmitted in S75. In S77, the communication unit 17 stores the N training models in the trained model storage unit 16 as trained model n for group n. This completes the series of processes in this flowchart. The communication unit 17 sequentially transmits N training data groups to the calculation device 3, whereby N training models (model parameters) of model 1, model 2, ..., model N are stored in the trained model storage unit 16. The learning device 1 transmits the N training models received from the calculation device 3 to the image processing device 2. The image processing device 2 stores the N training models received from the learning device 1 in the trained model storage unit 24.

[0056] FIG. 8 is a flowchart showing the processing executed by the inference system according to this embodiment. First, in S81, the communication unit 26 acquires all of the N post-learning models (model parameters) stored in the post-learning model storage unit 24. In S82, the communication unit 26 transmits the trained model acquired in S81 to the calculation device 4. In S83, the communication unit 66 receives the learning model transmitted in S82 from the image processing device 2. Then, the inference unit 65 expands the received learning model (model parameters) in the RAM 32 of the arithmetic device 4, etc. The processes in S84 to S87 are similar to those in S22 to S25.

[0057] In S88, the communication unit 26 transmits to the calculation device 4 the captured image acquired in S84 and the model selection information (for example, the model number) in S87. In S89, the communication unit 66 receives from the image processing device 2 the captured image and the model selection information transmitted in S88. In S90, the inference unit 65 performs inference processing (here, super-resolution processing) by inputting the captured image received in S89 into the learned model corresponding to the model number received in S89. In S91, the communication unit 66 transmits the inference result in S26 to the image processing device 2. In S92, the communication unit 26 receives from the arithmetic device 4 the inference result transmitted in S91.

[0058] In S93, the image output unit 25 outputs to the display device 40 an image representing the inference result received in S91. In S94, the image acquisition unit 21 determines whether or not the image capture by the imaging device 39 has ended. If it is determined that the image capture is continuing, the process returns to S84, and the image acquisition unit 21 acquires the captured image of the next frame. If it is determined that the image capture has ended, the series of processes in this flowchart ends. In S95, the communication unit 66 determines whether or not the image capturing by the imaging device 39 has ended. If it is determined that the image capturing is continuing, the process returns to S89, and data is received from the image processing device 2. If it is determined that the image capturing has ended, the series of processes in this flowchart ends.

[0059] According to the present embodiment as described above, the inference performance for images containing moving objects can be improved by switching the learning model to be applied according to the motion information of the object in the image. Furthermore, compared to a method of applying a learning model obtained by learning various images with different motion information of the object, the network size of the learning model can be reduced, and therefore a decrease in the inference speed can be suppressed.

[0060] [Other variations] In each of the above-mentioned embodiments, the motion information of the subject in the image of the learning data or the captured image is estimated only from the image of the learning data or the captured image. In addition to this method, the motion information of the subject may be estimated using the image of the learning data or the captured image, as well as the shooting parameter information of the imaging device 39, such as the frame rate and the shutter speed. For example, the movement amount of the subject per shutter opening time may be calculated from the movement amount of the subject between frames of consecutive images, the frame rate, and the shutter opening time of the imaging device 39, and the calculation result may be used. Also, the tracking information of the subject may be used. Also, the moving speed of the subject may be measured using an external measuring device, and the measurement result may be used.

[0061] In addition, in the above-described embodiments, the motion information of the subject in the image of the learning data or the captured image is defined as a scalar value, but the motion information of the subject may be defined as a vector value. For example, a vector value representing the movement direction and amount of movement of the subject in the image may be used as the motion information, and the range in which the vector value can be taken may be divided into a plurality of regions, and a group may be assigned to each of the plurality of regions to classify the image. Also, only one of the movement direction and the movement amount may be used. Note that the vector value may be a vector representing the motion blur of the subject, a vector representing the movement of the subject in the image, or the like.

[0062] In addition, in each of the above-mentioned embodiments, a configuration for selecting a learning model corresponding to a classified group has been described, but a configuration for detecting the presence or absence of motion of a subject in a captured image and selecting a learning model according to the detection result may be used. Specifically, the inference unit 23 may apply a learning model for a moving object when it is detected that the subject is moving, and may apply a learning model for a stationary object when it is detected that the subject is stationary. Also, a configuration for classifying the motion blur or the amount of movement of a subject in a captured image into a plurality of levels and selecting a learning model according to the level may be used.

[0063] In addition, in each of the above-mentioned embodiments, a configuration has been described in which images are classified into groups based on estimated values ​​of subject motion information, but a configuration using a learning model for group classification that has been trained to classify images into groups based on subject motion information may also be used. Specifically, a learning model for group classification that has been trained to classify images with similar subject motion information into the same group is used. By using the same learning model for group classification in the learning process and the inference process, it is possible to classify the images of the learning data and the captured images into groups based on the same criteria.

[0064] Furthermore, each of the above-described embodiments may also be realized by executing the following process. That is, computer-readable software (computer program) that realizes the functions of each of the above-described embodiments is supplied to a system or device via a network or various storage media. The computer (or CPU, MPU, etc.) of the system or device reads and executes the program.

[0065] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs. (Configuration 1) An inference means for performing inference processing by applying a learning model to an image; an estimation means for estimating motion information of a subject in the image; a selection means for selecting a learning model to be used in the inference process from among a plurality of learning models based on an estimation result by the estimation means; 13. An image processing device comprising: (Configuration 2) 2. The image processing device according to configuration 1, wherein the selection means performs a classification process for classifying the image based on the estimation result, and selects a learning model based on the result of the classification process. (Configuration 3) The image processing device according to configuration 2, wherein the plurality of learning models are trained for each of a plurality of learning data groups obtained by performing the same classification process as that performed by the selection means, based on motion information of a subject in each image of the learning data group. (Configuration 4) The image processing device according to configuration 2 or 3, further comprising a learning means for generating a plurality of learning models by learning a learning model for each of a plurality of learning data groups obtained by performing the same classification process as that performed by the selection means based on motion information of a subject in each image of a learning data group. (Configuration 5) the classification process is a process of dividing a possible range of the estimated value represented by the estimation result into a plurality of regions, assigning a group to each of the plurality of regions, and classifying the image into the group to which the estimated value by the estimation means belongs; 5. The image processing device according to any one of configurations 2 to 4, wherein the selection means selects a learning model corresponding to the group to which the estimated value belongs. (Configuration 6) 6. The image processing device according to configuration 5, wherein a boundary value of at least one of the plurality of regions is included in a region adjacent to the region. (Configuration 7) 6. The image processing device according to configuration 5, wherein the boundary value of at least one of the plurality of regions is included in a plurality of regions different from the at least one region. (Configuration 8) The method further comprises: acquiring means for continuously acquiring the images; The image processing device according to any one of configurations 5 to 7, characterized in that, when the estimated value belongs to multiple groups, the selection means selects from the multiple groups a group that is the same as or close to the group into which the previously acquired image is classified, and selects a learning model corresponding to the selected group. (Configuration 9) the selection means, when the estimated value belongs to a plurality of the groups, selects a learning model corresponding to each of the plurality of groups; 9. The image processing device according to any one of configurations 5 to 8, wherein the inference means generates an output value from a result of the inference process using the learning model selected by the selection means. (Configuration 10) 10. The image processing device according to any one of configurations 1 to 9, wherein the subject motion information indicates at least one of a magnitude and a direction of motion blur of the subject in the image. (Configuration 11) 11. The image processing device according to any one of configurations 1 to 10, wherein the subject movement information indicates at least one of an amount of movement and a direction of movement of the subject in the image. (Configuration 12) 12. The image processing device according to any one of configurations 1 to 11, wherein the inference process is a super-resolution process for increasing the resolution of the image to a value higher than the current resolution. (Configuration 13) An estimation means for estimating motion information of a subject in each image of a training data group; A classification means for classifying each image in a training data group based on an estimation result by the estimation means; A learning means for generating a plurality of learning models to be applied to an image to be inferred by learning a learning model for each of the plurality of learning data groups obtained by the classification means; A learning device comprising: (Configuration 14) An image processing system including an image processing device and a calculation device, The image processing device includes: An estimation means for estimating motion information of a subject in an image; a selection means for selecting a learning model to be used in an inference process from among a plurality of learning models based on an estimation result by the estimation means; A transmitting means for transmitting the image and selection information representing the learning model selected by the selecting means to the computing device; having The computing device includes: a receiving means for receiving the image and the selection information; an inference means for performing the inference process by applying a learning model corresponding to the selection information to the image; 1. An image processing system comprising: (method) An inference step of performing inference processing by applying a learning model to an image; an estimation step of estimating motion information of a subject in the image; a selection step of selecting a learning model to be used in the inference process from among a plurality of learning models based on an estimation result from the estimation step; 13. An image processing method comprising: (program) The computer of the image processing device An inference means for performing inference processing by applying a learning model to an image; an estimation means for estimating motion information of a subject in the image; a selection means for selecting a learning model to be used in the inference process from among a plurality of learning models based on an estimation result by the estimation means; A program to function as a [Explanation of symbols]

[0066] 1: Learning device, 2: Image processing device

Claims

1. An inference means for performing inference processing by applying a learning model to an image; an estimation means for estimating motion information of a subject in the image; a selection means for selecting a learning model to be used in the inference process from among a plurality of learning models based on an estimation result by the estimation means; 13. An image processing device comprising:

2. The image processing device according to claim 1 , wherein the selection means performs a classification process for classifying the image based on the estimation result, and selects a learning model based on the result of the classification process.

3. The image processing device according to claim 2, characterized in that the multiple learning models are trained for each of multiple learning data groups obtained by performing the same classification process as that performed by the selection means based on motion information of the subject in each image of the learning data group.

4. The image processing device according to claim 2, further comprising a learning means for generating a plurality of learning models by learning a learning model for each of a plurality of learning data groups obtained by performing the same classification process as that performed by the selection means based on motion information of a subject in each image of the learning data group.

5. the classification process is a process of dividing a possible range of the estimated value represented by the estimation result into a plurality of regions, assigning a group to each of the plurality of regions, and classifying the image into the group to which the estimated value by the estimation means belongs; The image processing apparatus according to claim 2 , wherein the selection means selects a learning model corresponding to the group to which the estimated value belongs.

6. The image processing apparatus according to claim 5 , wherein a boundary value of at least one of the plurality of regions is included in an area adjacent to the region.

7. The image processing apparatus according to claim 5 , wherein the boundary value of at least one of the plurality of regions is included in a plurality of regions different from the at least one region.

8. The method further comprises: acquiring means for continuously acquiring the images; The image processing device according to claim 5, characterized in that, when the estimated value belongs to multiple groups, the selection means selects from the multiple groups a group that is the same as or close to the group into which the previously acquired image is classified, and selects a learning model corresponding to the selected group.

9. the selection means, when the estimated value belongs to a plurality of the groups, selects a learning model corresponding to each of the plurality of groups; 6. The image processing apparatus according to claim 5, wherein the inference means generates an output value by combining a result of the inference process using the learning model selected by the selection means.

10. The image processing apparatus according to claim 1 , wherein the object motion information indicates at least one of a magnitude and a direction of motion blur of the object in the image.

11. 2 . The image processing apparatus according to claim 1 , wherein the subject movement information indicates at least one of an amount of movement and a direction of movement of the subject within the image.

12. The image processing apparatus according to claim 1 , wherein the inference processing is a super-resolution processing for increasing the resolution of the image to a value higher than the current resolution.

13. An estimation means for estimating motion information of a subject in each image of a training data group; A classification means for classifying each image in a training data group based on an estimation result by the estimation means; A learning means for generating a plurality of learning models to be applied to an image to be inferred by learning a learning model for each of the plurality of learning data groups obtained by the classification means; A learning device comprising:

14. An image processing system including an image processing device and a calculation device, The image processing device includes: An estimation means for estimating motion information of a subject in an image; a selection means for selecting a learning model to be used in an inference process from among a plurality of learning models based on an estimation result by the estimation means; A transmitting means for transmitting the image and selection information representing the learning model selected by the selecting means to the computing device; having The computing device includes: a receiving means for receiving the image and the selection information; an inference means for performing the inference process by applying a learning model corresponding to the selection information to the image; 1. An image processing system comprising:

15. An inference step of performing inference processing by applying a learning model to an image; an estimation step of estimating motion information of a subject in the image; a selection step of selecting a learning model to be used in the inference process from among a plurality of learning models based on an estimation result from the estimation step; 13. An image processing method comprising:

16. The computer of the image processing device An inference means for performing inference processing by applying a learning model to an image; an estimation means for estimating motion information of a subject in the image; a selection means for selecting a learning model to be used in the inference process from among a plurality of learning models based on an estimation result by the estimation means; A program to function as a

Images