Training data generation device, learning system, training data generation method, and program

Abstract

To generate video teaching data even if there is no video data.SOLUTION: A teaching data generator generates video teaching data for supervised learning, and acquires still image data, generates video data from the acquired still image data, and outputs the generated video data as teaching data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0005] , ,

[0001] This disclosure relates to a technique for generating teacher data for machine learning.

Background Art

[0002] In recent years, in various fields of image processing such as image recognition or image resolution conversion, a learned model obtained by deep learning has been used. When performing deep learning on a moving image, a large amount of moving image data (hereinafter referred to as "teacher moving image data") as teacher data is required. In deep learning for obtaining a learned model, if there is not enough teacher moving image data, image processing such as image recognition cannot be performed, or its accuracy will be significantly reduced. Also, when performing deep learning using only still image data as teacher data instead of teacher moving image data, since the movement as a moving image cannot be learned, the accuracy of image processing using the learned model obtained by the deep learning will be significantly reduced.

[0003] In order to solve such problems, Patent Document 1 discloses a technique for generating moving image data as new teacher moving image data using moving image data. Specifically, the technique disclosed in Patent Document 1 generates teacher moving image data by translating or rotating the pixels of a frame included in the original moving image data, or by changing the brightness or contrast of the frame.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technology disclosed in Patent Document 1 had the problem that it could not generate training video data unless some video data already existed. This disclosure aims to provide a training data generation device that can generate training video data even when no video data exists. [Means for solving the problem]

[0006] The training data generation device according to this disclosure is a training data generation device that generates video data as correct answers in supervised learning, among video training data for supervised learning, and comprises: an image acquisition means for acquiring still image data; a generation means for generating video data from the still image data; and an output means for outputting the video data as training data, wherein the video generation means comprises: a patch acquisition means for acquiring images corresponding to a part of the image region of the still image as patches from the still image; and a frame generation means for generating frame data of the video data using the patches. [Effects of the Invention]

[0007] According to this disclosure, training video data can be generated even when no video data exists. [Brief explanation of the drawing]

[0008] [Figure 1] This block diagram shows an example of the device configuration of the learning system according to Embodiment 1, and an example of the functional configuration of the training data generation device. [Figure 2] This block diagram shows an example of the hardware configuration of a training data generation device according to Embodiment 1. [Figure 3] This is a block diagram showing an example of the functional configuration of the video generation unit according to Embodiment 1. [Figure 4] This figure shows an example of a patch acquired by the patch acquisition unit according to Embodiment 1. [Figure 5] This figure shows an example of a frame generated by the frame generation unit according to Embodiment 1. [Figure 6] This flowchart shows an example of the processing flow of the training data generation device according to Embodiment 1. [Figure 7] This is a block diagram showing an example of the functional configuration of a learning device according to Embodiment 1. [Figure 8] This flowchart shows an example of the processing flow of the learning device according to Embodiment 1. [Figure 9] This is a block diagram showing an example of the functional configuration of a video generation unit according to a modified embodiment of Embodiment 1. [Figure 10] This flowchart shows an example of the processing flow of a training data generation device according to a modified embodiment of Embodiment 1. [Figure 11] This block diagram shows an example of the functional configuration of the training data generation device and the video generation unit according to Embodiment 2. [Figure 12] This flowchart shows an example of the processing flow of the training data generation device according to Embodiment 2. [Modes for carrying out the invention]

[0009] The following describes embodiments of this disclosure with reference to the attached drawings. Note that the configurations shown in the following embodiments are merely examples, and this disclosure is not limited to the illustrated configurations.

[0010] [Embodiment 1] <Configuration of the training data generation device according to Embodiment 1> The teacher data generation device 100 according to Embodiment 1 will be described with reference to Figures 1 to 6. First, the configuration of the teacher data generation device according to Embodiment 1 will be described with reference to Figures 1 and 2. Figure 1(a) is a block diagram showing an example of the device configuration of a learning system 1 to which the teacher data generation device 100 according to Embodiment 1 is applied. The learning system 1 comprises the teacher data generation device 100, a learning device 110, and a storage device 120. The teacher data generation device 100, the learning device 110, and the storage device 120 are connected to each other via a network 130 such as a LAN (Local Area Network) so that they can communicate with each other.

[0011] The training data generation device 100 generates training data for moving images (hereinafter referred to as "training moving image data"). The learning device 110 uses the training moving image data generated by the training data generation device 100 to train a learning model composed of a neural network, and generates a trained model for performing image processing on the moving image data input as explanatory variables. The storage device 120 stores data necessary for the training data generation device 100 to generate training moving image data, and data necessary for the learning device 110 to generate the trained model, etc. Note that the configuration shown in Figure 1(a) may be modified or changed as appropriate. For example, some functions of one device may be changed to be possessed by another device, or a certain function of one device may be divided and implemented by two or more devices. In this case, each device communicates with each other via the network 130 to perform cooperative operation.

[0012] Figure 1(b) is a block diagram showing an example of the functional configuration of the training data generation device 100 according to Embodiment 1. The training data generation device 100 includes an image acquisition unit 101, a moving image generation unit 102, a training data generation unit 103, a pattern determination unit 104, a counting unit 105, and an output unit 109. The processing of each unit (hereinafter also referred to as "functional unit") of the training data generation device 100 will be described later. Note that the configuration shown in Figure 1(b) may be modified or changed as appropriate. For example, one functional unit of the training data generation device 100 may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. In addition, some of the functional units shown in Figure 1(b) may be configured to be present in other devices shown as an example in Figure 1(a). In this case, each device communicates with each other via the network 130 to perform cooperative operation.

[0013] The processing of each part included in the teacher data generation device 100 is performed by hardware such as an ASIC (Application Specific Integrated Circuit) built into the teacher data generation device 100. The processing may be performed by hardware such as an FPGA (Field Programmable Gate Array). Further, the processing may be performed by software using a memory such as a RAM (Random Access Memory) and a processor such as a CPU (Central Processor Unit).

[0014] Referring to FIG. 2, the hardware configuration of the teacher data generation device 100 when each part included in the teacher data generation device 100 operates as software will be described. FIG. 2 is a block diagram showing an example of the hardware configuration of the teacher data generation device 100 according to Embodiment 1. The teacher data generation device 100 is configured by a computer, and the computer has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication unit 207, and a bus 208 as shown as an example in FIG. 2.

[0015] The CPU 201 is a processor that controls the computer by using programs or data stored in the ROM 202, the RAM 203, etc., causing the computer to function as each part included in the teacher data generation device 100 shown in FIG. 1. Note that the teacher data generation device 100 has one or more dedicated hardware different from the CPU 201, and the dedicated hardware may execute at least a part of the processing by the CPU 201. Examples of the dedicated hardware include an ASIC, an FPGA, and a DSP (Digital Signal Processor). The ROM 202 is a memory that stores programs that do not require modification. The RAM 203 is a memory that temporarily stores programs or data supplied from the auxiliary storage device 204, or data etc. supplied from the outside via the communication unit 207. The auxiliary storage device 204 is constituted by, for example, a hard disk drive, and stores various data such as image data or audio data, or programs etc.

[0016] The display unit 205 is constituted by, for example, a liquid crystal display or an LED, etc., and displays a GUI (Graphical User Interface) etc. for a user to operate the teacher data generation device 100 or view the state of processing in the teacher data generation device 100. The operation unit 206 is constituted by, for example, a keyboard, a mouse, a joystick, or a touch panel, etc., and receives an operation by the user and inputs various instructions to the CPU 201. The CPU 201 also operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206.

[0017] The communication unit 207 is used for communication between the training data generation device 100 and external devices. For example, if the training data generation device 100 is connected to an external device by a wired connection, a communication cable is connected to the communication unit 207. If the training data generation device 100 has a function for wireless communication with an external device, the communication unit 207 is equipped with an antenna. The bus 208 connects the various parts of the training data generation device 100 to transmit information. In Embodiment 1, the display unit 205 and the operation unit 206 are described as being located inside the training data generation device 100, but at least one of the display unit 205 and the operation unit 206 may be located outside the training data generation device 100 as a separate device.

[0018] <Processing of each functional unit of the training data generation device according to Embodiment 1> The processing of each functional unit of the training data generation device will now be explained. The image acquisition unit 101 acquires still image data (hereinafter also referred to as "still image data"). Specifically, the image acquisition unit 101 acquires still image data by reading still image data that has been pre-stored in the storage device 130 or auxiliary storage device 204, etc. The storage device 130 or auxiliary storage device 204, etc., stores data of various types of still images, such as nature photographs of landscapes or animals, portraits of people or sports scenes, or photographs of man-made objects such as buildings or industrial products. For example, the image acquisition unit 101 selects and acquires still image data that corresponds to the user's operation from among multiple still image data.

[0019] The motion image generation unit 102 generates motion image data (hereinafter also referred to as "motion image data") using still image data acquired by the image acquisition unit 101. Specifically, the motion image generation unit 102 generates multiple frames by adding pseudo-motion to the still images, and then generates motion image data that includes each of the generated frames. Details of the motion image generation unit 102 will be described later with reference to Figures 3 to 5.

[0020] The training data generation unit 103 generates training data for video (hereinafter also referred to as "training video data") based on the video data generated by the video data generation unit 102. Specifically, the training data generation unit 103 generates video data as training data by adding noise or blur to each frame of the video data, or by downsampling each frame to degrade the image quality of each frame.

[0021] The mode determination unit 104 determines the mode of the video data generated by the video generation unit 102. For example, the video generation unit 102 generates video data based on the mode determined by the mode determination unit 104. The counting unit 105 counts the number of times each mode determined by the mode determination unit 104 occurs. For example, the mode determination unit 104 determines the mode of the video data based on the number of times each mode has occurred, as counted by the counting unit 105. Specifically, for example, the mode determination unit 104 selects the mode with the fewest counts as determined by the counting unit 105 and determines that mode as the mode of the video data.

[0022] The output unit 109 outputs the video data generated by the video generation unit 102 as the training video data. The output unit 109 also outputs the training video data generated by the training data generation unit 103, associating it with the training video data. The output unit 109 outputs the training video data and training video data to the storage device 130, auxiliary storage device 204, or the learning device 110, etc. The training video data and training video data output by the output unit 109 are used for training by the learning device 110.

[0023] <Configuration of the video generation unit 102 according to Embodiment 1> The processing of the video generation unit 102 will be described with reference to Figure 3. Figure 3 is a block diagram showing an example of the functional configuration of the video generation unit 102 according to Embodiment 1. The video generation unit 102 includes a patch acquisition unit 301, an effect application unit 302, an overlay unit 303, and a frame generation unit 304.

[0024] <Processing of each part of the video generation unit 102 according to Embodiment 1> The patch acquisition unit 301 uses the still image data acquired by the image acquisition unit 101 to acquire images as patches from the still image, corresponding to a portion of the still image, such as a square or rectangle. Specifically, for example, the patch acquisition unit 301 acquires patches in the same image format as the still image data acquired by the image acquisition unit 101. For example, if the image format of the still image data acquired by the image acquisition unit 101 is the Bayer array image format used as one of the RAW data, the patch acquisition unit 301 acquires patches in the Bayer array image format.

[0025] Referring to Figure 4, the relationship between the still image and the patch will be explained. Figure 4 is a diagram showing an example of a patch acquired by the patch acquisition unit 301 according to Embodiment 1. In Figure 4, image 400 is an example of a still image shown by still image data acquired by the image acquisition unit 101. Furthermore, the region 401 enclosed by a rectangle shown by a solid line within image 400 is an example of an image region arbitrarily placed by the patch acquisition unit 301. The patch acquisition unit 301 acquires image 402 as a patch by cropping image 400 by region 401.

[0026] The frame generation unit 304 generates frame data of the video image data using the patch acquired by the patch acquisition unit 301. The video image data generated by the video image generation unit 102 is composed of multiple frame data generated by the frame generation unit 304. The relationship between the patch and the frame that constitutes the video image will be explained with reference to Figure 5. Figure 5 is a diagram showing an example of a frame generated by the frame generation unit 304 according to Embodiment 1.

[0027] Figure 5(a) shows multiple patches 501 to 503 obtained by cropping image regions at different positions in the still image 500. The frame generation unit 304 generates data for frames 504 to 506, each using one of the patches 501 to 503 as a frame. In the example shown in Figure 5(a), the patch acquisition unit 301 first places an image region of a predetermined size at an arbitrary position in the still image 500 and acquires the patch corresponding to the placed image region as patch 501. Next, with the position of the placed image region fixed, the patch acquisition unit 301 moves the pixel positions of the still image 500 parallel on the image plane by a geometric transformation process. It also acquires the patch corresponding to the image region in the still image 500 after the transformation process as patch 502. Furthermore, with the position of the placed image region fixed, the patch acquisition unit 301 further moves the pixel positions of the still image 500 parallel on the image plane by a geometric transformation process. Furthermore, the patch corresponding to the image region in the still image 500 after the conversion process is obtained as patch 503.

[0028] In this way, the patch acquisition unit 301 fixes the position of the placed image region and repeatedly moves the positions of all or some pixels of the still image 500 in parallel multiple times by transformation processing using geometric transformation, acquiring a patch each time it moves. The frame generation unit 304 generates multiple frame data, each of the multiple patches acquired by the patch acquisition unit 301 being a frame. By constructing moving image data from the multiple frame data thus generated, moving image data can be generated using still image data. The method of transformation processing using geometric transformation is not limited to moving the positions of all or some pixels of the still image 500 in parallel on the image plane. For example, the patch acquisition unit 301 may move the positions of all or some pixels of the still image 500 by rotational movement around an arbitrary point on the image plane while the position of the placed image region is fixed.

[0029] The parameters of each geometric transformation, as well as the direction and amount of movement due to the geometric transformation, can be determined arbitrarily, for example. Furthermore, the patch acquisition unit 301 may, while fixing the position of the placed image region, enlarge or reduce all or part of the still image 500, and acquire the image corresponding to that image region in the enlarged or reduced still image 500 as a patch.

[0030] The patches 501 to 503 shown in Figure 5(a) were obtained by repeatedly moving the still image 500 through a geometric transformation process, thereby cropping image regions at different positions in the still image 500. The method for cropping image regions at different positions in the still image 500 is not limited to the method described above. For example, the patch acquisition unit 301 may move the position of the placed image region little by little, and each time it moves, it may acquire the image corresponding to the moved image region in the still image 500 as a patch.

[0031] Figure 5(b) shows frames 511 to 513 corresponding to multiple patches obtained by cropping image regions at different positions in the still image 500. In the example shown in Figure 5(b), the patch acquisition unit 301 first places an image region of a predetermined size at an arbitrary position in the still image 500 and acquires a patch 510 corresponding to the placed image region. The frame generation unit 304 generates frame data using the patch 510 acquired by the patch acquisition unit 301 as frame 511. Furthermore, the patch acquisition unit 301 moves the position of the placed image region parallel to the image plane of the still image 500 and acquires a patch corresponding to the moved image region in the still image 500. The frame generation unit 304 generates frame data using the patch acquired by the patch acquisition unit 301 as frame 512. Furthermore, the patch acquisition unit 301 moves the moved image region further parallel to the image plane of the still image 500 and acquires a patch corresponding to the further moved image region in the still image 500. The frame generation unit 304 generates frame data using the patch acquired by the patch acquisition unit 301 as frame 513.

[0032] Furthermore, the method for moving the image region is not limited to moving it parallel to the image plane of the still image 500. For example, the patch acquisition unit 301 may move the image region on the image plane of the still image 500 by rotating it around an arbitrary point on the image plane. Also, the direction and amount of movement of the image region can be determined arbitrarily, for example.

[0033] The method for generating moving image data described with reference to Figures 5(a) and 5(b) was a method for generating multiple frame data that constitute moving image data using multiple patches obtained by cropping image regions at different positions in a still image 500. However, the method for generating moving image data is not limited to this. For example, the moving image generation unit 102 may acquire one patch corresponding to an image region placed at an arbitrary position in the patch acquisition unit 301, and the frame generation unit 304 may generate multiple frame data using the one patch acquired by the patch acquisition unit 301.

[0034] Specifically, for example, the frame generation unit 304 generates frame data in which an image of a predetermined object (hereinafter referred to as "object image") is superimposed on an arbitrary position in a patch acquired by the patch acquisition unit 301. Here, the process of superimposing the object image onto the patch is performed by the superposition unit 303. The superposition unit 303 generates multiple images in which the position where the object image is superimposed on the patch is slightly changed. The frame generation unit 304 uses each image generated by the superposition unit 303 to generate multiple frame data that constitute the moving image data. Figure 5(c) shows a patch 520 corresponding to an image region placed at an arbitrary position, and frames 522 to 524, which are frames in which the image with the object image 521 superimposed on patch 520 is used as the frame. The direction in which the position where the object image is superimposed is changed, and the magnitude of the change, etc., can be determined arbitrarily, for example.

[0035] Furthermore, for example, the frame generation unit 304 performs multiple image transformations on a patch acquired by the patch acquisition unit 301, changing at least one of its brightness, contrast, saturation, and hue. In addition, the frame generation unit 304 generates frame data for each image transformation, thereby generating multiple frame data that constitute the moving image data. Figure 5(d) shows a patch 530 corresponding to an image region placed at an arbitrary position, and frames 531 to 533, which are frames with images obtained by changing the brightness of patch 520. The amount of change in brightness, contrast, saturation, and hue due to each image transformation can be determined arbitrarily, for example.

[0036] The frame generation unit 304 generates frame data using the patch acquired by the patch acquisition unit 301, using one of the frame data generation modes described above as an example, or a combination of two or more generation modes. Specifically, the frame generation unit 304 generates frame data using one of the frame data generation modes described above, or a combination of two or more generation modes, based on the mode determined by the mode determination unit 104. In other words, the mode determined by the mode determination unit 104 is the frame data generation mode by the frame generation unit 304.

[0037] The effect application unit 302 applies a motion blur effect to the patches acquired by the patch acquisition unit 301 based on the direction in which the image region was moved. The frame generation unit 304 generates frame data using the patches after the motion blur effect has been applied by the effect application unit 302. In this way, by applying a motion blur effect to the frames, it is possible to generate dynamic image data that represents the movement of objects within the frames.

[0038] <Processing flow of the training data generation device according to Embodiment 1> The operation of the training data generation device 100 will be explained with reference to Figure 6. Figure 6 is a flowchart showing an example of the processing flow of the training data generation device 100 according to Embodiment 1. In the following explanation, the symbol "S" means step. First, in S601, the image acquisition unit 101 acquires still image data. Next, in S602, the mode determination unit 104 determines the mode of the moving image data based on the number of times each mode type has been counted by the counting unit 105. Next, in S603, the counting unit 105 counts the number of times each mode type has been determined in S602. Next, in S604, the patch acquisition unit 301 acquires patches.

[0039] Next, in S605, the effect application unit 302 determines whether or not to apply a motion blur effect to the patch acquired in S604. Specifically, for example, if the mode of the moving image data determined in S602 includes a mode in which the image region from which the patch is acquired is moved to generate frame data, the effect application unit 302 determines to apply a motion blur effect to the patch. Conversely, if the mode of the moving image data determined in S602 does not include a mode in which the image region from which the patch is acquired is moved to generate frame data, the effect application unit 302 determines not to apply a motion blur effect to the patch. Here, a mode in which the image region is moved to generate frame data includes, for example, a mode in which an object image is superimposed on the patch, or the brightness, contrast, saturation, or hue of the patch is changed to generate frame data. If it is determined in S605 to apply a motion blur effect, in S606 the effect application unit 302 applies a motion blur effect to the patch.

[0040] If, after S606, or if it is determined in S605 that the motion blur effect should not be applied, in S607 the superposition unit 303 determines whether or not to superimpose an object image onto the patch. Specifically, for example, if the mode of the motion image data determined in S602 includes a mode in which an object image is superimposed onto the patch to generate frame data, the superposition unit 303 determines to superimpose an object image onto the patch. Conversely, if the mode of the motion image data determined in S602 does not include a mode in which an object image is superimposed onto the patch to generate frame data, the superposition unit 303 determines not to superimpose an object image onto the patch. If it is determined in S607 to superimpose an object image onto the patch, in S608 the superposition unit 303 superimposes the object image onto the patch.

[0041] If, after S608, or if it is determined in S607 that an object image will not be superimposed on the patch, the frame generation unit 304 generates frame data for the moving image data in S609. Specifically, the frame generation unit 304 generates frame data for the moving image data using the patch acquired in S604, the image with the motion blur effect applied to the patch in S606, or the image with the object image superimposed on the patch in S608. After S609, in S610, the frame generation unit 304 determines whether or not the generation of the moving image data is complete. Specifically, the frame generation unit 304 determines whether or not the generation of all frame data constituting the moving image data as training moving image data is complete.

[0042] If it is determined in S610 that the generation of video data has not been completed, the patch acquisition unit 301, i.e., the video generation unit 102, returns to S604 and acquires the next patch. The video generation unit 102 repeats the process from S604 to S610 until it is determined in S610 that the generation of video data has been completed. If the video data format determined in S602 does not include a format that generates frame data by moving the image region from which to acquire the patch, the video generation unit 102 only needs to acquire a patch similar to the one already acquired. In this case, the video generation unit 102 may also return to S605 and execute the process from S605 to S610 using the already acquired patch. The training data generation device 100 may also determine the format of video data each time it generates frame data in S609. In this case, if it is determined in S610 that the generation of video data has not been completed, the training data generation device 100 returns to S602 and executes the process from S602 to S610.

[0043] If it is determined in S610 that the generation of video data has been completed, in S611 the training data generation unit 103 generates training video data based on the video data generated by the video data generation unit 102. The training data generation unit 103 may use one video data generated by the video data generation unit 102 to generate multiple training video data corresponding to that video data.

[0044] After S611, in S612, the output unit 109 outputs the video data generated by the video generation unit 102 as the training video data. The output unit 109 also outputs the learning video data generated in S611, associated with the training video data. After S612, the training data generation device 100 completes the processing of the flowchart shown in Figure 6, returns to S601, and repeatedly executes the processing of the flowchart, for example, by acquiring other still image data.

[0045] With the teacher data generation device 100 configured as described above, even if teacher video data does not exist in advance, teacher video data can be generated using still image data. Furthermore, with the teacher data generation device 100 configured as described above, even if training video data cannot be prepared, training video data can be generated using teacher video data generated from still image data.

[0046] In Embodiment 1, the mode determination unit 104 was described as determining the mode of the video data based on the number of times each mode type was counted by the counting unit 105, but it is not limited to this. Also, in Embodiment 1, the training data generation device 100 was described as having a counting unit 105, but the training data generation device 100 does not necessarily have a counting unit 105. In this case, for example, the mode determination unit 104 may randomly select and determine one or more combinations from the frame data generation modes described above.

[0047] <Description of the learning device according to Embodiment 1> The learning device 110 will be described with reference to Figures 7 and 8. First, the configuration of the learning device according to Embodiment 1 will be described with reference to Figure 7. Figure 7 is a block diagram showing an example of the functional configuration of the learning device 110 according to Embodiment 1. The learning device 110 has a learning data acquisition unit 710, a model acquisition unit 720, a learning unit 730, and a model output unit 740. Furthermore, the learning unit 730 has an error calculation unit 731 and a model update unit 732. The processing of each part of the learning device 110 is performed by hardware such as an ASIC or FPGA built into the learning device 110, or by software using memory such as RAM 203 and a processor such as CPU 201, as shown as an example in Figure 2.

[0048] The learning data acquisition unit 710 acquires the teacher video data output by the teacher data generation device 100 and the learning video data associated with the teacher video data. Specifically, for example, the learning data acquisition unit 710 acquires the teacher video data and learning video data via the network 130 by reading the teacher video data and learning video data pre-stored in the storage device 120 from the storage device 120. The source of the teacher video data and learning video data is not limited to the storage device 120; the learning data acquisition unit 710 may also request this data from the teacher data generation device 100 and acquire it from the teacher data generation device 100. In this case, the teacher data generation device 100 outputs the teacher video data and learning video data stored in the auxiliary storage device 204, etc., to the learning device 110 via the communication unit 207 in accordance with the request.

[0049] The model acquisition unit 720 acquires data of the learning model before or during training. The learning model is composed of a neural network such as a CNN (Convolutional Neural Network), performs image processing on the video data input as explanatory variables, and outputs the video data after image processing as the target variable. The learning unit 730 uses the training video data and training video data acquired by the training data acquisition unit 710 to repeatedly train the learning model acquired by the model acquisition unit 720 and generate a trained model. Specifically, first, the learning unit 730 inputs the training video data acquired by the training data acquisition unit 710 as explanatory variables into the learning model. Next, the learning unit 730 acquires the video data after image processing (hereinafter referred to as "output video data") that the learning model outputs as the target variable.

[0050] The error calculation unit 731 compares the training video data acquired by the training data acquisition unit 710 with the output video data output from the training model to calculate the error between the training video data and the output video data. Specifically, the error calculation unit 731 compares each frame in the output video data with the corresponding frame in the training video data to calculate the error for each frame. The error calculation unit 731 may also calculate a statistical value such as the sum, mean, or median of the calculated errors and use the calculated statistical value as the error between the training video data and the output video data. For example, a well-known loss function may be used to calculate the error corresponding to each frame. The model update unit 732 updates the parameters of the training model based on the error calculated by the error calculation unit 731 so that the error becomes smaller.

[0051] The model output unit 740 outputs the data of the trained model as trained model data when the training termination conditions in the training unit 730 are met. Specifically, for example, the model output unit 740 outputs the trained model data to the storage device 120 via the network 130 and stores it in the storage device 120. The trained model data output by the model output unit 740 is acquired by an inference device (not shown in Figure 1), for example, and used as an image processor for image processing of moving image data. Here, the termination conditions include reaching a predetermined number of training cycles, reaching a predetermined training period, or the error calculated by the error calculation unit 731 being less than or equal to a predetermined threshold over a predetermined number of cycles or period.

[0052] The operation of the learning device 110 will be explained with reference to Figure 8. Figure 8 is a flowchart showing an example of the processing flow of the learning device 110 according to Embodiment 1. First, in S801, the model acquisition unit 720 acquires data of the learning model. Next, in S802, the learning data acquisition unit 710 acquires training video data and training video data. Next, in S803, the learning unit 730 inputs training video data to the learning model. Next, in S804, the learning unit 730 acquires output video data from the learning model. Next, in S805, the error calculation unit 731 calculates the error between the training video data and the output video data. Next, in S806, the model update unit 732 updates the parameters of the learning model.

[0053] Next, in S807, the model output unit 740 determines whether the learning termination conditions have been met. If it is determined in S807 that the learning termination conditions have not been met, the learning device 110 returns to S802 and repeats the process from S802 to S807 until it is determined in S807 that the learning termination conditions have been met. If it is determined in S807 that the learning termination conditions have been met, in S808, the model output unit 740 outputs the data of the trained model. After S808, the learning device 110 terminates the process shown in the flowchart in Figure 8.

[0054] With the learning system 1 configured as described above, it is possible to obtain a trained model that has been trained using training video data generated from still video data and training video data generated from the training video data.

[0055] In Embodiment 1, a configuration was described in which the training data generation device 100 generates training video data corresponding to the training video data using the training video data generated by the training data generation device 100, but the invention is not limited to this configuration. For example, the learning device 120 may generate training video data corresponding to the training video data using the training video data generated by the training data generation device 100. In this case, for example, the training data acquisition unit 710 of the learning device 120 may be equipped with a function equivalent to the function of the training data generation unit 103 of the training data generation device 100. The training data acquisition unit 710 acquires training video data corresponding to the training video data by acquiring training video data and generating training video data using the acquired training video data.

[0056] [Modified example of Embodiment 1] <Configuration of a training data generation device according to a modified example of Embodiment 1> The configuration of the training data generation device 100 according to a modified embodiment of Embodiment 1 will now be described. The training data generation device 100 according to a modified embodiment of Embodiment 1 has an image acquisition unit 101, a moving image generation unit 102, a training data generation unit 103, a pattern determination unit 104, a counting unit 105, and an output unit 109, similar to the training data generation device 100 according to Embodiment 1. The training data generation device 100 according to a modified embodiment of Embodiment 1 (hereinafter simply referred to as "training data generation device 100") has only the configuration and processing of the moving image generation unit 102 changed compared to the training data generation device 100 according to Embodiment 1. The processing of each part of the training data generation device 100 is performed by hardware such as an ASIC or FPGA built into the training data generation device 100. This processing may also be performed by software using memory such as RAM 203 and a processor such as CPU 201 built into the training data generation device 100, as shown as an example in Figure 2.

[0057] <Configuration of the video generation unit according to a modified example of Embodiment 1> Referring to Figure 9, the functional configuration of the video generation unit 102 according to a modified example of Embodiment 1 will be described. Figure 9 is a block diagram showing an example of the functional configuration of the video generation unit 102 according to a modified example of Embodiment 1. The video generation unit 102 according to a modified example of Embodiment 1 includes a patch acquisition unit 301, an effect application unit 302, an overlay unit 303, a frame generation unit 304, and a score calculation unit 905. The video generation unit 102 according to a modified example of Embodiment 1 (hereinafter simply referred to as "video generation unit 102") has a score calculation unit 905 added compared to the video generation unit 102 according to Embodiment 1. Furthermore, the patch acquisition unit 301 according to a modified example of Embodiment 1 (hereinafter simply referred to as "patch acquisition unit 301") has a different processing method compared to the patch acquisition unit 301 according to Embodiment 1. The effect application unit 302, the superimposing unit 303, and the frame generation unit 304 of the video generation unit 102 are the same as those of the video generation unit 102 according to Embodiment 1, and therefore the same reference numerals are used, and their descriptions are omitted.

[0058] The patch acquisition unit 301 acquires patches in the same manner as the patch acquisition unit 301 according to Embodiment 1.

[0059] The score calculation unit 905 calculates a score that indicates the number of edges or features of the patch acquired by the patch acquisition unit 301.

[0060] The patch acquisition unit 301 reacquires a patch corresponding to an image region at a different position from the image region corresponding to the patch in question, based on the score calculated by the score calculation unit 905. Specifically, the patch is reacquired if the score calculated by the score calculation unit 905 is below a predetermined threshold. The patch acquisition unit 301 may also reacquire a patch corresponding to an image region at a different position from the image region corresponding to the patch in question, based on the mode determined by the mode determination unit 104 and the score calculated by the score calculation unit 905. In this case, for example, the patch acquisition unit 301 reacquires a patch if the mode includes a mode that moves the image region from which the patch is to be acquired to generate frame data, and the score is below a threshold.

[0061] When generating frame data by moving the image region from which to acquire patches, for patches with high scores, the edges indicating the contours of objects in the patch move along with the movement of the image region, thus enabling the generation of dynamic image data that expresses the movement of objects. Conversely, for patches with low scores, it may be difficult to generate dynamic image data that expresses the movement of objects. Therefore, when generating frame data by moving the image region from which to acquire patches, it is desirable for the patch acquisition unit 301 to acquire patches whose scores calculated by the score calculation unit 905 are greater than a predetermined threshold. By configuring the patch acquisition unit 301 as described above, it becomes possible to repeat the acquisition process until a patch whose score calculated by the score calculation unit 905 is greater than a predetermined threshold is acquired. As a result, by configuring the patch acquisition unit 301 as described above, the training data generation device 100 can generate suitable dynamic image data that expresses the movement of objects.

[0062] <Processing flow of the training data generation device according to a modified example of Embodiment 1> The operation of the training data generation device 100 will be explained with reference to Figure 10. Figure 10 is a flowchart showing an example of the processing flow of the training data generation device 100 according to a modified example of Embodiment 1. In Figure 10, the same reference numerals are used for processes similar to those shown in Figure 6, and their explanations are omitted. First, the training data generation device 100 executes the processes from S601 to S604. After S604, in S1001, the score calculation unit 905 calculates the score of the patch acquired in S604. Next, in S1002, the patch acquisition unit 301 determines whether the score calculated in S1001 satisfies the conditions. Specifically, for example, the patch acquisition unit 301 determines whether the score calculated in S1001 is greater than a predetermined threshold. If it is determined in S1002 that the score does not meet the conditions, the training data generator 100 returns to S604 to reacquire the patch and repeats the process from S604 to S1002 until it is determined in S1002 that the score meets the conditions. If it is determined in S1002 that the score meets the conditions, the training data generator 100 executes the process from S605 to S612.

[0063] With the training data generation device 100 configured as described above, even if training motion image data does not exist in advance, it is possible to generate more suitable training motion image data from still image data.

[0064] [Embodiment 2] Referring to Figures 11 and 12, the training data generation device 100a according to Embodiment 2 will be described. For example, the training data generation device 100a is applied to the learning system 1 in place of the training data generation device 100 shown in Figure 1(a). The training video data generation device 100 according to Embodiment 1 or a modified version of Embodiment 1 determined the mode of the video data based on the number of occurrences for each mode type. In contrast, the training video data generation device 100a determines the mode of the video data based on the acquired patch.

[0065] <Configuration of the training data generation device according to Embodiment 2> First, the configuration of the training data generation device 100a will be described with reference to Figure 11. Figure 11(a) is a block diagram showing an example of the functional configuration of the training data generation device 100a according to Embodiment 2. The training data generation device 100a has an image acquisition unit 101, a moving image generation unit 102a, a training data generation unit 103, a pattern determination unit 104a, and an output unit 109. Compared to the training video data generation device 100, the counting unit 105 of the training data generation device 100a has been removed, and the moving image generation unit 102 and the pattern determination unit 104 have been changed to a moving image generation unit 102a and a pattern determination unit 104a. The processing of each part of the training data generation device 100a is performed by hardware such as an ASIC or FPGA built into the training data generation device 100a. This process may be performed by software or the like using a memory such as RAM 203 and a processor such as CPU 201 built into the training data generation device 100a, as shown as an example in Figure 2.

[0066] <Processing of each functional unit of the training data generation device according to Embodiment 2> The processing of each part of the training data generation device 100a will now be described. Note that for parts of the training data generation device 100a that are common to Embodiment 1 or a modified example thereof, the same reference numerals are used, and detailed explanations are omitted. The moving image generation unit 102a generates moving image data using still image data acquired by the image acquisition unit 101. Specifically, the moving image generation unit 102a generates multiple frames by adding simulated motion to still images, and then generates moving image data including each of the generated frames. Figure 11(b) is a block diagram showing an example of the functional configuration of the moving image generation unit 102a according to Embodiment 2. The moving image generation unit 102a includes a patch acquisition unit 301, an effect application unit 302, an overlay unit 303, a frame generation unit 304, and a score calculation unit 905. The processing of each part of the motion image generation unit 102a is the same as that of each part of the motion image generation unit 102 in the modified embodiment 1 shown as an example in Figure 9, so the same reference numerals are used and detailed explanations are omitted.

[0067] The mode determination unit 104a determines the mode of the video data generated by the video generation unit 102a. Specifically, the mode determination unit 104a determines the mode of the video data based on the patches acquired by the patch acquisition unit 301 of the video generation unit 102a. More specifically, the mode determination unit 104a determines the mode of the video data based on the score of the patch first acquired by the patch acquisition unit 301, which is a score indicating the number of edges or feature quantities of the patch calculated by the score calculation unit 905. The video generation unit 102a generates video data based on the mode determined by the mode determination unit 104a.

[0068] For example, if the score of the patch initially acquired by the patch acquisition unit 301 is greater than a predetermined threshold, the edges indicating the contours of objects in the patch move as the image region moves, making it possible to generate moving image data that expresses the movement of objects. In this case, the mode determination unit 104a determines the mode of the moving image data to be a mode in which the image region is moved to generate frame data. On the other hand, if the score of the patch initially acquired is less than or equal to a predetermined threshold, it may be difficult to generate moving image data that expresses the movement of objects in the mode in which the image region is moved to generate frame data. In this case, the mode determination unit 104a determines the mode of the moving image data to be a mode in which an object image is superimposed on the patch, or at least one of the brightness, contrast, saturation, and hue of the patch is changed to generate frame data. In the mode in which an object image is superimposed on the patch, or in the mode in which the brightness, contrast, saturation, or hue of the patch is changed, movement as a moving image can be expressed even in patches with a small number of edges or features.

[0069] <Processing flow of the training data generation device according to Embodiment 2> The operation of the training data generation device 100a will be explained with reference to Figure 12. Figure 12 is a flowchart showing an example of the processing flow of the training data generation device 100a according to Embodiment 2. In Figure 12, the same reference numerals are used for processes that are the same as those shown in Figures 6 or 10, and their explanations are omitted. First, the training data generation device 100a performs the same processing as in S601. After S601, in S1201, the patch acquisition unit 301 acquires an image corresponding to an image region at an arbitrary position in the still image as a patch, based on the still image data acquired in S601. Next, in S1202, the score calculation unit 905 calculates the score of the patch acquired in S1201. Next, in S1203, the mode determination unit 104a determines the mode of the moving image data based on the score calculated in S1202. After S1203, the training data generation device 100a performs the same processing as in S603, and then performs the processes from S605 to S610.

[0070] If it is determined in S610 that the generation of video data has not been completed, in S1204 the patch acquisition unit 301, i.e., the video data generation unit 102a, acquires the next patch. After S1204, the training data generation device 100a executes the processes from S605 to S610. That is, the training data generation device 100a repeatedly executes processes S1204 and S605 to S610 until it is determined in S610 that the generation of video data has been completed. If it is determined in S610 that the generation of video data has been completed, the training data generation device 100a executes processes S611 and S612. After S612, the training data generation device 100a completes the process shown in the flowchart in Figure 12, returns to S601, and repeatedly executes the process in the flowchart.

[0071] Furthermore, as described above, if it is determined in S610 that the generation of the video data has not been completed, the video data generation unit 102a will acquire the next patch in S1204. However, it is not limited to this. For example, if the mode of the video data determined in S1203 does not include a mode in which the image region for acquiring the patch is moved to generate frame data, the video data generation unit 102a may acquire a patch similar to the one already acquired. In this case, the video data generation unit 102a may skip the process in S1204 and execute the processes from S605 to S610 using the patch already acquired in S1201.

[0072] With the training data generation device 100a configured as described above, training video data can be generated from still video data even if training video data does not exist beforehand. In particular, when generating training video data using still video data, a patch score is calculated, and the form of the training video data to be generated is determined according to that score, thereby enabling the efficient generation of suitable video data. Furthermore, with the training data generation device 100a configured as described above, even if training video data cannot be prepared, training video data can be generated using training video data generated from still video data.

[0073] [Other embodiments] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit such as an ASIC that implements one or more functions.

[0074] Within the scope of this disclosure, it is possible to freely combine the embodiments, modify any component of each embodiment, or omit any component in each embodiment. [Explanation of Symbols]

[0075] 100 Training Data Generation Device 101 Image acquisition unit 102 Motion image generation unit 109 Output section

Claims

1. A training data generation device that generates video image data as the correct answer in training, among the training data of video images for supervised learning, Image acquisition means for acquiring data of still images, A motion image generation means that generates motion image data from the data of the still image, Output means for outputting the aforementioned video data as training data, It has, The aforementioned video generation means is A patch acquisition means for acquiring images corresponding to a portion of the image region of the still image as a patch from the still image, A frame generation means that generates frame data of the video image data using the aforementioned patch, Having A training data generation device characterized by the following.

2. The patch acquisition means deforms the still image multiple times by geometric transformation processing to generate multiple data of the still image after the transformation processing, and acquires the patch from each of the multiple still images after the transformation processing. The frame generation means generates the frame data using each of the multiple patches acquired by the patch acquisition means. A training data generation device according to claim 1, characterized by the following:

3. The aforementioned video generation means is An effect application means that applies a motion blur effect to the patch acquired by the patch acquisition means based on the direction of pixel movement in the still image resulting from the conversion process. It has, The frame generation means generates the frame data using the patch after the effect has been applied by the effect application means. A training data generation device according to claim 2, characterized by the following:

4. The conversion process is at least one of the following: a process of translating the position of pixels in the still image, and a process of rotating them. A training data generation device according to claim 2 or 3, characterized by the above.

5. The conversion process is a process of reducing or enlarging the still image. A training data generation device according to any one of claims 2 to 4, characterized by the following:

6. The patch acquisition means acquires from the still image images images corresponding to each of the multiple image regions of the still image that are in different positions from each other, as multiple patches. The frame generation means generates the frame data using each of the multiple patches acquired by the patch acquisition means. A training data generation device according to any one of claims 1 to 5, characterized in that

7. The patch acquisition means acquires the patches corresponding to each of the multiple image regions that are in different positions by moving the image region on the image plane of the still image by at least one of translation and rotation. A training data generation device according to claim 6, characterized by the following:

8. The frame generation means generates frame data by superimposing an image of a predetermined object onto the patch, and generates multiple frame data by moving the position on which the image of the object is superimposed multiple times. A training data generation device according to any one of claims 1 to 7, characterized by the following:

9. The frame generation means performs an image transformation on the patch multiple times, changing at least one of the brightness, contrast, saturation, and hue of the patch, to generate multiple patches after the image transformation, and generates the frame data using each of the generated patches after the image transformation. A training data generation device according to any one of claims 1 to 8, characterized by the following:

10. A mode determination means for determining the mode of the video data generated by the video data generation means, It has, The motion image generation means generates motion image data based on the above configuration. A training data generation device according to any one of claims 1 to 9, characterized in that

11. The mode determination means determines the mode based on the patch acquired by the patch acquisition means. A training data generation device according to claim 10, characterized by the following:

12. The mode determination means calculates the score of the patch and determines the mode based on the calculated score. A training data generation device according to claim 11, characterized by the following:

13. The aspect determination means calculates the score based on the feature quantities of the patch. A training data generation device according to claim 12, characterized by the following:

14. The aspect determination means calculates the score based on the amount of edges included in the patch. A training data generation device according to claim 12 or 13, characterized by the above.

15. The patch acquisition means acquires the patch based on the above configuration. A training data generation device according to claim 11, characterized by the following:

16. The patch acquisition means calculates an image score corresponding to a portion of the image region of the still image, and determines the image region to be acquired as a patch based on the calculated score and the configuration. A training data generation device according to claim 15, characterized by the following:

17. The patch acquisition means calculates the score based on the features of the image. A training data generation device according to claim 16, characterized by the following:

18. The patch acquisition means calculates the score based on the number of edges included in the image. A training data generation device according to claim 16 or 17, characterized by the above.

19. A counting means for counting the number of times the mode determined by the mode determination means is used for each type of mode. It has, The mode determination means determines the mode based on the number of times each mode type has been counted by the counting means. A training data generation device according to claim 10, characterized by the following:

20. The data of the still image and the data of the patch are in the same image format. A training data generation device according to any one of claims 1 to 19, characterized in that

21. The image format of the data for the still image and the data for the patch is the image format of the Bayer array. A training data generation device according to claim 20, characterized by the following:

22. Image acquisition means for acquiring data of still images, A first motion image generation means for generating first motion image data from the data of the aforementioned still image, A second video generation means generates a second video image from the first video image generated by the first video image generation means, A learning means that uses the first video data as the correct answer in training among the training data, and uses the second video data as training data to perform supervised learning on a learning model, Having A learning system characterized by the following:

23. A program for operating a computer as a training data generation device according to any one of claims 1 to 21.

Images