Image generation device, control method for image generation device, and program

Abstract

This makes it easier to acquire sequential training images that closely resemble the shooting conditions. [Solution] An original training image consisting of a first image group that is sequentially continuous in time and time-series information associated with the first image group are acquired, and change information indicating a change instruction for the first image group is acquired based on user operations, etc. The change information, the time-series information, and frames selected sequentially in time from the first image group are input to an image generation model, a generated image reflecting the change instruction is output, and the generated images generated from each frame are combined based on the time-series information to generate a second image group (new training image) that is sequentially continuous in time.

Description

Technical Field

[0001] The present invention relates to an image generation device, a control method for an image generation device, and a program.

Background Art

[0002] Conventionally, an automatic tracking camera equipped with AI (Artificial Intelligence) that automatically tracks a tracking target and captures an image thereof is known (see Patent Document 1). The AI is trained to detect a tracking target included in a captured image using learning images. When the tracking target moves, the pan-tilt-zoom drive mechanism is controlled so that the tracking target is included in the captured image, and automatic tracking is realized.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to effectively perform automatic tracking using the AI of an automatic tracking camera, it is desirable that the situation of the shooting to be tracked is as close as possible to the situation pre-trained. That is, it is desirable that the features of the image of the site where shooting is to be performed are close to those of the learning images used for learning. For example, when trying to automatically track and photograph a player in a sports game, in order to distinguish the player from other people (e.g., referees) shown in the image, the color and pattern of the player's uniform are used as features and learned by the AI. In that case, it is desirable that the conditions of the game venue are as close as possible between the image of the site and the learning images.

[0005] In general, even within the same sport, the colors and patterns of players' uniforms often differ during sports matches. Furthermore, the conditions of the match venue also vary; for example, in indoor sports, the floor and wall colors often differ from match to match. When attempting to automatically track and photograph players, it is necessary to train the AI ​​beforehand so that it can distinguish between players and other people and automatically track players, even if their uniforms and match venues differ. However, generating training images by going to a location similar to the tracking shooting conditions, taking and editing images for each match venue and uniform is extremely time-consuming. It is also conceivable to use image search or still image generation AI to prepare images from a location similar to the tracking shooting conditions and use them as training images, but since the images do not have continuity over time, they are not suitable for training an AI that performs automatic tracking.

[0006] In view of the aforementioned problems, the present invention aims to make it easier to acquire a series of learning images that closely resemble the conditions under which they were taken. [Means for solving the problem]

[0007] The image generation apparatus according to the present invention is characterized by comprising: a first acquisition means for acquiring a first group of images that are sequentially continuous in time and time-series information associated with the first group of images; a second acquisition means for acquiring change information indicating a change instruction for the first group of images; an image generation means for acquiring a second group of images that reflects the change instruction by using the change information, the time-series information, and images selected sequentially in time from the first group of images as input data for an image generation model; and a coupling means for generating a second group of images that are sequentially continuous in time by linking the second group of images and the time-series information. [Effects of the Invention]

[0008] According to the present invention, it is possible to more easily acquire a series of learning images that closely resemble the conditions under which the images were taken. [Brief explanation of the drawing]

[0009] [Figure 1]This is a block diagram showing an example of the functional configuration of an image generation device according to an embodiment. [Figure 2] This figure shows an example of an original training image. [Figure 3] This flowchart shows an example of the overall processing procedure by the image generation apparatus according to the first embodiment. [Figure 4] This figure shows an example of a generated image produced by the image generation unit. [Figure 5] This figure shows an example of a screen where a user can enter change information. [Figure 6] This is the original training image with metadata attached, or a diagram used to explain the training image. [Figure 7] This flowchart shows an example of the overall processing procedure by the image generation apparatus according to the second embodiment. [Figure 8] This is a block diagram showing an example of the hardware configuration of an image generation device. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the embodiments described below, and various forms that do not depart from the spirit of the invention are also included. Furthermore, each embodiment described below is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0011] (First Embodiment) Figure 8 is a block diagram showing an example of the hardware configuration of the image generation device 100 according to this embodiment. The image generation device 100 includes a CPU 151, a ROM 152, a RAM 153, an HDD 154, a display unit 155, an input unit 156, and a communication unit 157.

[0012] The CPU 151 reads the control program stored in the ROM 152 and executes various processes. The RAM 153 is used as the CPU 151's main memory and temporary storage area such as the work area. The HDD (Hard Disk Drive) 154 stores various data and programs. The display unit 155 is a display device such as an LCD that displays user operations and various information. The input unit 156 has a keyboard and mouse and accepts various operations from the user.

[0013] The communication unit 157 performs communication processing with external devices via a network. Specifically, the communication unit 157 consists of terminals and processing circuits for communication standards such as USB, LAN, and wireless LAN, or for image communication standards such as HDMI® and DisplayPort®.

[0014] The functions and processing of the image generation device 100, described later, are realized by the CPU 151 reading a program stored in the ROM 152 or HDD 154 and executing this program. Alternatively, the CPU 151 may read a program stored in a recording medium such as an SD card instead of the ROM 152.

[0015] Furthermore, in this embodiment, the image generation device 100 uses one processor (CPU 151) and one memory (ROM 152) to execute each process shown in the flowchart described later, but other configurations are also possible. For example, multiple processors, multiple RAMs, ROMs, and storages can work together to execute each process shown in the flowchart described later. Alternatively, some processes may be executed using hardware circuits. In addition, the functions and processes of the image generation device 100 described later may be realized using a processor other than the CPU. For example, a GPU (Graphics Processing Unit) may be used instead of the CPU 151, or a configuration including both a CPU and a GPU may be used.

[0016] FIG. 1 is a block diagram showing a functional configuration example of an image generation apparatus 100 according to the present embodiment. The control unit 101 performs control of each component of the image generation apparatus 100 and various arithmetic processes necessary for the operation of the image generation apparatus 100. The storage unit 102 stores various information necessary for the operation of the image generation apparatus 100.

[0017] The image acquisition unit 103 acquires an original learning image (first image group) that is a learning image. The image output unit 108 outputs a learning image (second image group) generated by the image generation apparatus 100. The image selection unit 104 extracts an image and time series information from the original learning image.

[0018] The change information acquisition unit 105 generates a user interface (UI), receives a user operation from the input unit 156, and displays the operation content of the user on the display unit 155. Further, the change information acquisition unit 105 acquires change information indicating the change content of the original learning image. Examples of the change content include changes in the color or shape of the clothing of a person in the image, changes in the physique or appearance of a person in the image, and changes in the background image of a person in the image. The form of the change information may be any information of a change instruction interpretable by the image generation unit 106, and may be, for example, text data, image data, voice data, or the like. In the present embodiment, it will be described assuming that the change information is text data.

[0019] The image generation unit 106 converts image data into different image data according to the change information using a learned image generation model generated by machine learning and deep learning. The image combining unit 107 generates a new learning image by combining the image data generated by the image generation unit 106 in time series.

[0020] Figure 2 shows an example of an original training image. The original training image shown in Figure 2 shows player 201, player 202, and referee 203. Player 201 is wearing a white uniform, and player 202 is wearing a gray uniform. Referee 203 is wearing a black uniform. The dashed frame 204 shows an example of the desirable shooting range when automatic tracking is performed. When automatic tracking is performed, it is desirable that players 201 and 202 are largely visible within the shooting range, while referee 203 may or may not be within the shooting range. In this embodiment, the AI ​​of the automatic tracking camera is trained to automatically track within the shooting range of the dashed frame 204. Below, in this embodiment, an example of generating a new training image based on the original training image in Figure 2, in which the uniform colors of both players are changed to white, will be described.

[0021] Figure 3 is a flowchart showing an example of the overall processing procedure by the image generation device 100 according to this embodiment. The processing details of the image generation device 100 will be described below with reference to Figure 3.

[0022] In step S301, the image acquisition unit 103 acquires an original training image. The original training image may be acquired from a device that trains a learning model for tracking with an automatic tracking camera, or it may be an original training image that has been pre-stored in the memory unit 102. The original training image to be acquired is a moving image.

[0023] Next, in step S302, the change information acquisition unit 105 acquires the change information generated by the user's operation from the input unit 156 and sends it to the image generation unit 106. In this embodiment, the text information acquired as change information is "Change the color of all players' uniforms to white."

[0024] In step S303, the control unit 101 initializes the frame number n from the start of the frames that make up the original training image to 1. Then, in step S304, the image selection unit 104 selects the frame corresponding to frame number n from the original training image. Subsequently, in step S305, the image selection unit 104 sends the selected frame corresponding to frame number n and the time-series information of frame number n to the image generation unit 106.

[0025] Next, in step S306, the image generation unit 106 generates a generated image corresponding to frame number n based on the input data. Figure 4 shows an example of a generated image. For example, suppose the image shown in Figure 2 is input to the image generation unit 106 as frame number n of the original training image. In this case, based on the change information "change the color of all players' uniforms to white," a generated image is produced in which the color of player 202's uniform has changed from gray to white.

[0026] As described above, the image generation unit 106 outputs a generated image using a trained image generation model generated by machine learning and deep learning. The trained image generation model may be trained by the image generation device 100, or it may be trained by an external device other than the image generation device 100 and acquired by the image generation device 100. In this embodiment, the image generation unit 106 inputs the frame with frame number n, change information, and time-series information associated with the original training image to the image generation model and outputs a generated image in which the color of player 202's uniform has changed from gray to white. In addition, the image generation model may be held by a device other than the image generation device 100 connected to the cloud. In this case, the image generation unit 106 may send input data to the device that holds the image generation model to generate a generated image and acquire the generated image from that device.

[0027] In step S307, the image generation unit 106 stores the generated image in the storage unit 102. Then, in step S308, the control unit 101 determines whether the frame with frame number n is the last frame that constitutes the original training image. If it is the last frame, the process moves to step S310; otherwise, it moves to step S309. Then, in step S309, the control unit 101 increments the frame number n by one.

[0028] Meanwhile, in step S310, the image merging unit 107 sequentially reads and merges the generated images from frame number 1 to the last frame stored in the memory unit 102 to generate a training image for the video. Then, in step S311, the image output unit 108 outputs the training image.

[0029] In this embodiment, the change information acquisition unit 105 may allow the user to input change information using a user interface such as the one shown in Figure 5. In this case, the change information acquisition unit 105 acquires one frame of the original training image from the image acquisition unit 103 in step S302 and analyzes the part of the image that contains a person. The change information acquisition unit 105 then displays the acquired frame on the display unit 155 and, when the user selects a person from the input unit 156, displays a pull-down menu 501 or a color selection menu 502 to allow the user to select the color of the uniform. When the user selects the color of the uniform from the input unit 156, the change information acquisition unit 105 converts that information into text information and sends the change information to the image generation unit 106. For example, if the user selects white in the pull-down menu 501, it is converted into text information that says "Change the color of the player's uniform to white".

[0030] Alternatively, after the user selects a uniform color, the image generation unit 106 may generate a generated image based on the frame of the selected original learning image, and the change information acquisition unit 105 may display the generated image on the display unit 155. In this way, the user can confirm the changed uniform color and input the change information in an easily visible format.

[0031] Furthermore, in this embodiment, the original training image may be one to which metadata indicating the position within the image has been added. For example, information about the type of object and the coordinate information of the object's position are embedded as metadata for each frame during the blanking period. The metadata is added to the original training image by an automatic tracking camera that has been trained to track. The image generation unit 106 can use the added metadata to detect the position of the object more accurately.

[0032] Similarly, metadata indicating the position within the image may be added to the training images generated by the image generation device 100. In this case, the image generation unit 106 adds metadata to each frame according to the position of the detected object, the image merging unit 107 generates a training image including the metadata, and the image output unit 108 outputs it. In an automatic tracking camera that performs tracking training using training images, training can be performed more effectively by using the added metadata.

[0033] Figure 6 is an original training image with metadata added, or a diagram illustrating a training image. In the example in Figure 6, the metadata includes object (person) type information and the position indicated by the coordinate information. In terms of type information, the dashed line area represents a player, and the dotted line area represents a referee. In the example in Figure 6, metadata is added to player 201, player 202, and referee 203, as shown by the dashed line area 601, dashed line area 602, and dotted line area 603, respectively.

[0034] Furthermore, in this embodiment, generated images from other frames stored in the storage unit 102 may also be input to the image generation model of the image generation unit 106 along with other data. By inputting generated images from other frames, the variation in the generated images can be reduced.

[0035] As described above, in this embodiment, the original training image, modification information for the original training image, and time-series information associated with the original training image are input to the image generation model. Then, a group of images reflecting the time-series continuous modification information is generated, and the group of images is linked to the time-series information and combined to generate a new training image. This makes it possible to generate a training image with new features suitable for learning tracking photography with continuous images over time, and reduces the effort required to prepare training images in an environment similar to the tracking photography environment.

[0036] (Second embodiment) In the first embodiment, the case in which a still image (frame) is generated by the image generation model of the image generation unit 106 was described. In this embodiment, an example in which a moving image is generated from a still image by the image generation model of the image generation unit 106 will be described. Note that the internal configuration of the image generation device in this embodiment is the same as in the first embodiment, so the description will be omitted. Below, only the differences from the first embodiment will be described.

[0037] Figure 7 is a flowchart showing an example of the overall processing procedure by the image generation device 100 according to this embodiment. Note that explanations of processes with the same step numbers as the flowchart in Figure 3 are omitted.

[0038] In step S703, the image selection unit 104 divides the original training image into multiple segmented images in the time direction and assigns a segmented image number d to each segmented image in chronological order. In this embodiment, the original training image is divided into segmented images of a video every m minutes, but the duration of the video may differ for each segmented image.

[0039] Next, in step S704, the control unit 101 initializes the segmented image number d to 1. Then, in step S705, the image selection unit 104 selects the segmented image corresponding to segmented image number d. Furthermore, in step S706, the image selection unit 104 sends the representative image extracted from the segmented image corresponding to segmented image number d and time-series information including information for the length m of the segmented image to the image generation unit 106. Here, the representative image is an image of any frame extracted from the segmented image, for example, the first frame of the segmented image is used as the representative image.

[0040] Next, in step S707, the image generation unit 106 generates a numbered image with length m as a video based on the input data using the image generation model.

[0041] In step S709, the control unit 101 determines whether the segmented image corresponding to segmented image number d is the last segmented image of the original training image. If it is the last segmented image, the process moves to step S711; otherwise, it moves to step S710. In step S710, the control unit 101 increments the segmented image number d by one.

[0042] Meanwhile, in step S711, the image merging unit 107 sequentially reads the generated images corresponding to the segmented images from segmented image number 1 to the end stored in the storage unit 102, merges them, and generates a training image.

[0043] In this embodiment, the length of the training image generated will be equal to the length of the original training image, but it may be made to be different. For example, in step S706, when the image selection unit 104 sends a representative image to the image generation unit 106, it may request that the length of the specified generated image be different from the length of the divided image.

[0044] Furthermore, in this embodiment, an example was described in which the image generation model of the image generation unit 106 generates generated images for a video from a still image (representative image). On the other hand, if the image generation model of the image generation unit 106 can generate generated images for a video from a video, in step S706, the image selection unit 104 may send the divided images directly to the image generation unit 106. In this case, in step S707, the image generation unit 106 only needs to input the divided images (multiple frames that are consecutive in time) and output the generated image.

[0045] As described above, according to this embodiment, a representative image of the divided images separated from the original training image, the aforementioned change information, and time-series information including the length m of the divided images linked to the original training image are input to the image generation model that generates the video. Then, a time-series continuous set of moving images that reflects the change information is generated, and the set of moving images is linked to the time-series information and combined to generate a new training image. This reduces the effort required to generate a training image with new features suitable for learning continuous tracking of images over time.

[0046] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0047] This embodiment includes the following configurations, methods, and programs.

[0048] (Composition 1) A first acquisition means for acquiring a first series of images that are sequentially continuous in time and time-series information associated with the first series of images, A second acquisition means for acquiring change information indicating a change instruction for the first group of images, Image generation means that obtains a second set of images in which the change instruction is reflected by using the aforementioned change information, the aforementioned time-series information, and images selected in time-series from the first set of images as input data for an image generation model. A coupling means that links the second group of images and the time-series information to generate a second group of images that are sequentially continuous in time, An image generation device characterized by having the following features.

[0049] (Configuration 2) The image generation apparatus according to configuration 1, characterized in that the image generation means acquires a group of still images as the second group of images. (Composition 3) The image generation apparatus according to configuration 1, characterized in that the image generation means acquires a group of moving images as the second group of images. (Composition 4) The image generation apparatus according to any one of configurations 1 to 3, characterized in that the second acquisition means acquires text data as the change information. (Composition 5) The image generation apparatus according to any one of configurations 1 to 3, characterized in that the second acquisition means acquires image data as the change information.

[0050] (Composition 6) The image generation apparatus according to any one of configurations 1 to 5, characterized in that the second acquisition means acquires the change information based on user operation. (Composition 7) The image generation apparatus according to any one of configurations 1 to 6, characterized in that the second acquisition means acquires a change instruction to change the color or shape of an object in the image as the change information. (Composition 8) The image generation apparatus according to any one of configurations 1 to 7, characterized in that the second acquisition means acquires a change instruction to change the background of an object in the image as the change information. (Composition 9) The image generation apparatus according to any one of configurations 1 to 8, characterized in that the first group of images and the second group of images are training images used for learning to track objects in the images.

[0051] (Composition 10) The image generation apparatus according to configuration 3, characterized in that the image generation means obtains a second group of images in which the change instruction is reflected by using the change information, the time-series information, and a plurality of time-series consecutive images from the first group of images as input data for the image generation model. (Composition 11) The image generation apparatus according to any one of configurations 1 to 10, characterized in that the image generation means also uses a portion of the acquired second group of images as input data to acquire a second group of images in which the change instruction is reflected. (Composition 12) The image generation apparatus according to any one of configurations 1 to 11, characterized in that the first acquisition means acquires a group of images to which metadata relating to objects in the images is attached as the first group of images. (Composition 13) The image generation apparatus according to any one of configurations 1 to 12, characterized in that the combining means generates a group of images to which metadata relating to objects in the images is attached as the second group of images.

[0052] (method) A first acquisition step involves acquiring a first series of images that are sequentially continuous in time and time-series information associated with the first series of images, A second acquisition step involves acquiring change information indicating a change instruction for the first group of images, An image generation process to obtain a second set of images in which the change instructions are reflected, by using the aforementioned change information, the aforementioned time-series information, and images selected in chronological order from the first set of images as input data for an image generation model. A combining step of linking the second group of images and the time-series information to generate a second group of images that are sequentially continuous in time, A control method for an image generation device, characterized by having the following features.

[0053] (program) A first acquisition step involves acquiring a first series of images that are sequentially continuous in time and time-series information associated with the first series of images, A second acquisition step involves acquiring change information indicating a change instruction for the first group of images, An image generation process to obtain a second set of images in which the change instructions are reflected, by using the aforementioned change information, the aforementioned time-series information, and images selected in chronological order from the first set of images as input data for an image generation model. A combining step of linking the second group of images and the time-series information to generate a second group of images that are sequentially continuous in time, A program that causes a computer to execute something. [Explanation of Symbols]

[0054] 103: Image acquisition unit, 105: Change information acquisition unit, 106: Image generation unit, 107: Image merging unit

Claims

1. A first acquisition means for acquiring a first series of images that are sequentially continuous in time and time-series information associated with the first series of images, A second acquisition means for acquiring change information indicating a change instruction for the first group of images, Image generation means that obtains a second group of images in which the change instruction is reflected by using the aforementioned change information, the aforementioned time-series information, and images selected in time-series from the first group of images as input data for an image generation model. A coupling means that links the second group of images and the time-series information to generate a second group of images that are sequentially continuous in time, An image generation device characterized by having the following features.

2. The image generation apparatus according to claim 1, characterized in that the image generation means acquires a group of still images as the second group of images.

3. The image generation apparatus according to claim 1, characterized in that the image generation means acquires a group of moving images as the second group of images.

4. The image generation apparatus according to claim 1, characterized in that the second acquisition means acquires text data as the change information.

5. The image generation apparatus according to claim 1, characterized in that the second acquisition means acquires image data as the change information.

6. The image generation apparatus according to claim 1, characterized in that the second acquisition means acquires the change information based on user operation.

7. The image generation apparatus according to claim 1, characterized in that the second acquisition means acquires a change instruction to change the color or shape of an object in the image as the change information.

8. The image generation apparatus according to claim 1, characterized in that the second acquisition means acquires a change instruction to change the background of an object in the image as the change information.

9. The image generation apparatus according to claim 1, characterized in that the first group of images and the second group of images are training images used for learning to track objects in the images.

10. The image generation apparatus according to claim 3, characterized in that the image generation means obtains a second group of images in which the change instruction is reflected by using the change information, the time-series information, and a plurality of time-series consecutive images from the first group of images as input data for the image generation model.

11. The image generation apparatus according to claim 1, characterized in that the image generation means also uses a portion of the acquired second group of images as input data to acquire a second group of images in which the change instruction is reflected.

12. The image generation apparatus according to claim 1, characterized in that the first acquisition means acquires a group of images to which metadata relating to objects in the images is attached as the first group of images.

13. The image generation apparatus according to claim 1, characterized in that the combining means generates a group of images to which metadata relating to objects in the images is attached as the second group of images.

14. A first acquisition step involves acquiring a first series of images that are sequentially continuous in time and time-series information associated with the first series of images, A second acquisition step involves acquiring change information indicating a change instruction for the first group of images, An image generation process to obtain a second set of images in which the change instructions are reflected, by using the aforementioned change information, the aforementioned time-series information, and images selected in chronological order from the first set of images as input data for an image generation model. A combining step of linking the second group of images and the time-series information to generate a second group of images that are sequentially continuous in time, A control method for an image generation device, characterized by having the following features.

15. A first acquisition step involves acquiring a first series of images that are sequentially continuous in time and time-series information associated with the first series of images, A second acquisition step involves acquiring change information indicating a change instruction for the first group of images, An image generation process to obtain a second set of images in which the change instructions are reflected, by using the aforementioned change information, the aforementioned time-series information, and images selected in chronological order from the first set of images as input data for an image generation model. A combining step of linking the second group of images and the time-series information to generate a second group of images that are sequentially continuous in time, A program that causes a computer to execute something.

Images