Caption generation device and program

Abstract

To improve the accuracy of generating captions for images in television programs.SOLUTION: A caption generation device includes: a metadata extraction unit that extracts a first image and first metadata related to a first program, from a television broadcast stream; and a caption generation unit that generates a caption corresponding to text based on the first metadata and the first image using a machine learning model that has learned the correspondence between text and images, and, correct captions for the text and images.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a caption generating device and a program. [Background technology]

[0002] Conventionally, there are technologies that automatically generate text (captions) that explain the content of images such as still images or videos. Non-Patent Document 1 discloses a captioning technology that uses a neural network. The neural network receives two inputs, an image and instructional text, and outputs text. By changing the instructional text, it is possible to generate any caption (according to the instructional text) for the same image. For example, if the instruction text is "What does this image represent? (Since Literature 1 uses an English technique, "what does the image describe?" is used), a caption that describes the entire image is generated. Also, if the instruction text is "What kind of person is in the image?", a caption that describes the person in the image is generated.

[0003] In Patent Documents 1 and 2, additional information such as driving behavior characteristics and hierarchical labels is used to improve the accuracy of image description generation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-174172 [Patent Document 2] Japanese Patent Publication No. 2022-52481 [Non-patent literature]

[0005] [Non-Patent Document 1] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang, "OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework", Proceedings of the 39th International Conference on Machine Learning, PMLR 162:23318-23340, 2022 Summary of the Invention [Problem to be solved by the invention]

[0006] The captions generated by previous studies are heavily dependent on the dataset used for training. Therefore, the accuracy drops significantly for footage not included in the training data. For example, if a model trained on an overseas dataset is used on footage from a Japanese television program, it will not be able to accurately convey scenes unique to Japan.

[0007] Furthermore, the additional information used by the techniques disclosed in Patent Documents 1 and 2 is not suitable for television program images (it cannot be used directly for television program images).

[0008] The present invention has been made in view of the above points, and has as its object to improve the accuracy of generating captions for images in television programs. [Means for solving the problem]

[0009] To solve the above problem, the caption generation device has a metadata extraction unit that extracts a first image and first metadata related to a first program from a television broadcast stream, and a caption generation unit that generates a caption corresponding to text based on the first metadata and the first image using a machine learning model that has learned the correspondence between text and images and correct captions for the text and images. [Effects of the Invention]

[0010] The accuracy of generating captions for images in television programs can be improved. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of a caption generation device 10 according to a first embodiment. [Figure 2] 1 is a diagram illustrating an example of a functional configuration of a caption generation device 10 according to a first embodiment. [Figure 3] FIG. 2 is a diagram for explaining a captioning model m1 in the first embodiment. [Figure 4] 10 is a flowchart illustrating an example of a processing procedure for caption generation processing according to the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of a functional configuration of a caption generation device 10 according to a second embodiment. [Figure 6] FIG. 10 is a diagram illustrating a captioning model m2 in the second embodiment. [Figure 7] 10 is a flowchart illustrating an example of a processing procedure for caption generation processing according to the second embodiment. [Figure 8] 10 is a flowchart illustrating an example of a processing procedure for generating learning data for a captioning model m2. [Figure 9] FIG. 11 is a diagram for explaining a captioning model m3 in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] This embodiment discloses a technology for performing appropriate captioning on images (video) of television programs. Streams containing images of Japanese television programs are provided with metadata such as an EPG (Electronic Programming Guide) and closed captions (subtitles) in addition to the images. The EPG contains program information such as the program name, program description, genre information, cast information, and broadcast date and time. This embodiment utilizes this metadata to improve the accuracy of captioning images of television programs. Note that the following description describes a method for generating captions from a still image, which is one frame of a video of a television program; however, a method for generating captions from video instead of still images can also be achieved using exactly the same method.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] Fig. 1 is a diagram showing an example of the hardware configuration of a caption generation device 10 in the first embodiment. The caption generation device 10 in Fig. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, which are all interconnected by a bus B.

[0015] A program that realizes processing in the caption generation device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0016] When an instruction to start the program is received, the memory device 103 reads out and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the caption generation device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0017] FIG. 2 is a diagram showing an example of the functional configuration of the caption generation device 10 in the first embodiment. In FIG. 2, the caption generation device 10 has a metadata extraction unit 11, an instruction text generation unit 12, a caption generation unit 13, and a captioning model m1. Each of these units is realized by processing executed by the processor 104 of one or more programs installed in the caption generation device 10. The caption generation device 10 also uses a file storage unit 121 and a caption storage unit 122. Each of these storage units can be realized using, for example, the auxiliary storage device 102, or a storage device connectable to the caption generation device 10 via a network.

[0018] The file storage unit 121 stores files (hereinafter referred to as "TS files") that store television broadcast streams. The streams are, for example, data sequences in MPEG2-TS format. The TS files are, for example, files that contain images and metadata of television programs broadcast during a certain time period. The metadata includes EPG and closed captions (subtitles). The following description focuses on one program (hereinafter referred to as the "target program") included in the stream, but similar processing is performed for each of the other programs.

[0019] The metadata extraction unit 11 extracts images and metadata related to the target program from the stream stored in the TS file. The images and metadata of the target program are contained in the same stream. The format of the stream is publicly known. Therefore, the images and metadata of the target program can be extracted (obtained) from the stream based on the stream format. Note that the images of the target program refer to the images of each frame that make up the video of the target program. Images of all frames may be extracted, or images of some frames may be extracted by thinning them out at regular intervals, for example. Note that metadata that differs for each frame, such as closed captions (subtitles), is extracted for each frame.

[0020] The instruction text generation unit 12 generates instruction text for each image extracted by the metadata extraction unit 11, the instruction text indicating an instruction to generate a caption for the image, based on the metadata extracted by the metadata extraction unit 11. Of the metadata extracted for each frame, the metadata used to generate instruction text for a certain image is the metadata extracted for the frame related to the image. However, like the metadata constituting the EPG, one piece of metadata for a program is common to all images. Note that in the first embodiment, the instruction text generated by the instruction text generation unit 12 is an example of text based on metadata.

[0021] In the first embodiment, a rule-based method is used to generate instruction text. Specifically, a template of instruction text is prepared in advance. Instruction text generation unit 12 generates instruction text by applying metadata to the template.

[0022] For example, if the template says "This image is part of a program called [XXX]. What does this image represent?" and a predetermined rule is set for replacing [XXX] with the program name, the instruction text generation unit 12 replaces [XXX] with the program name included in the metadata. Alternatively, the template may specify which parameter in the metadata will replace the part to be replaced by the metadata, such as "This image is part of a program called [Program Name]. What does this image represent?" In this case, the instruction text generation unit 12 determines, based on the character string "Program Name" in [Program Name], that the character string to replace [Program Name] is the program name in the metadata, and generates instruction text by replacing [Program Name] with the program name.

[0023] For example, using such instruction text in sports programs improves caption accuracy. Specifically, when conventional technology is applied to a high school baseball program, there is about a 30% chance of misrecognizing it as another sport, such as cricket. However, by using instruction text that includes the program name, the possibility of misidentifying the type of sport is almost zero. In addition, by extracting genre information and performer information contained in the EPG, and location information contained in closed captions, and adding this to the instruction text, the caption accuracy for television programs can be improved.

[0024] For each image extracted by the metadata extraction unit 11, the caption generation unit 13 generates a caption corresponding to the image and the instruction text generated by the instruction text generation unit 12 for that image using the captioning model m1, and stores the caption in the caption storage unit 122. Each caption is stored in association with the image corresponding to the caption or information that can identify the image. This makes it easy to identify an image corresponding to a certain caption.

[0025] 3 is a diagram illustrating the captioning model m1 in the first embodiment. As shown in FIG. 3, the captioning model m1 is a machine learning model (e.g., a neural network) that receives an image and text (instruction text) as input and outputs a caption corresponding to the image and the text. The captioning model m1 has previously learned the correspondence between the image and the text and the caption based on training data including the image and the text (instruction text) and a correct answer corresponding to the image and the text.

[0026] A caption corresponding to an image and instruction text refers to text corresponding to the instruction indicated by the instruction text for that image. For example, if the instruction text is "This image is part of Program A. What does this image represent?", the caption content is a string of characters indicating what the image represents. Also, if the instruction text is "This image is part of Program A. What kind of person is in the image?", the caption content is a string of characters indicating the person in the image.

[0027] The following describes the processing procedure executed by the caption generation device 10. Fig. 4 is a flowchart for explaining an example of the processing procedure of the caption generation process in the first embodiment.

[0028] In step S101, the metadata extraction unit 11 extracts images and metadata related to a target program from a stream stored in a TS file.

[0029] Next, the instruction text generation unit 12 generates, for each image extracted by the metadata extraction unit 11, instruction text indicating an instruction to generate a caption for the image, based on the metadata extracted by the metadata extraction unit 11 (S102).

[0030] Next, for each image extracted by the metadata extraction unit 11, the caption generation unit 13 generates a caption corresponding to the instruction text generated for that image by the instruction text generation unit 12 and the image using the captioning model m1, and stores the caption in the caption storage unit 122 (S103). Specifically, the caption generation unit 13 inputs the instruction text and the image into the captioning model m1 (FIG. 3) to obtain a caption output from the captioning model m1. The caption generation unit 13 stores the caption in the caption storage unit 122.

[0031] As described above, according to the first embodiment, for an image constituting a television program, text based on metadata related to the television program can be input to the captioning model m1 to generate a caption. That is, the content of the metadata can be used as a constraint on the generation of captions. Therefore, the accuracy of generating captions for images in a television program can be improved, and more appropriate captions than before can be generated.

[0032] Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be described. Points not specifically mentioned in the second embodiment may be the same as those in the first embodiment.

[0033] Fig. 5 is a diagram showing an example of the functional configuration of the caption generation device 10 in the second embodiment. In Fig. 5, parts that are the same as or correspond to those in Fig. 2 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0034] 5, the caption generation device 10 has an instruction text acquisition unit 14 instead of the instruction text generation unit 12. The caption generation device 10 further has a training data generation unit 15 and a training unit 16. Each of these units is realized by a process in which one or more programs installed in the caption generation device 10 are executed by the processor 104. The caption generation device 10 also uses a training file storage unit 123 and a training data storage unit 124. Each of these storage units can be realized using, for example, the auxiliary storage device 102, or a storage device connectable to the caption generation device 10 via a network.

[0035] The instruction text acquisition unit 14 acquires instruction text. The instruction text may be input by the user or may be stored in advance in the auxiliary storage device 102 or the like. That is, in the second embodiment, the instruction text is not generated from metadata.

[0036] For each image extracted by the metadata extraction unit 11, the caption generation unit 13 generates a caption corresponding to the instruction text acquired for the image by the instruction text acquisition unit 14, the metadata extracted by the metadata extraction unit 11, and the image using the captioning model m2, and stores the caption in the caption storage unit 122.

[0037] FIG. 6 is a diagram illustrating a captioning model m2 in the second embodiment. As shown in FIG. 6, the captioning model m2 is a machine learning model (for example, a neural network such as Transformer) that receives an image and two pieces of text (instruction text and metadata) as input and outputs a caption corresponding to the image and the two pieces of text. The captioning model m2 learns the correspondence between the image, the text, and the caption based on a set of training data including the image, instruction text, metadata, and a caption corresponding to the image and the text as a correct answer. Note that in the second embodiment, the metadata itself is an example of text based on metadata.

[0038] The training data generation unit 15 generates training data for the captioning model m2 and stores the training data in the training data storage unit 124. When generating the training data, the training data generation unit 15 uses images and metadata extracted from a stream (hereinafter referred to as a "training stream") stored in a TS file for training the captioning model m2 stored in the training file storage unit 123. That is, the training file storage unit 123 stores a TS file different from that stored in the file storage unit 121 (for example, a TS file relating to a television broadcast earlier than the TS file stored in the file storage unit 121).

[0039] The learning unit 16 uses the learning data stored in the learning data storage unit 124 to learn the captioning model m2.

[0040] Fig. 7 is a flowchart for explaining an example of the processing procedure of the caption generation processing in the second embodiment. In Fig. 7, the same step numbers as in Fig. 4 are assigned, and their explanations are omitted. It is assumed that the captioning model m2 has already been trained when the processing procedure in Fig. 7 is executed.

[0041] In step S102a, the instruction text acquisition unit 14 acquires the instruction text. Note that step S102a may be executed before step S101.

[0042] Next, for each image extracted by the metadata extraction unit 11, the caption generation unit 13 generates a caption corresponding to the instruction text acquired by the instruction text acquisition unit 14, the image, and the metadata related to the image, using the captioning model m2, and stores the caption in the caption storage unit 122 (S103a). Specifically, the caption generation unit 13 inputs the instruction text, the image, and the metadata into the captioning model m2 (FIG. 6) to acquire a caption output from the captioning model m2. The caption generation unit 13 stores the caption in the caption storage unit 122.

[0043] Next, the generation of training data for the captioning model m2 will be described. The training data may be generated manually, or may be generated efficiently using the captioning model m1, as will be described below.

[0044] FIG. 8 is a flowchart illustrating an example of a processing procedure for generating training data for the captioning model m2.

[0045] In step S201, the metadata extraction unit 11 extracts, from the learning stream, images corresponding to all or some of the frames of a television program included in the learning stream, and metadata corresponding to the images.

[0046] Next, for each extracted image, the learning data generation unit 15 generates a caption based on the image and the instruction text for each instruction text that is scheduled as an input candidate for the captioning model m2, using the captioning model m1 (FIG. 3) (S202). Specifically, the learning data generation unit 15 inputs the image and the instruction text to the captioning model m1 and obtains the caption output by the captioning model m1. For example, if there are M images and N types of instruction text that are input candidates for the captioning model m2, M×N captions are generated.

[0047] Next, for each of the M×N captions, the training data generation unit 15 generates a set of the image and instruction text from which the caption was generated, the metadata corresponding to the image, and the caption as a training data candidate. Thus, M×N candidates, each set of {image, instruction text, metadata, caption}, are generated.

[0048] Next, the learning data generation unit 15 executes a loop process including steps S204 and S205 for each candidate. A candidate that is the processing target in the loop process is hereinafter referred to as a "target candidate."

[0049] In step S204, the learning data generation unit 15 determines whether the caption included in the target candidate is appropriate for the image and instruction text included in the target candidate. This determination may be made, for example, by displaying {image, instruction text, metadata, caption} included in the target candidate so that the user can visually recognize it, and receiving an input from the user regarding the appropriateness of the caption.

[0050] If the user inputs that the caption is appropriate (Yes in S204), the learning data generation unit 15 stores the target candidate in the learning data storage unit 124 as learning data (S205); otherwise (No in S204), the learning data generation unit 15 does not store the target candidate in the learning data storage unit 124.

[0051] When the loop processing is completed for all the candidates, the processing procedure in FIG. 8 ends.

[0052] According to the processing procedure in Figure 8, images and metadata are automatically extracted from the training stream, and captions for the images are also automatically generated, so the user only needs to judge whether the captions are correct or not. This significantly reduces the operational costs compared to manually collecting all the data.

[0053] The learning unit 16 can learn the captioning model m2 (FIG. 6) using the collection of learning data {images, instruction text, metadata, captions} generated in this way. Specifically, the learning unit 16 inputs the images, instruction text, and metadata contained in the learning data to the captioning model m2. The learning unit 16 updates the parameters of the captioning model m2 so as to reduce the loss (error) between the caption output by the captioning model m2 and the caption contained in the learning data. The learning unit 16 repeats this process for all learning data to learn the captioning model m2.

[0054] Next, a third embodiment will be described. In the third embodiment, differences from the second embodiment will be described. Points not specifically mentioned in the third embodiment may be the same as those in the second embodiment.

[0055] Fig. 9 is a diagram illustrating a captioning model m3 in the third embodiment. In Fig. 9, the captioning model m3 differs from the captioning model m2 (Fig. 6) in that it does not receive instruction text as input. That is, the captioning model m3 is a machine learning model (for example, a neural network such as Transformer) that receives an image and metadata as input and outputs a caption. Note that in the third embodiment, the metadata itself is an example of text based on the metadata.

[0056] When using the captioning model m3, the caption generation device 10 does not need to have the instruction text acquisition unit 14. Also, in Fig. 7, step S102a does not need to be executed, and in step S103a, the image and metadata extracted in step S101 may be input into the captioning model m3.

[0057] 8, a set of {image, metadata, caption} may be generated as a candidate for training data in step S203. Therefore, in step S205, training data including {image, metadata, caption} may be stored in the training data storage unit 124.

[0058] The learning unit 16 inputs images and metadata contained in the learning data into the captioning model m3. The learning unit 16 updates the parameters of the captioning model m3 so as to reduce the loss (error) between the caption output by the captioning model m3 and the caption contained in the learning data. The learning unit 16 repeats this process for all learning data to learn the captioning model m3.

[0059] The above-described embodiments can be used for the following purposes, for example.

[0060] [Application 1] When a video summarizing a program after broadcast (hereinafter referred to as "summary video") is generated and distributed to social media, etc., scenes (image groups) to be included in the summary video are identified based on the captions generated for each frame of the program using the method described above. Identification of scenes based on captions can be performed manually or by using a large-scale language model (LLM).

[0061] [Application 2] The system efficiently identifies a desired scene by identifying a group of images corresponding to captions that match the input keywords among the captions generated by the above method for each frame of each program. For example, if captions indicating the names of people appearing in a program have been generated, scenes in which a certain person appears can be efficiently identified.

[0062] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims. [Explanation of symbols]

[0063] 10 Caption Generator 11 Metadata Extraction 12 Instruction text generation unit 13 Caption Generation Unit 14 Instruction text acquisition unit 15 Training data generation unit 16 Learning Department 100 Drive device 101 Recording media 102 Auxiliary storage device 103 Memory Device 104 processors 105 Interface Device 121 File storage section 122 Caption Memory Unit 123 Learning file storage 124 Learning data storage unit B Bus m1, m2, m3 captioning models

Claims

1. a metadata extractor for extracting a first image and first metadata relating to a first program from a television broadcast stream; a caption generation unit that generates a caption corresponding to the text based on the first metadata and the first image using a machine learning model that has learned a correspondence relationship between text and images and a caption that is a correct answer for the text and the image; A caption generating device comprising:

2. an instruction text generation unit that generates instruction text indicating an instruction to generate a caption for the first image based on the metadata; and the caption generation unit generates a caption corresponding to the instruction text and the first image using the machine learning model; 2. The caption generating device according to claim 1, wherein:

3. the caption generation unit generates captions corresponding to the input instruction text, the first metadata, and the images included in the stream for the first program, using the machine learning model that has learned correspondence relationships between instruction text indicating an instruction to generate captions for images constituting a television program, metadata related to the television program, and images constituting the television program, and the instruction text, the metadata, and captions that are correct answers for the images; 2. The caption generating device according to claim 1, wherein:

4. a metadata extraction step for extracting a first image and first metadata relating to a first program from the television broadcast stream; a caption generation step of generating a caption corresponding to the text based on the first metadata and the first image using a machine learning model that has learned correspondences between text and images and correct captions for the text and the image; A program that causes a computer to execute the following.

Images