Caption generation device and program thereof

Abstract

To provide a caption generation device 1 that can easily generate captions containing proper nouns.SOLUTION: A caption generation device 1 comprises: an image captioning unit 10 that generates a first caption c; an object region and label output unit 11 that outputs a pair of an object region and an object label of the first caption c; a proper noun association processing unit 12 that associates the object label with a proper noun label; and a caption generation unit 13 that generates a second caption containing the proper noun by replacing the object label with the proper noun label.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a caption generating device and a program therefor. [Background technology]

[0002] Broadcasting stations provide commentary broadcasts so that visually impaired people can enjoy television broadcasts. Because producing commentary broadcasts requires a lot of time and effort, technology to automatically generate commentary broadcasts is needed.

[0003] Therefore, an image captioning model has been proposed that takes an image as input and generates a sentence (hereinafter, a caption) that describes the situation shown in the image in a natural language. For example, Patent Documents 1 and 2 and Non-Patent Document 1 propose a method of inputting an image and generating a caption using a neural network. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7101315 [Patent Document 2] Patent No. 6946508 [Non-patent literature]

[0005] [Non-Patent Document 1] M. Stefanini, et al., “From Show to Tell: A Survey on Deep Learning-Based Image Captioning”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45, pp.539-559, 2021. Summary of the Invention [Problem to be solved by the invention]

[0006] Generally, image captioning models are often unable to generate captions that include proper nouns such as people's names. However, proper nouns such as people's names are important elements for understanding the contents of a program. Automatic generation of captions that include proper nouns is essential for automatic generation of commentary broadcasts.

[0007] When generating captions that include proper nouns using a conventional image captioning model, it is necessary to prepare captions that include proper nouns as training data. For example, when the proper noun is a person's name, there are countless variations of the name, and the characters vary from program to program, making it extremely difficult to prepare training data that includes the person's name.

[0008] An object of the present invention is to provide a caption generation device and a program therefor that can easily generate captions that include proper nouns. [Means for solving the problem]

[0009] In order to solve the above problems, the caption generation device of the present invention is a caption generation device that generates a caption using an input image to which a pair of a subject area, which is an area of ​​a subject having a proper noun, and a proper noun label representing the proper noun is pre-assigned, and is configured to include an image captioning unit, an object area / label output unit, a proper noun correspondence processing unit, and a caption generation unit.

[0010] According to this configuration, the image captioning unit inputs the input image to the image captioning model, thereby generating a first caption in which a proper noun is expressed as a common noun. The object region / label output unit inputs the input image and the first caption into the PhraseGrounding model, and outputs pairs of object regions and object labels corresponding to the common nouns in the first caption. The proper noun correspondence processing unit calculates an IoU that represents the degree of overlap between the subject region and the object region for each pair of the subject region and the object region, and corresponds the object label of the object region to the proper noun label of the subject region based on the calculated IoU. The caption generating unit replaces the object label associated by the proper noun association processing unit with a proper noun label in the first caption, thereby generating a second caption including a proper noun.

[0011] In this way, the caption generation device replaces the object label included in the first caption with a proper noun label, so there is no need to prepare learning data containing proper nouns, and captions containing proper nouns can be easily generated.

[0012] The present invention can also be realized by a program for causing a computer to function as the above-mentioned caption generating device. Effect of the Invention

[0013] According to the present invention, captions including proper nouns can be easily generated. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration of a caption generation device according to an embodiment. [Diagram 2] 4 is a flowchart showing the operation of the caption generation device according to the embodiment. [Diagram 3] 3 is a flowchart showing the process of step S3 in FIG. 2. [Figure 4] 3 is a flowchart showing the process of step S4 in FIG. 2. [Diagram 5] 13(a) and (b) are examples of input images in an embodiment. [Figure 6] 11 is a table showing the results of an evaluation experiment in the examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical idea of ​​the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, the same symbols are used for the same means, and descriptions thereof may be omitted.

[0016] [Caption generation device configuration] The configuration of a caption generation device 1 according to an embodiment will be described with reference to FIG. The caption generation device 1 generates a caption using an input image I to which a pair of a subject region, which is a region of a subject having a proper noun, and a proper noun label, which represents the proper noun, has been assigned in advance.

[0017] In this embodiment, the subject is a person, the proper noun is a person's name, and the subject region is a face region. i face and name label L i psn (where i=1, . . . , n, n is an integer equal to or greater than 1).

[0018] Input image I has face region B i face and name label L i psn The method of adding the face region B is arbitrary. i face and name label L i psn In addition, face detection and face recognition processes are applied to the input image I to identify the face area B i face and name label L i psn More specifically, the face region is clustered and a person's name is assigned to the face cluster, so that the face region B i face and name label L i psn can be granted.

[0019] As shown in FIG. 1, the caption generation device 1 includes an image captioning unit 10, an object region / label output unit 11, a proper noun processing unit 12, and a caption generation unit 13.

[0020] The image captioning unit 10 generates a first caption c in which a proper noun is expressed as an ordinary noun by inputting an input image I into an image captioning model. In other words, the first caption c generated by the image captioning unit 10 does not include a proper noun such as a person's name.

[0021] In this embodiment, the image captioning unit 10 can use a known image captioning model. A feed forward neural network (FFNN) can be applied to the image captioning model. In addition, a known model (e.g., OFA) can be used for the structure of the image captioning model.

[0022] The image captioning unit 10 outputs the generated first caption c to the object region / label output unit 11.

[0023] The object region / label output unit 11 inputs an input image I and a first caption c to the PhraseGrounding model, and outputs a pair of an object region and an object label corresponding to a common noun in the first caption c.

[0024] In this embodiment, the object region / label output unit 11 can use a known PhraseGrounding model. A forward propagation neural network can be applied to the PhraseGrounding model. In addition, a known model (for example, GLIP) can be used for the structure of the PhraseGrounding model.

[0025] The object region represents the region of an object contained in the input image I. The object label is a label that represents the type of object. When the object label is related to a person, both a label representing one person and a label representing multiple people may exist (details will be described later). Here, of the pairs of object region and object label output by the PhraseGrounding model, a pair whose object label represents a person is referred to as object region B. j obj and object label L j obj (where j=1,...,m, m is an integer equal to or greater than 1).

[0026] The object region / label output unit 11 outputs the object region B j obj and the object label L j obj The pair and the first caption c are output to the proper noun processing unit 12.

[0027] The proper noun processing unit 12 detects the face area B i face and object region B j obj For each pair, face region B i face and object region B j obj In other words, the proper noun processing unit 12 calculates an IoU (Intersection of Union) that represents the overlapping degree of the face area B i face and object region B j obj For all combinations of and, calculate the IoU. This IoU is an index that indicates how much two regions overlap, and can be calculated by dividing the intersection of the two regions by their union.

[0028] Then, the proper noun processing unit 12 calculates the object region B j obj Object label L j obj Face area B i face Name label L i psnIn this embodiment, the proper noun processing unit 12 associates the object region B j obj Object label L j obj Face area B i face Name label L i psn For example, the proper noun processing unit 12 judges whether the maximum value of IoU exceeds a preset threshold value. This threshold value can be set arbitrarily (for example, '0'). If the maximum value of IoU exceeds the threshold value, the proper noun processing unit 12 determines whether the object region B j obj Object label L j obj Face area B i face Name label L i psn In this manner, the proper noun processing unit 12 identifies the names of the people included in the input image I.

[0029] Then, the proper noun correspondence processor 12 performs the correspondence P k The corresponding correspondence P k The name label L i psn and the object label L j obj Pair with (L i psn ,L j obj ) (where k = 1, ..., K, K is an integer greater than or equal to 1).

[0030] The caption generation unit 13 generates the object label L associated by the proper noun processing unit 12 in the first caption c. j obj Name label L i psn By replacing

[0031] In this embodiment, the caption generating unit 13 replaces the singular or plural object label of the first caption c with a person name label according to a preset rule. j obj Name label L i psn Replace with.

[0032] <Examples of substitution> The following describes a specific example of replacement by the caption generating unit 13. In the following description, it is assumed that the first caption c is in English.

[0033] First, the caption generation unit 13 extracts an object label L of a person's name from the first caption c. j obj For example, the caption generation unit 13 detects object labels L of singular names of people, such as "a man," "a woman," "a boy," "a girl," "a child," "a family," "a priest," and "a bartender." j obj In addition, for example, the caption generating unit 13 detects object labels L of plurals of people's names such as "men", "women", "boys", "girls", "children", "students" and "people". j obj Detect.

[0034] Here, the caption generation unit 13 uses a regular expression to generate an object label L representing one person or multiple people. j obj In addition, it goes without saying that the object labels detected by the caption generating unit 13 can be manually set by the user and are not limited to the above-mentioned examples.

[0035] Next, the caption generation unit 13 generates an object label L of the person name detected from the first caption c according to the following rules 1 to 4. j obj Name label L i psnIt goes without saying that the rules set in the caption generating unit 13 are not limited to the above example.

[0036] Rule 1: Object label L j obj If represents a single person, the object label L j obj Name label L i psn Replace with. For example, consider a case where the first caption c is "a man standing in a room" and the object label "a man" corresponds to the person name label "Taro". In this case, the caption generation unit 13 replaces the object label "a man" of the first caption c with the person name label "Taro". In this way, the caption generation unit 13 generates the second caption "Taro standing in a room".

[0037] Rule 2: Object label L j obj If represents multiple people, the object label L j obj Name label L i psn and the object label L j obj Replace with a noun phrase connected by a conjunction. For example, consider a case where the first caption c is "people standing in a room" and the object label "people" corresponds to the person name label "Hanako". In this case, the caption generation unit 13 replaces the object label "people" of the first caption c with the noun phrase "Hanako and people" in which the person name label "Hanako" and the object label "people" are connected with the conjunction "and". In this way, the caption generation unit 13 generates the second caption "Hanako and people standing in a room".

[0038] Rule 3: Object label L j objIf L is a plural form containing a numeral, the value of the numeral is reduced or the object label L j obj After converting to singular form, the name label L i psn and the transformed object label L j ob Replace with a noun phrase connected by a conjunction. For example, consider a case where the first caption c is "two men standing in a room" and the object label "two men" corresponds to the person name label "Jiro". In this case, the caption generation unit 13 converts the plural object label "two men" in the first caption c to the singular "a man". Then, the caption generation unit 13 converts the person name label "Jiro" and the converted object label "a man" into a noun phrase "Jiro and a man" connected by the conjunction "and". In this way, the caption generation unit 13 generates the second caption "Jiro and a man standing in a room".

[0039] Rule 4: Object label L j obj When a noun phrase representing a group is formed, the person name label L is placed before the noun phrase. i psn and converting the conjunction into an additional noun phrase. For example, consider a case where the first caption c is "a group of man standing in a room" and the object label "man" corresponds to the person name label "Bob". In this case, the caption generation unit 13 converts the noun phrase "a group of man" representing a group into a noun phrase "Bob and a group of man" by adding the person name label "Bob" and the conjunction "and" before it. In this way, the caption generation unit 13 generates "Bob and a group of man standing in a room".

[0040] [Caption generation device in action] The operation of the caption generation device 1 will be described with reference to FIG. As shown in FIG. 2, in step S1, the image captioning unit 10 generates a first caption c by inputting an input image I into an image captioning model.

[0041] In step S2, the object region / label output unit 11 inputs the input image I and the first caption c to the PhraseGrounding model to obtain the object region B j obj and the object label L j obj Outputs the pair with .

[0042] In step S3, the proper noun processing unit 12 calculates the object label L j obj and name label L i psn The correspondence is made with.

[0043] <Matching object labels and person name labels> The process of step S3 will be described in detail with reference to Fig. 3. Here, it is assumed that the variable i is initialized to 1.

[0044] As shown in FIG. 3, in step S10, the proper noun processing unit 12 i face and object region B j obj Calculate the IoU (Intersection of Union) with. In step S11, the proper noun processing unit 12 selects the object region B j obj The object label L corresponding to j obj Request.

[0045] In step S12, the proper-noun processing unit 12 determines whether or not the maximum value of IoU exceeds a threshold value. If the maximum IoU value exceeds the threshold value (Yes in step S12), the proper-noun handling processor 12 proceeds to the process of step S13. If the maximum IoU value is less than the threshold value (No in step S12), the proper-noun handling processor 12 proceeds to the process of step S14.

[0046] In step S13, the proper noun processing unit 12 j obj Object label L j obj and face area B i face Name label L i psn Mapping P k Request. In step S14, the proper noun processing unit 12 converts the person name label L i psn Mark as handled.

[0047] In step S15, the proper noun processing unit 12 extracts all the personal name labels L i psn It is determined whether or not the request has been handled. All Name Labels L i psn If the proper noun has been dealt with (Yes in step S15), the proper noun handling processor 12 proceeds to the process in step S16. All Name Labels L i psn If the corresponding variable i has not been dealt with (No in step S15), the proper noun handling processor 12 increments the variable i and then returns to the process in step S10.

[0048] In step S16, the proper noun processing unit 12 calculates the object label L j obj and name label L i psn Mapping P k to the caption generating unit 13, and the process ends.

[0049] Returning to FIG. 2, the description of the operation of the caption generation device 1 will be continued. In step S4, the caption generating unit 13 generates a second caption.

[0050] <Generating secondary captions> The process of step S4 will be described in detail with reference to Fig. 4. Here, it is assumed that the variable k is initialized to 1.

[0051] As shown in FIG. 4, in step S20, the caption generating unit 13 generates a caption based on the correspondence P k Object label L j obj Name label L i psn Replace with. In step S21, the caption generating unit 13 converts the correspondence P k Mark as handled.

[0052] In step S22, the caption generating unit 13 k It is determined whether or not the request has been handled. All mappings P k If the corresponding process has been completed (Yes in step S22), the caption generating unit 13 proceeds to the process in step S23. All Name Labels L i psn If the variable k has not been dealt with (No in step S22), the caption generating unit 13 increments the variable k, and then returns to the process of step S20.

[0053] In step S22, the caption generation unit 13 outputs the generated second caption to the outside. After that, the caption generation device 1 ends the operation.

[0054] [Actions and Effects] As described above, the caption generation device 1 extracts the L j obj Name label L i psnSince the caption generation device 1 replaces the caption with the name of a person, there is no need to prepare learning data including the name of a person, and the second caption including the name of a person can be easily generated. This allows the caption generation device 1 to improve the production efficiency of commentary broadcasts.

[0055] Although the embodiment has been described in detail above, the present invention is not limited to the above-described embodiment, and includes design modifications and the like within the scope of the present invention.

[0056] (Variation 1) In the above embodiment, the captions are described as being in English, but the present invention is not limited to this, that is, the caption generation device can be applied to languages ​​other than English (for example, Japanese).

[0057] Specifically, the caption generating device may use an image captioning model that generates captions in Japanese and a PhraseGrounding model that supports Japanese. In this case, the caption generating unit may detect object labels of person names in the singular form such as "man" or "woman" and detect object labels of person names in the plural form such as "people."

[0058] Alternatively, the caption generation device may generate a second caption in English, and the second caption in English may be machine-translated into Japanese.

[0059] (Other variations) In the above embodiment, the proper nouns are described as personal names, but the present invention is not limited to this. In other words, the caption generation device can be applied to proper nouns such as corporate names, place names, country names, and product names, in addition to personal names.

[0060] The secondary captions generated by the caption generator can be used to automatically generate commentary broadcasts, and can also be used as metadata for broadcast programs.

[0061] In the above embodiment, the caption generation device is described as an independent piece of hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program for causing hardware resources such as a CPU, memory, and hard disk of a computer to function as the above-mentioned caption generation device. This program may be distributed via a communication line, or may be written on a recording medium such as a CD-ROM or a flash memory and distributed. EXAMPLES

[0062] Hereinafter, as an example, an evaluation experiment of the captions generated by the caption generation device 1 of FIG. 1 will be described.

[0063] First, an evaluation experiment was conducted to determine whether a person's name was included in the caption generated by the caption generation device 1. In Fig. 5(a) and Fig. 5(b), a black frame represents an object region, and an object label is shown within the black frame. In Fig. 5(a), the object label corresponding to the left object region (male) is "a man," and the object label corresponding to the right object region (female) is "a woman." In Fig. 5(b), the object label corresponding to all object regions is "poeple."

[0064] In the conventional method, only a caption that does not include a person's name, "a man and a woman standing in a restaurant," could be generated from the input image I in Fig. 5(a). In contrast, the caption generation device 1 could generate a second caption that includes the person's name "Hanako," "a man and Hanako standing in a restaurant," from the input image I in Fig. 5(a).

[0065] In addition, the conventional method was only able to generate a caption that did not include people's names, "a group of people standing in a restaurant," from the input image I in Fig. 5(b). In contrast, the caption generation device 1 was able to generate a second caption that included people's names, "Taro and Jiro and a group of people standing in a restaurant," from the input image I in Fig. 6(a).

[0066] Next, an experiment was conducted to evaluate the accuracy of the captions generated by the caption generation device 1. The conditions of the evaluation experiment are as follows. The program footage was "A Little Journey" broadcast on April 7, 14, and 21, 2019. The evaluation data was 168 input images I, each frame randomly selected from shots in which the main characters of the program appeared. The face region was extracted using a face detection model developed by the Japan Broadcasting Corporation, based on RetinaFace with DarkNet-53 as the backbone. The detected face region was tracked using ByteTrack, and the name labels were manually assigned to the results. The image captioning model was OFA, and the phrase grounding model was GLIP. The ground truth data was created by generating captions from the evaluation data using the image captioning model, and manually replacing noun phrases representing people with names.

[0067] As shown in Fig. 6, second captions were generated from three program videos, and the average accuracy rate of the three second captions was used as the evaluation value. As shown in Fig. 6, the evaluation value of the second captions generated by the caption generation device 1 was 90.5%, which is extremely high. [Explanation of symbols]

[0068] 1. Caption Generator 10 Image Captioning Department 11 Object region and label output unit 12 Proper noun processing unit 13 Caption Generation Unit

Claims

1. A caption generation device that generates a caption using an input image to which a pair of a subject region, which is a region of a subject having a proper noun, and a proper noun label representing the proper noun is previously assigned, comprising: an image captioning unit that generates a first caption in which the proper noun is expressed as a common noun by inputting the input image to an image captioning model; an object region / label output unit that outputs a pair of an object region and an object label corresponding to a common noun in the first caption by inputting the input image and the first caption into a PhraseGrounding model; a proper noun correspondence processing unit that calculates an IoU representing an overlapping degree between the subject region and the object region for each pair of the subject region and the object region, and associates an object label of the object region with a proper noun label of the subject region based on the calculated IoU; a caption generating unit that generates a second caption including the proper noun by replacing an object label associated by the proper noun corresponding processing unit in the first caption with the proper noun label; A caption generating device comprising:

2. The caption generating device according to claim 1 , wherein the proper noun correspondence processing unit corresponds the object label of the object region with the maximum IoU to a proper noun label of the subject region.

3. The caption generation device according to claim 1, characterized in that a rule for replacing the object label of the first caption with the proper noun label is set in advance, and the caption generation unit replaces the object label of the first caption with the proper noun label in accordance with the rule.

4. the subject is a person, The proper noun is a person's name, The caption generating device according to claim 1 , wherein the subject area is a face area.

5. A program for causing a computer to function as the caption generating device according to any one of claims 1 to 4.

Images