Caption generation apparatus, caption generation method, and non-transitory computer readable medium

Abstract

The present application provides a subtitle generation device, a subtitle generation method, and a non-transitory computer readable medium. A subtitle of an instance included in a moving image is automatically generated from the moving image. The subtitle generation device generates a subtitle of an instance within a moving image, including an acquisition unit and a control unit. The acquisition unit acquires a moving image. The control unit divides the acquired moving image based on a transition of a scene, and detects an instance in one frame of a plurality of frames included in each divided scene. The control unit tracks the instance in the plurality of frames included in the same scene, and assigns a subtitle to the instance in each frame included in the same scene. The control unit combines a plurality of subtitles assigned to the same instance in each frame of the plurality of frames included in the same scene.

Description

Technical Field

[0001] This disclosure relates to a subtitle generation apparatus, a subtitle generation method, and a non-transitory computer-readable medium. Background Technology

[0002] The following techniques are proposed, which use machine learning models to generate captions (explanatory text) for instances contained in dynamic images. For example, Patent Document 1 proposes a technique that automatically generates commentary statements for character speech based on visual information from the metaverse space.

[0003] Patent Document 1: International Publication No. 2024 / 224928

[0004] There is room for improvement in techniques for generating captions for individual instances contained in a moving image. Summary of the Invention

[0005] The purpose of this disclosure, made in view of this situation, is to improve the technique for generating captions for instances contained in moving images.

[0006] An embodiment of the present disclosure relates to a subtitle generation apparatus that generates subtitles for instances within a moving image, comprising: an acquisition unit for acquiring the moving image; and a control unit that segments the acquired moving image based on scene migration, detects an instance in one frame of multiple frames contained in each segmented scene, tracks the instance in multiple frames contained in the same scene, assigns subtitles to the instance in each frame contained in the same scene, and combines multiple subtitles assigned to the same instance in each frame of the multiple frames contained in the same scene.

[0007] One embodiment of this disclosure relates to a subtitle generation method executed by a control unit of a subtitle generation apparatus, which generates subtitles for instances within a moving image. The method includes: acquiring the moving image; segmenting the acquired moving image based on scene transitions; detecting an instance in one frame of a plurality of frames contained in each segmented scene; tracking the instance in the plurality of frames contained in the same scene; assigning subtitles to the instance in each frame of the same scene; and combining the plurality of subtitles assigned to the same instance in each frame of the plurality of frames contained in the same scene.

[0008] An embodiment of this disclosure relates to a process that causes the control unit of a subtitle generation apparatus for generating subtitles of instances within a moving image to perform processing, wherein the processing includes: acquiring a moving image; segmenting the acquired moving image based on scene migration; detecting an instance in one of a plurality of frames contained in each segmented scene; tracking the instance in the plurality of frames contained in the same scene; assigning subtitles to the instance in each of the frames contained in the same scene; and combining a plurality of subtitles assigned to the same instance in each of the plurality of frames contained in the same scene.

[0009] According to this disclosure, techniques for generating captions for instances contained in moving images can be improved. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating the general structure of a subtitle generation system according to one embodiment of the present disclosure.

[0011] Figure 2 It is by Figure 1 The flowchart shows the subtitle generation process performed by the control unit of the subtitle generation device.

[0012] Figure 3 This diagram illustrates the steps involved in subtitle generation.

[0013] Figure 4 This is a diagram illustrating one example of instance tracking.

[0014] Figure 5 This is a diagram illustrating the generation method of the instance / subtitle involved in the comparative example.

[0015] Explanation of reference numerals in the attached figures

[0016] 1…Subtitle generation system; 10…Subtitle generation device; 11…Input / output interface (acquisition unit); 12…Communication interface (acquisition unit); 13…Control unit; 14…Storage unit; 14a…Program; 14b…Motion image file; 20…Visual Large Scale Language Model (VLLM); 30…Large Scale Language Model (LLM); 40…Motion image; 41…Frame; 42, 42a, 42b…Instance; 43, 43a, 43b…Bounding box; 44a, 44b…Frame level / Instance / Subtitle; 45a, 45b…Motion image level / Instance / Subtitle. Detailed Implementation

[0017] As a multimodal machine learning model capable of processing textual information and image information containing dynamic images, the Vision Language Model (VLM) is well-known. For example, a Vision Language Model can take a dynamic image as input and output a text (subtitle) that explains the input dynamic image.

[0018] To train a visual language model that improves the recognition of individual instances contained within a moving image, a dataset containing captions for instances within the moving image is needed. However, most existing models for generating captions from moving images are limited to generating captions for the entire image and cannot generate detailed captions for each instance. In the following embodiments of this disclosure, the aim is to generate detailed captions for each instance contained within a moving image.

[0019] In this disclosure, "example" refers to any object contained in an image, such as a specific object, person, or animal. Additionally, "caption" refers to explanatory text attached to an image or video. The caption of an example is explanatory text that describes the example. Hereinafter, the caption of an example will sometimes be referred to as "example / caption".

[0020] Hereinafter, one embodiment of the present disclosure will be described with reference to the accompanying drawings.

[0021] (Subtitle generation system)

[0022] Figure 1 This is a diagram illustrating the schematic structure of a subtitle generation system 1 according to one embodiment of the present disclosure. The subtitle generation system 1 includes a subtitle generation device 10, a visual language model 20, and a large-scale language model (LLM) 30.

[0023] (Structure of the subtitle generation device)

[0024] The subtitle generation apparatus 10 is an apparatus for generating instances / subtitles of instances contained in a moving image based on the moving image. The subtitle generation apparatus 10 is, for example, a computer such as a server device. The subtitle generation apparatus 10 can be a general-purpose computer or a dedicated computer specifically for the functions of the subtitle generation apparatus 10. The subtitle generation apparatus 10 can also be a server device belonging to a cloud computing system. In one embodiment, such as... Figure 1 As shown, the subtitle generation device 10 includes an input / output interface 11, a communication interface 12, a control unit 13, and a storage unit 14.

[0025] The input / output interface 11 includes input devices such as a keyboard and mouse, and output devices such as a display, for inputting and outputting information between the user operating the subtitle generation device 10 and the user. The input / output interface 11 also includes a machine connection interface for data transmission between the user and other devices connected to the subtitle generation device 10. The machine connection interface includes interfaces conforming to various standards such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), IEEE 1394, and SATA (Serial Advanced Technology Attachment).

[0026] Communication interface 12 comprises hardware and software for sending and receiving information with other information processing devices. Communication interface 12 includes interfaces corresponding to wired and wireless communication. The wired communication interface 12 includes interfaces based on wired LAN standards such as Ethernet or IEEE 802.3. The wireless communication interface includes interfaces based on wireless communication standards such as 3G, LTE (Long Term Evolution), 5G, and Wi-Fi. The subtitle generation device 10 can communicate with the visual language model 20 and / or the large-scale language model 30 via communication interface 12.

[0027] The control unit 13 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or GPU (graphics processing unit), or a dedicated processor for specific processing. The dedicated circuit is, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The control unit 13 controls the various parts of the subtitle generation apparatus 10 while performing processing related to the operation of the subtitle generation apparatus 10. (Refer to the following...) Figure 2 The flowchart illustrates the processing performed by the control unit 13.

[0028] The storage unit 14 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of them. The semiconductor memory is, for example, RAM (random access memory) or ROM (read-only memory). RAM is, for example, SRAM (static random access memory) or DRAM (dynamic random access memory). ROM is, for example, EEPROM (electrically erasable programmable read-only memory). The storage unit 14 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 14 stores the program and data used in the operation of the subtitle generation device 10, as well as the data obtained through the operation of the subtitle generation device 10. The storage unit 14 may store a program 14a executed by the control unit 13, and a dynamic image file 14b, which stores a dynamic image that becomes the object of the instance / subtitle generated by the control unit 13.

[0029] The function of the subtitle generation device 10 is achieved by the processor, which corresponds to the control unit 13, executing program 14a. That is, the function of the subtitle generation device 10 is implemented by software. Program 14a enables the computer to function as the subtitle generation device 10 by causing the computer to execute the actions of the subtitle generation device 10. In other words, the computer functions as the subtitle generation device 10 by executing the actions of the subtitle generation device 10 according to program 14a.

[0030] In this embodiment, program 14a can be pre-recorded on a computer-readable recording medium. Computer-readable recording media include non-transitory computer-readable media, such as magnetic recording devices, optical discs, optical-magnetic recording media, or semiconductor memory. For example, the program can be circulated by selling, transferring, or lending portable recording media such as DVDs (digital versatile discs) or CD-ROMs (compact disc read-only memory) containing the program. Alternatively, program 14a can be circulated by pre-storing it in the memory of an external server and sending program 14a from the external server to other computers. Furthermore, program 14a can also be provided as a program product.

[0031] (Visual language models and large-scale language models)

[0032] Visual language model 20 is a trained artificial intelligence model corresponding to textual and image information, capable of comprehensively processing visual information such as images and / or animations and linguistic information such as text. Visual language model 20 can accept image input and generate captions describing the image content. Visual language model 20 includes, for example, Llama (large language model meta AI) 3.2, but is not limited to it. As visual language model 20, it can utilize any machine learning model corresponding to visual information and text.

[0033] Large-scale language model 30 is a pre-learned artificial intelligence model that has trained on a large amount of text data. Large-scale language model 30 is capable of text generation, query response, and text summarization. It includes models such as GPT (Generative Pre-trained Transformer) and BERT (Bidirectional Encoder Representations from Transformers). However, large-scale language model 30 is not limited to these and can utilize any large-scale language model.

[0034] The visual language model 20 and the large-scale language model 30 can be mounted on a server device located either locally or remotely, capable of communicating with the subtitle generation device 10. Alternatively, the visual language model 20 and the large-scale language model 30 can be mounted on a server device belonging to a cloud computing system. The subtitle generation device 10 and the server device equipped with the visual language model 20 and the large-scale language model 30 can communicate via a communication network such as a network. Alternatively, the visual language model 20 and the large-scale language model 30 can also be mounted on the same computer as the subtitle generation device 10. Furthermore, the visual language model 20 and the large-scale language model 30 can be the same multimodal model (multimodal LLM) capable of processing both images and text.

[0035] (Example / Subtitle Generation Processing)

[0036] The following is for reference Figure 2 as well as Figure 3 The example / subtitle generation process performed by the control unit 13 of the subtitle generation device 10 will be described. Figure 3 Using simple graph pairs Figure 2 The flowchart describes the processing flow of each step.

[0037] First, the control unit 13 acquires the moving image 40 from outside the subtitle generation device 10 or from the moving image file 14b stored in the storage unit 14 (S101). The moving image 40 acquired by the control unit 13 can be any form including MP4 (MPEG-4 Part 14), AVI (Audio Video Interleave), WebM, and AVCHD (Advanced Video Coding High Definition) (registered trademark).

[0038] The control unit 13 segments the motion image 40 into multiple segments based on scene transitions (S102). The control unit 13 can determine a scene transition when the entire motion image changes, and segment the motion image before and after that transition. More specifically, the control unit 13 can calculate the difference in pixel values ​​between consecutive frames 41, and determine a scene transition if the difference is greater than a predetermined value. Alternatively, the control unit 13 can calculate a histogram of the color distribution of each frame 41, and determine a scene transition if the difference in the histogram between consecutive frames is greater than a predetermined value. Furthermore, the control unit 13 can use a machine learning model that has learned scene transition patterns using a large amount of animation data to determine scene transitions. For example, as a method for detecting scene transitions in animation, a learning model called AutoShot can be used. Each segment of the motion image contains multiple frames 41.

[0039] The control unit 13 extracts instances 42a and 42b from one of the multiple frames 41 contained in each scene (S103). When detecting instances 42a and 42b based on the image of frame 41, a machine learning model for object detection can be used. For example, GroundingDINO (open vocabulary / zero-shot object detection model) can be used as the machine learning model for object detection. This machine learning model can detect and track objects within an image based on text-based instructions. Alternatively, the detection of instances 42a and 42b within a moving image can also use object detection models such as YOLO (You Only Look Once) or Faster R-CNN (Faster Region-based Convolutional Neural Network). The user of the subtitle generation device 10 can adjust the types of instances that the control unit 13 can detect by adjusting the instructions for the object detection model. For example, the user can instruct the object detection model to detect only people and animals.

[0040] In S103, the control unit 13 can select the central frame 41 from multiple frames 41 of the time series contained in the same segment and extract instances 42a and 42b contained in that frame 41. The control unit 13 preferably extracts all instances 42a and 42b that can be detected from the central frame. By extracting instances 42a and 42b from the temporally central frame 41, it is easier to extract instances 42a and 42b contained in more frames 41 of the scene. The central frame 41 does not necessarily have to be the exact center of the time series frames 41 within the scene. The central frame 41 can be a frame that is approximately in the center temporally within the scene. The frame 41 that becomes the object of extracting instances 42a and 42b in S103 is not limited to the approximately central frame 41; for example, it can also be the first frame 41 or the last frame 41.

[0041] The control unit 13 determines the bounding boxes 43a and 43b of the rectangles surrounding each instance 42a and 42b. The bounding boxes 43a and 43b are determined based on the segmented mask detected by object detection. The bounding boxes 43a and 43b are represented by specifying the coordinates of the top-left vertex and the bottom-right vertex of the rectangle.

[0042] The control unit 13 tracks instances 42a and 42b across multiple frames 41 within a segment. In S103, if instances 42a and 42b are detected in the frame 41 that is temporally central, the control unit 13 tracks instances 42a and 42b sequentially for the frames preceding and following it (S104). That is, the tracking of instances 42a and 42b can include both forward and reverse temporal tracking. Methods such as Kalman filtering and / or optical flow can be used in the tracking of instances 42a and 42b. Alternatively, the control unit 13 can use a machine learning model that has learned the tracking pattern of objects using a large amount of animation data to track instances 42a and 42b.

[0043] In each frame 41, the control unit 13 determines the bounding boxes 43a and 43b surrounding instances 42a and 42b. The bounding boxes 43a and 43b represent the position and size of instances 42a and 42b within the frame image. Furthermore, by comparing the position and size between frames 41, the bounding boxes 43a and 43b can be used for the purpose of tracking the movement of instances 42a and 42b.

[0044] The control unit 13 independently inputs the images of each frame 41 and the images of each bounding box 43a, 43b within frame 41 into the visual language model 20 to obtain the captions for each instance 42a, 42b of each frame 41. The caption for an instance within a frame image is called a frame level / instance / caption. The control unit 13 assigns the obtained frame level / instance / captions 44a, 44b to each instance 42a, 42b (S105). The control unit 13 sequentially stores the frame level / instance captions 44a, 44b for each instance 42a, 42b and for each frame 41 in the storage unit 14.

[0045] For the same instances 42a and 42b, the control unit 13 combines multiple frame levels / instances / subtitles 44a and 44b generated and assigned according to each frame 41 to generate an instance / subtitle for the entire segmented motion picture (S106). Specifically, the control unit 13 inputs multiple frame levels / instances / subtitles 44a and 44b into the large-scale language model 30 and combines these subtitles. The large-scale language model 30 generates an instance / subtitle summarizing the overall action based on each frame level / instance / subtitle 44a and 44b. Thus, frame levels / instances / subtitles 44a and 44b are combined into a single instance / subtitle, which summarizes the action of the entire segmented motion picture instance. The instance / subtitle for the entire segmented motion picture is set as motion picture levels / instances / subtitles 45a and 45b.

[0046] (Example)

[0047] Reference Figure 4 An example is given of combining frame-level / instance / caption into motion picture-level / instance / caption. Figure 4 In the example 42, there are three park benches, two people sitting on different benches, the sun, clouds, and several trees. The control unit 13 generates subtitles for each detectable example 42, but the following example uses the male who initially sits on the central bench as example 42.

[0048] First, when the segmented dynamic image contains frames from 0 to k, the control unit 13 detects instance 42 in the central frame located in the middle of the time series. The control unit 13 tracks the male instance 42 detected in the central frame for the frames before and after the time series. The control unit 13 assigns a bounding box 43 surrounding instance 42 to all frames. The control unit 13 sends the image within the bounding box 43 to the visual language model 20 to obtain the frame level / instance / caption of instance 42 for each frame.

[0049] exist Figure 4 In the example, the frame-level instance caption for frame 0 is as follows.

[0050] "The man is sitting on a bench in the center of the park. He is wearing a white shirt and brown trousers. He has his left hand on his forehead and is looking to his left."

[0051] Additionally, the frame level / example / subtitle of the central frame is as follows.

[0052] "The man stands up from his seat and walks to the right in front of the seat. He raises his left hand and waves."

[0053] Additionally, the frame level / example / subtitle of frame k is as follows.

[0054] "The man is located near the seat in the center and the seat on the right. The man is walking towards the right."

[0055] The control unit 13 inputs the frame levels / instances / captions from frame 0 to frame k into the large-scale language model 30 to generate motion picture levels / instances / captions. As an example, the motion picture levels / instances / captions for instance 42 are as follows.

[0056] "The man, wearing a white shirt and brown trousers, initially sat on a park bench. Next, he stood up and waved his left hand to the right. Then, he walked to the right and came to the space between the benches in the center and on the right."

[0057] Furthermore, the control unit 13 can detect all instances detectable in the central frame. For all instances detected in the central frame, the control unit 13 generates frame level / instance / caption for each frame and combines them to generate motion picture level / instance / caption. Thus, the control unit 13 can generate motion picture level / instance / caption for each instance, such as people, seats, the sun, clouds, and trees.

[0058] As described above, the caption generation apparatus 10 according to this disclosure can automatically generate captions for instances contained in a moving image upon input. The generated instances / captions can then be used, along with the moving image, as a training dataset for a visual language model during the learning phase. This visual language model is... Figure 1 There are 20 different visual language models. Therefore, visual language models can be trained to better recognize the actions of instances.

[0059] (Comparative example)

[0060] As a method for generating instance-specific captions, one approach considers enabling a visual language model to generate captions for the entire dynamic image, and then extracting captions for each instance from the generated captions. Figure 5 The image below illustrates an example of the processing flow of this method. The following section discusses... Figure 5 The process will be explained simply.

[0061] Acquire the dynamic image (S201). This step is related to... Figure 2 It is the same as S101.

[0062] Scene-based image segmentation (S202). This step is related to... Figure 2 It is the same as S102.

[0063] For each segmented scene, the overall dynamic image of the scene is input into the visual language model 20 to generate captions for the dynamic image (S203).

[0064] The captions of the animated images are input into a large-scale language model, and the captions for each instance are extracted (S204).

[0065] For example, regarding Figure 4 The same animated image, the caption of the animated image obtained in step S203 is as follows.

[0066] "This depicts a park during the day. The sun is in the center of the sky, surrounded by several clouds. Trees are on the left and right sides of the image. Three benches are arranged in the park, with a man sitting on the left and in the center. The man on the left bench remains seated. The man in the center stands up and walks to the right. The sun moves to the right. The clouds move to the left."

[0067] For example, the example / caption generated in step S204 is as follows.

[0068] "Sun: Located in the center of the sky, moving to the right."

[0069] Clouds: There are several clouds near the sun, moving to the left.

[0070] Trees: located on the right and left sides of the image.

[0071] Seating: Three seats will be arranged in the park.

[0072] The man in the seat on the left: sit in the seat on the left and remain still.

[0073] The man in the central seat: He initially sat in the central seat, then stood up and walked to the right.

[0074] Thus, in techniques that generate instance / caption based on captions of the entire image, only general information about the overall image can be utilized in most cases. Therefore, for each instance, only short phrases and / or abstract descriptions can be generated. Consequently, instance / captions generated from captions of the entire dynamic image do not focus on the detailed content of each instance. Furthermore, even if instance / captions generated by decomposing the captions of the entire image generated by inputting the dynamic image into the visual language model 20 are used as input data for training a visual language model with the same performance, it may not improve the instance recognition performance.

[0075] In the subtitle generation apparatus 10 and subtitle generation method according to the embodiments of this disclosure, frame-level / instance / subtitles are generated by providing the visual language model 20 with instances 42 extracted by the bounding box 43 in each frame. Furthermore, motion picture-level / instance / subtitles are generated by combining these frame-level / instance / subtitles. Therefore, compared to the comparative example described above, more detailed subtitles about instances can be obtained. Therefore, the instances / subtitles generated by the subtitle generation apparatus 10 of this disclosure can be appropriately used for training the visual language model during the learning phase to improve the accuracy of instance recognition.

[0076] Furthermore, the instances / subtitles generated by the subtitle generation apparatus 10 of this disclosure are not limited to machine learning applications of visual language models; they can be used for other purposes as long as they are explanatory texts generated from the animation instances.

[0077] This disclosure has been described based on the accompanying drawings and embodiments, but it should be noted that those skilled in the art can make various modifications and changes based on this disclosure. Therefore, it should be understood that such modifications and changes are included within the scope of this disclosure. For example, the functions included in each component or step can be reconfigured logically without contradiction, and multiple components or steps can be combined into one or divided.

[0078] [Postscript 1]

[0079] A subtitle generation apparatus for generating subtitles for instances within a moving image, wherein the apparatus comprises:

[0080] The acquisition unit acquires dynamic images; and

[0081] The control unit segments the acquired dynamic images based on scene transitions.

[0082] For each frame in the segmented scene, a frame detection instance is performed.

[0083] Track the above instances across multiple frames within the same scene.

[0084] Subtitles are assigned to instances in each frame of the same scene described above.

[0085] The multiple subtitles assigned to the same instance will be combined in each of the multiple frames contained in the same scene.

[0086] [Postscript 2]

[0087] According to the subtitle generation device described in Appendix 1, wherein,

[0088] The above-mentioned subtitles in the above example are used for learning machine learning models corresponding to text information and dynamic image information.

[0089] [Postscript 3]

[0090] According to the subtitle generation device described in Appendix 1 or 2, wherein,

[0091] The aforementioned control unit detects instances of frames that are located in the center of time among the multiple frames contained in each segmented scene.

[0092] [Postscript 4]

[0093] According to the subtitle generation device described in Appendix 3, among which,

[0094] The aforementioned control unit assigns subtitles to all instances that can be detected in the frame located in the center.

[0095] [Postscript 5]

[0096] According to the subtitle generation device described in any of Appendix 1 to 4, wherein,

[0097] The control unit uses bounding boxes to track the instances, which represent the regions that surround the instances in each of the frames.

[0098] [Postscript 6]

[0099] According to the subtitle generation device described in Appendix 5, in which,

[0100] The control unit uses a visual language model capable of generating subtitles from images to obtain the subtitles assigned to each instance.

[0101] [Postscript 7]

[0102] According to the subtitle generation device described in Appendix 6, in which,

[0103] The control unit inputs the image within the bounding box into the visual language model for each frame to obtain the subtitles.

[0104] [Postscript 8]

[0105] According to any of the appendices 1 to 7, the subtitle generation device, wherein,

[0106] The aforementioned control unit uses a large-scale language model to combine multiple subtitles assigned to the same instance across the aforementioned frames.

[0107] [Postscript 9]

[0108] A subtitle generation method, executed by a control unit of a subtitle generation device, the subtitle generation device generating subtitles for instances within a moving image, wherein the method includes:

[0109] Acquire dynamic images;

[0110] The obtained dynamic images are segmented based on scene migration.

[0111] For each segmented scene, detect an instance of one frame from among the multiple frames.

[0112] Track the above instances across multiple frames within the same scene;

[0113] Add captions to instances in each frame of the same scene described above; and

[0114] The multiple subtitles assigned to the same instance will be combined in each of the multiple frames contained in the same scene.

[0115] [Postscript 10]

[0116] According to the subtitle generation method described in Appendix 9, among which,

[0117] The above-mentioned subtitles in the above example are used for learning machine learning models corresponding to text information and dynamic image information.

[0118] [Postscript 11]

[0119] According to the subtitle generation method described in Appendix 9 or 10, among which,

[0120] The aforementioned control unit detects instances of frames that are located in the center of time among the multiple frames contained in each segmented scene.

[0121] [Postscript 12]

[0122] According to the subtitle generation method described in Appendix 11, among which,

[0123] The aforementioned control unit assigns subtitles to all instances that can be detected in the frame located in the center.

[0124] [Postscript 13]

[0125] According to the subtitle generation method described in any of the appendices 9 to 12, among which,

[0126] The aforementioned control unit uses a bounding box to track the aforementioned instances, which represents the region surrounding the instances in each of the aforementioned frames.

[0127] [Postscript 14]

[0128] According to the subtitle generation method described in Appendix 13, among which,

[0129] The aforementioned control unit uses a visual language model capable of generating subtitles from images to generate the aforementioned subtitles for each instance.

[0130] [Postscript 15]

[0131] According to the subtitle generation method described in Appendix 14, among which,

[0132] The control unit inputs the image within the bounding box into the visual language model for each frame to generate the subtitles.

[0133] [Postscript 16]

[0134] According to the subtitle generation method described in any of the appendices 9 to 15, among which,

[0135] The aforementioned control unit uses a large-scale language model to combine multiple subtitles assigned to the same instance for the aforementioned multiple frames.

[0136] [Postscript 17]

[0137] A program that causes a control unit of a captioning device for generating captions of instances within a moving image to perform processing, wherein the processing includes:

[0138] Acquire dynamic images;

[0139] The obtained dynamic images are segmented based on scene migration.

[0140] For each segmented scene, detect an instance of one frame from among the multiple frames.

[0141] Track the above instances across multiple frames within the same scene;

[0142] Add captions to instances in each frame of the same scene described above; and

[0143] The multiple subtitles assigned to the same instance will be combined in each of the multiple frames contained in the same scene.

[0144] [Postscript 18]

[0145] According to the procedure described in Appendix 17, in which,

[0146] The above-mentioned subtitles in the above example are used for learning machine learning models corresponding to text information and dynamic image information.

[0147] [Postscript 19]

[0148] According to the procedure described in Appendix 17 or 18, in which,

[0149] The aforementioned control unit detects instances of frames that are located in the center of time among the multiple frames contained in each segmented scene.

[0150] [Postscript 20]

[0151] According to the procedure described in any of Notes 17 to 19, in which,

[0152] The aforementioned control unit tracks the aforementioned instances by using bounding boxes, which represent the regions surrounding the instances in each of the aforementioned frames.

Claims

1. A subtitle generation apparatus for generating subtitles for instances within a moving image, wherein, The aforementioned subtitle generation device includes: The acquisition unit acquires dynamic images; and The control unit segments the acquired dynamic images based on scene transitions. For each frame in the segmented scene, a frame detection instance is performed. Track the above instances across multiple frames within the same scene. Subtitles are assigned to instances in each frame of the same scene described above. The multiple subtitles assigned to the same instance will be combined in each of the multiple frames contained in the same scene.

2. The subtitle generation device according to claim 1, wherein, The above-mentioned subtitles in the above example are used for learning machine learning models corresponding to text information and dynamic image information.

3. The subtitle generation device according to claim 1, wherein, The aforementioned control unit detects instances of frames that are located in the center of time among the multiple frames contained in each segmented scene.

4. The subtitle generation device according to claim 3, wherein, The aforementioned control unit assigns subtitles to all instances that can be detected in the frame located in the center.

5. The subtitle generation device according to claim 1, wherein, The control unit uses bounding boxes to track the instances, which represent the regions that surround the instances in each of the frames.

6. The subtitle generation apparatus according to claim 5, wherein, The control unit uses a visual language model capable of generating subtitles from images to obtain the subtitles assigned to each instance.

7. The subtitle generation apparatus according to claim 6, wherein, The control unit inputs the image within the bounding box into the visual language model for each frame to obtain the subtitles.

8. The subtitle generation apparatus according to claim 1, wherein, The aforementioned control unit uses a large-scale language model to combine multiple subtitles assigned to the same instance for the aforementioned multiple frames.

9. A subtitle generation method, executed by a control unit of a subtitle generation device, the subtitle generation device generating subtitles for instances within a moving image, wherein, The above-mentioned subtitle generation methods include: Acquire dynamic images; The obtained dynamic images are segmented based on scene migration. For each segmented scene, detect an instance of one frame from among the multiple frames. Track the above instances across multiple frames within the same scene; Add captions to instances in each frame of the same scene described above; and The multiple subtitles assigned to the same instance will be combined in each of the multiple frames contained in the same scene.

10. The subtitle generation method according to claim 9, wherein, The above-mentioned subtitles in the above example are used for learning machine learning models corresponding to text information and dynamic image information.

11. The subtitle generation method according to claim 9, wherein, The aforementioned control unit detects instances of frames that are located in the center of time among the multiple frames contained in each segmented scene.

12. The subtitle generation method according to claim 11, wherein, The aforementioned control unit assigns subtitles to all instances that can be detected in the frame located in the center.

13. The subtitle generation method according to claim 9, wherein, The control unit uses bounding boxes to track the instances, which represent the regions that surround the instances in each of the frames.

14. The subtitle generation method according to claim 13, wherein, The aforementioned control unit uses a visual language model capable of generating subtitles from images to generate the aforementioned subtitles for each instance.

15. The subtitle generation method according to claim 14, wherein, The control unit inputs the image within the bounding box into the visual language model for each frame to generate the subtitles.

16. The subtitle generation method according to claim 9, wherein, The aforementioned control unit uses a large-scale language model to combine multiple subtitles assigned to the same instance for the aforementioned multiple frames.

17. A non-transitory computer-readable medium storing a program that causes a control unit of a subtitle generation apparatus for generating subtitles of instances within a moving image to perform processing, wherein... The process includes: Acquire dynamic images; The obtained dynamic images are segmented based on scene migration. For each segmented scene, detect an instance of one frame from among the multiple frames. Track the above instances across multiple frames within the same scene; Add captions to instances in each frame of the same scene described above; and The multiple subtitles assigned to the same instance will be combined in each of the multiple frames contained in the same scene.

18. The non-transitory computer-readable medium according to claim 17, wherein, The above-mentioned subtitles in the above example are used for learning machine learning models corresponding to text information and dynamic image information.

19. The non-transitory computer-readable medium according to claim 17, wherein, The aforementioned control unit detects instances of frames that are located in the center of time among the multiple frames contained in each segmented scene.

20. The non-transitory computer-readable medium according to claim 17, wherein, The aforementioned control unit tracks the aforementioned instances by using bounding boxes, which represent the regions surrounding the instances in each of the aforementioned frames.

Images