Text generation device, text generation method, and program

Abstract

A text generation device according to the present disclosure: acquires an image sequence composed of a plurality of captured images; detects an object from each of the plurality of captured images; extracts a representative image from the image sequence; superimposes, on the representative image, an annotation related to the object in the captured image other than the representative image, to generate an annotated representative image; and uses the annotation representative image to generate an output text related to the object.

Description

Text generation device, text generation method, and program

[0001] The present disclosure relates to a text generation device, a text generation method, and a program.

[0002] Techniques for generating text based on images have been developed. For example, Non-Patent Document 1 discloses a technique for generating answer text in response to input of a sequence of images and a question related to the contents of the sequence of images. More specifically, in Non-Patent Document 1, several key images are selected from the sequence of images, and an answer is generated based on the question and features extracted from the selected key images.

[0003] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal, "Self-Chained Image-Language Model for Video Localization and Question Answering," [online], arXiv, May 11, 2023, [accessed July 9, 2024], Internet <URL: https: / / arxiv.org / pdf / 2305.06988v1>

[0004] In Non-Patent Document 1, information contained in images other than the key image is not used to generate an answer. The present disclosure has been made in light of this problem, and one of its objectives is to provide a new technology for generating text based on an image sequence.

[0005] The text generation device according to the present disclosure includes an acquisition means for acquiring an image sequence consisting of a plurality of captured images, a detection means for detecting an object from each of the plurality of captured images, a first generation means for extracting a representative image from the image sequence and generating an annotation representative image by superimposing annotations related to the object in the captured images other than the representative image on the representative image, and a second generation means for generating output text related to the object using the annotation representative image.

[0006] A text generation method according to the present disclosure is executed by a computer and includes an acquisition step of acquiring an image sequence consisting of a plurality of captured images, a detection step of detecting an object from each of the captured images, a first generation step of extracting a representative image from the image sequence and generating an annotation representative image by superimposing annotations related to the object in the captured images other than the representative image on the representative image, and a second generation step of generating output text related to the object using the annotation representative image.

[0007] The program according to the present disclosure causes a computer to execute an acquisition step of acquiring an image sequence consisting of a plurality of captured images; a detection step of detecting an object from each of the plurality of captured images; a first generation step of extracting a representative image from the image sequence and generating an annotated representative image by superimposing annotations related to the object in the captured images other than the representative image onto the representative image; and a second generation step of generating output text related to the object using the annotated representative image.

[0008] According to the present disclosure, a new technique for generating text based on a sequence of images is provided.

[0009] 1 is a diagram illustrating an overview of the operation of a text generator. FIG. 2 is a block diagram illustrating the functional configuration of a text generator. FIG. 3 is a block diagram illustrating the hardware configuration of a computer that realizes the text generator. FIG. 4 is a flowchart illustrating the flow of processing executed by the text generator. FIG. 5 is a diagram illustrating the configuration of object tracking information. FIG. 6 is a diagram illustrating an annotation representing the history of an object's position. FIG. 7 is a second diagram illustrating an annotation representing the history of an object's position. FIG. 7 is a diagram illustrating an annotation realized by the history of an object region. FIG. 8 is a diagram illustrating an annotation realized by the history of skeleton information. FIG. 9 is a diagram illustrating an annotation realized by a combination of the history of an object's position and the history of an object region. FIG. 10 is a diagram illustrating an annotation realized by a combination of the history of an object's position and the history of skeleton information. FIG. 11 is a diagram illustrating an annotation realized by a combination of the history of an object region and the history of skeleton information. FIG. 12 is a diagram illustrating an annotation realized by a combination of the history of an object's position, the history of an object region, and the history of skeleton information. FIG. 13 is a diagram illustrating the operation of a text generation model. FIG. 14 is a diagram illustrating an annotation representative image including a map image.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0011] <Overview> Fig. 1 is a diagram illustrating an example of an overview of the operation of text generator 2000. Note that Fig. 1 is a diagram for facilitating understanding of the overview of text generator 2000, and the operation of text generator 2000 is not limited to the operation shown in Fig. 1.

[0012] The text generator 2000 acquires an image sequence 10. The image sequence 10 is time-series data made up of a plurality of captured images 20. For example, the image sequence 10 is a sequence of image frames constituting video data generated by a video camera. In this case, the captured images 20 are image frames constituting the video data.

[0013] The text generator 2000 detects a predetermined object from each of the multiple captured images 20 included in the image sequence 10. For example, the text generator 2000 detects a predetermined type of object, such as a person or a vehicle, from the captured image 20. In this case, multiple objects may be detected from one captured image 20. Alternatively, for example, the text generator 2000 may detect a specific object, such as a specific person or a specific vehicle, from the captured image 20.

[0014] The text generator 2000 extracts one or more of the captured images 20 included in the image sequence 10 as a representative image 50. For example, the text generator 2000 extracts a captured image 20 that exists in a predetermined position (e.g., the end or the beginning) of the image sequence 10 as the representative image 50.

[0015] The text generator 2000 generates annotations 40 based on the results of object detection performed on a plurality of captured images 20, and generates an annotated representative image 60 by superimposing the annotations 40 on a representative image 50. The annotations 40 represent information about changes in objects detected from the image sequence 10. For example, the annotations 40 represent the history of the object's position in the image sequence 10 (in other words, the trajectory of the object's position). Here, the information about changes in an object represented by the annotations 40 may include not only information about how the object has changed, but also information about the fact that the object has not changed.

[0016] The text generator 2000 generates output text 70, which is text relating to the content of the annotated representative image 60. For example, the output text 70 may include a description of the behavior of the objects in the image sequence 10.

[0017] <Example of Effects> According to the text generator 2000, an annotated representative image 60 is generated by superimposing an annotation 40, which represents information about changes in objects in the image sequence 10, on a representative image 50 extracted from the image sequence 10. Then, an output text 70, which is text related to the content of the annotated representative image 60, is generated. In this way, the text generator 2000 provides a new technique for generating text from images.

[0018] Here, the annotated representative image 60 includes, as annotations 40, information about changes in objects detected from the captured images 20 other than the representative image 50. Therefore, by generating text related to the content of the annotated representative image 60, it is possible to obtain text related to changes in objects in the image sequence 10, taking into account not only the appearance of the object in the representative image 50 but also the appearance of the object in the captured images 20 other than the representative image 50. Therefore, by obtaining text related to the content of the annotated representative image 60 using the text generator 2000, it is possible to obtain text that more accurately represents changes in objects in the image sequence 10, compared to obtaining text related to the content of only the representative image 50.

[0019] The text generator 2000 of this embodiment will be described in more detail below.

[0020] <Example of Functional Configuration> Fig. 2 is a block diagram illustrating an example of the functional configuration of the text generator 2000. For example, the text generator 2000 includes an acquisition unit 2020, a detection unit 2040, a first generation unit 2060, and a second generation unit 2080. The acquisition unit 2020 acquires an image sequence 10. The detection unit 2040 detects a predetermined object from each of the multiple captured images 20 included in the image sequence 10. The first generation unit 2060 extracts a representative image 50 from the image sequence 10 and generates an annotation representative image 60 by superimposing an annotation 40 on the representative image 50. The second generation unit 2080 generates output text 70 using the annotation representative image 60.

[0021] <Example of Hardware Configuration> Each functional component of text generator 2000 may be realized by hardware that realizes that functional component (e.g., a hardwired electronic circuit, etc.), or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it). Below, a case where each functional component of text generator 2000 is realized by a combination of hardware and software will be further described.

[0022] 3 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the text generator 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. The computer 1000 may be a dedicated computer designed to realize the text generator 2000, or may be a general-purpose computer.

[0023] For example, by installing a predetermined application on computer 1000, each function of text generator 2000 is realized on computer 1000. The application is configured with a program for realizing each functional component of text generator 2000. Note that any method for acquiring the program is possible. For example, the program can be acquired from a storage medium (such as a DVD (Digital Versatile Disc) or a USB (Universal Serial Bus) memory) on which the program is stored. Alternatively, the program can be acquired by downloading it from a server device that manages the storage device on which the program is stored.

[0024] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0025] The processor 1040 is one of various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0026] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0027] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0028] Storage device 1080 stores programs (programs that realize the above-mentioned applications) that realize the various functional components of text generator 2000. Processor 1040 reads these programs into memory 1060 and executes them to realize the various functional components of text generator 2000.

[0029] The text generator 2000 may be realized by one computer 1000 or by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from each other.

[0030] <Processing Flow> Figure 4 is a flowchart illustrating the processing flow executed by the text generator 2000. The acquisition unit 2020 acquires the image sequence 10 (S102). The detection unit 2040 detects a predetermined object from each of the multiple captured images 20 (S104). The first generation unit 2060 extracts a representative image 50 from the image sequence 10 (S106). The second generation unit 2080 generates an annotation representative image 60 by superimposing the annotation 40 on the representative image 50 (S108). The second generation unit 2080 generates output text 70 using the annotation representative image 60 (S110).

[0031] <Acquiring Image Sequence 10: S102> The acquisition unit 2020 acquires the image sequence 10 (S102). Any method can be used to acquire the image sequence 10. For example, the image sequence 10 is stored in any storage unit in a format that can be acquired by the text generator 2000. In this case, the acquisition unit 2020 acquires the image sequence 10 by reading it from the storage unit. Another example is that the image sequence 10 is transmitted to the text generator 2000 from another device. In this case, the acquisition unit 2020 acquires the image sequence 10 by receiving the image sequence 10 transmitted from the other device. The device that transmits the image sequence 10 to the text generator 2000 is, for example, the camera that generated the image sequence 10.

[0032] The acquisition unit 2020 may acquire multiple image sequences 10. In this case, the multiple image sequences 10 may be image sequences generated by the same camera, or may be image sequences generated by different cameras. In the latter case, it is preferable that the image sequences 10 are acquired in association with the identifiers of the cameras that generated the image sequences 10.

[0033] When the multiple image sequences 10 are generated by different cameras, an object may be captured by multiple cameras. For example, a situation in which cameras are installed both inside and outside a building may be envisioned. Another example may be a situation in which a camera is installed in each of multiple locations within a building (such as multiple rooms or corridors). Another example may be a situation in which multiple cameras capture the same location from different angles.

[0034] The text generator 2000 may divide an image sequence generated by one camera and treat each of the multiple image sequences obtained by the division as an image sequence 10. For example, by dividing 10 minutes of video data generated by one camera into 10 segments, ten 1-minute image sequences 10 are obtained.

[0035] <Detecting an Object: S104> The detection unit 2040 detects a predetermined object from each of the captured images 20 included in the image sequence 10 (S104). The object to be detected from the captured image 20 may be fixed and determined in advance, or may be specified by the user of the text generator 2000.

[0036] Assume that the detection unit 2040 is configured to detect a predetermined type of object from the captured image 20. A specific method for detecting a predetermined type of object from an image is arbitrary. For example, the detection unit 2040 has a machine learning model (e.g., a neural network) that is pre-trained to detect a predetermined type of object from an image. Hereinafter, this model will be referred to as an object detection model.

[0037] For example, the object detection model is configured to output, in response to an input image, information about an image region representing a predetermined type of object contained in the image. Hereinafter, the image region representing the object will be referred to as an object region. Furthermore, the information about the object region will be referred to as object region information.

[0038] The object region is, for example, an image region that represents the inside of a circumscribing rectangle of an object. The object region information indicates, for example, the position of a specific point (e.g., the upper left corner) of the object region and the size (e.g., width and height) of the object region. The object region information may further include the object region itself.

[0039] The detection unit 2040 inputs each of the multiple captured images 20 included in the image sequence 10 into the object detection model. As a result, the detection unit 2040 obtains object region information for each captured image 20. Note that if the captured image 20 includes multiple objects of a predetermined type, object region information is obtained for each object.

[0040] Here, the object detection process may be performed on all captured images 20 included in the image sequence 10, or on only some of the captured images 20 included in the image sequence 10. In the latter case, for example, the detection unit 2040 extracts captured images 20 from the image sequence 10 at a rate of one captured image 20 per predetermined number of images, and detects an object from the extracted captured images 20.

[0041] As described above, the detection unit 2040 may be configured to detect only a specific object (e.g., a specific person). In this case, the detection unit 2040 uses information representing the characteristics of the object to be detected (hereinafter, referred to as feature information). The feature information is, for example, a feature amount obtained from an image of the object to be detected. The feature information is, for example, stored in advance in an arbitrary storage unit in a form that can be acquired from the text generator 2000. Alternatively, for example, the feature information may be provided by the user of the text generator 2000.

[0042] For example, the detection unit 2040 identifies an object region representing the object to be detected by comparing the object region detected by the object detection model described above with the features represented by the feature information. Alternatively, for example, the object detection model may be configured to detect only objects having specified features. In this case, the object detection model is configured to receive the feature information as input in addition to the captured image 20. The detection unit 2040 inputs the captured image 20 and the feature information of the object to be detected into the object detection model. As a result, object region information of the object to be detected is output from the object detection model.

[0043] The detection unit 2040 identifies the object using the object region information obtained from each of the multiple captured images 20. That is, it is determined whether or not objects detected from different captured images 20 are the same object. Various methods such as tracking can be used to identify the object.

[0044] As a result of identifying the objects, for example, the detection unit 2040 generates information indicating the history of the object region of each object (hereinafter referred to as object tracking information). Fig. 5 is a diagram illustrating an example of the configuration of the object tracking information. In Fig. 5, the object tracking information 80 indicates an identifier 82 and a history 84. The identifier 82 is an identifier uniquely assigned to each object.

[0045] The history 84 indicates one or more pairs of a time point 86 and object region information 88 for an object assigned an identifier indicated in the corresponding identifier 82. The time point 86 indicates the time point at which the captured image 20 was generated when the corresponding object region information 88 was generated. The object region information 88 indicates the position and size of the object region. In FIG. 5, p1 and p2 indicate the position of the object region. w1 and w2 indicate the width of the object region. h1 and h2 indicate the height of the object region.

[0046] When only a specific object is detected from the captured image 20, the object tracking information 80 may indicate only information related to the specific object. In this case, an identifier does not need to be assigned to the object.

[0047] As described above, the text generator 2000 may acquire a plurality of image sequences 10. In this case, the detection unit 2040 performs object detection for each image sequence 10.

[0048] The detection unit 2040 may generate object tracking information 80 for each image sequence 10, or may generate one object tracking information 80 by integrating the results of object detection performed on multiple image sequences 10. In the latter case, the history 84 includes information about the image sequence 10 in which the object was detected, in addition to the time point 86 and object region information 88. The information about the image sequence 10 in which the object was detected is, for example, the identifier of the camera that generated the image sequence 10. In this case, the history 84 indicates the camera identifier, the time point, and the object region information in association with each other.

[0049] <Extraction of Representative Image 50: S106> The first generation unit 2060 extracts a representative image 50 from the image sequence 10 (S106). For example, conditions for the captured images 20 to be extracted as the representative images 50 (hereinafter, extraction conditions) are determined in advance. In this case, the detection unit 2040 extracts, from the image sequence 10, captured images 20 that meet the extraction conditions as the representative images 50.

[0050] Various conditions can be used as the extraction conditions. For example, the extraction conditions indicate a condition regarding a position in the image sequence 10, such as "the end of the image sequence 10" or "the beginning of the image sequence 10." In this way, when the extraction conditions are conditions regarding a position in the image sequence 10, the detection unit 2040 extracts, as the representative image 50, the captured image 20 that exists in the image sequence 10 at the position indicated by the extraction conditions.

[0051] If it is desired to focus on a specific object, the extraction conditions may further include a condition related to the object. For example, extraction conditions such as "the end of the time range in the image sequence 10 in which the specific object appears" or "the beginning of the time range in the image sequence 10 in which the specific object appears" can be used. The time range in the image sequence 10 in which the specific object appears can be identified by referring to the object tracking information 80 described above.

[0052] The number of representative images 50 extracted from one image sequence 10 is not limited to one. For example, the extraction condition may further indicate the number of representative images 50 to be extracted from the image sequence 10. For example, assume that the extraction condition is "the end of each partial range obtained by dividing the time range of the entire image sequence 10 into N parts" (N is a positive integer). In this case, the detection unit 2040 divides the time range of the entire image sequence 10 into N equal parts and extracts, as representative images 50, the captured images 20 located at the end of each of the N partial time ranges obtained by the division. In this way, N representative images 50 are extracted from the image sequence 10. Note that if "the time range of the entire image sequence 10" is replaced with "a time range in which a specific object appears," N representative images 50 can be obtained from the time range of the image sequence 10 in which the specific object appears.

[0053] Alternatively, the extraction condition may indicate a condition for the time interval for extracting the representative image 50. For example, an extraction condition such as "the end of each partial range obtained by dividing the time range of the entire image sequence 10 into intervals of length T" (T is a positive integer) can be adopted (T is a positive integer). In this case, too, "the time range in which a specific object appears" may be used instead of "the time range of the entire image sequence 10."

[0054] The method of extracting the representative image 50 from the image sequence 10 is not limited to the method using extraction conditions. For example, the first generation unit 2060 extracts one or more key frames from the image sequence and uses the extracted key frames as the representative image 50.

[0055] A specific method for extracting keyframes from an image sequence is to use, for example, a machine learning model (such as a neural network) that has been trained in advance to extract keyframes from an image sequence. Hereinafter, this model will be referred to as a representative image extraction model. For example, LOCALIZER, which is disclosed in Non-Patent Document 1, can be used as the representative image extraction model.

[0056] The first generation unit 2060 inputs the image sequence 10 to the representative image extraction model. The representative image extraction model extracts one or more key frames in response to the input of the image sequence 10. The first generation unit 2060 uses each key frame extracted from the image sequence 10 as a representative image 50.

[0057] The number of key frames extracted from the representative image extraction model may be determined in advance, or may be dynamically determined by the representative image extraction model.

[0058] When the acquisition unit 2020 acquires a plurality of image sequences 10 , the detection unit 2040 extracts one or more representative images 50 from each of the plurality of image sequences 10 .

[0059] <Generation of Annotated Representative Image 60: S106> The first generation unit 2060 generates annotations 40 based on the results of object detection for each captured image 20, and superimposes the annotations 40 on the representative image 50 to generate the annotation representative image 60 (S106). Some specific examples of the annotations 40 will be given below.

[0060] <<Specific Example 1 of Annotation 40>> For example, the annotation 40 represents the position history of an object (in other words, the position trajectory). By including the position history of the object as the annotation 40 in the annotated representative image 60, it is possible to incorporate information about the position of the object obtained from multiple captured images 20 into one captured image 20.

[0061] 6 is a diagram illustrating an example of an annotation 40 that indicates the history of object positions. In the example of FIG. 6, the object detected from the captured image 20 is a person. The representative image 50 is the captured image 20 located at the end of the image sequence 10. The annotation 40-1 indicates the position of the detected object with a cross mark for each captured image 20 from the captured image 20 located at the beginning of the image sequence 10 to the captured image 20 located immediately before the representative image 50 in the image sequence 10.

[0062] The position of an object detected from each captured image 20 is indicated in object region information 88 of object tracking information 80. Therefore, the first generation unit 2060 superimposes a predetermined mark on each position of the object region indicated in the object region information 88 on the representative image 50. As a result, an annotation 40 indicating the history of the object's position is superimposed on the representative image 50, and an annotated representative image 60 is generated.

[0063] If multiple predetermined objects are detected in the image sequence 10, the first generation unit 2060 generates, for each object, an annotation 40 that indicates the position history of the object. In this case, the first generation unit 2060 may change the characteristics (e.g., color or shape) of the annotation 40 for each object.

[0064] The annotation 40 representing the position history of an object is not limited to a mark representing the position of the object in each captured image 20. Fig. 7 is a second diagram illustrating an example of the annotation 40 representing the position history of an object. In Fig. 7, the annotation 40-2 is represented by a polygonal line generated by connecting the positions of the object in each captured image 20 in chronological order. Here, by utilizing various interpolation processes, the position history of the object may be represented by a curved line instead of a polygonal line.

[0065] The annotation 40 may further indicate the direction of movement of the object. For example, in Figure 7, the annotation 40 may be realized as an arrow to indicate the direction of movement of the object.

[0066] <<Specific Example 2 of Annotation 40>> Alternatively, for example, an object region detected from one or more captured images 20 other than the representative image 50 may be used as the annotation 40. That is, an image region of the same object as the object detected in the representative image 50 is extracted from the captured images 20 other than the representative image 50, and the extracted image region is used as the annotation 40. The object region extracted from one or more captured images 20 other than the representative image 50 can also be referred to as a history of the object region. By superimposing the annotation 40 representing the history of the object region on the representative image 50, visual information of the object obtained from each of the multiple captured images 20 can be incorporated into a single annotated representative image 60.

[0067] 8 is a diagram illustrating an example of an annotation 40 realized by the history of an object region. In the example of FIG. 8, a representative image 50 is the last captured image 20 in an image sequence 10. Annotation 40-3 is an object region extracted from each of the five captured images 20 other than the representative image 50.

[0068] Note that the object region superimposed on the representative image 50 as the annotation 40 may be processed so as to be distinguishable from the object region originally included in the representative image 50. For example, the first generation unit 2060 reduces the opacity of the annotation 40 below the default value (i.e., makes the annotation 40 semi-transparent). As a result, in the annotation representative image 60, the annotation 40 appears lighter than the object region originally included in the representative image 50. Note that, for convenience of illustration, the fact that the opacity of the annotation 40 is lower than the default value is represented by a dotted line in FIG. 8 .

[0069] By varying the opacity among the multiple object regions included in the annotation 40, the order of changes in the object regions may be easily recognized in the annotation 40. For example, the first generation unit 2060 reduces the opacity of an object region extracted from the captured image 20 at an earlier generation point in time. In this way, the opacity of the object region increases (i.e., the object region becomes darker) as time passes. Note that this process of varying the opacity of the annotation 40 from the opacity of the representative image 50 can also be applied to annotations 40 that represent the position history of an object, as described above, and annotations 40 that represent the history of skeletal information of an object, as described below.

[0070] The object regions constituting the annotation 40 may be extracted from all of the captured images 20, or may be extracted from some of the captured images 20. In the latter case, the probability that the object regions will be separated to some extent increases, and therefore the probability that the object regions in the annotation 40 will overlap can be reduced.

[0071] For example, the first generation unit 2060 extracts captured images 20 at a predetermined interval from the image sequence obtained by excluding the representative image 50 from the image sequence 10, and extracts an object region to be included in the annotation 40 from each extracted captured image 20. For example, if the predetermined interval is 5, the first generation unit 2060 extracts captured images 20 at a frequency of 1 in 5 from the image sequence obtained by excluding the representative image 50 from the image sequence 10.

[0072] <<Specific Example 3 of Annotation 40>> Alternatively, for example, skeletal information of an object detected from one or more captured images 20 other than the representative image 50 may be used as the annotation 40. The first generation unit 2060 extracts image regions of the same object as the object detected in the representative image 50 from the captured images 20 other than the representative image 50, and generates skeletal information from each extracted image region. The first generation unit 2060 then uses each piece of generated skeletal information as the annotation 40. The skeletal information extracted from one or more captured images 20 can also be referred to as a history of the skeletal information. By superimposing the annotation 40 representing the history of the skeletal information on the representative image 50, information representing changes in the posture of the object in the image sequence 10 can be incorporated into a single annotated representative image 60.

[0073] In addition to extracting skeletal information of an object from captured images 20 other than the representative image 50 , the first generating unit 2060 may also extract skeletal information of an object from the representative image 50 and add it to the annotation 40 .

[0074] The skeleton information includes a plurality of feature points of the object. A feature point here is a point that represents the position of a specific part of the object. If the object is a person, a joint point or the like may be used as a feature point. If the object is a vehicle, a specific part such as a tire, bumper, or window may be used as a feature point. The skeleton information may also include a link between two adjacent feature points.

[0075] The skeletal information can be generated, for example, by using a machine learning model (such as a neural network) that has been trained in advance to extract skeletal information of an object from an image region of the object. In this case, the first generator 2060 generates the skeletal information by inputting the object region obtained from the captured image 20 into the model.

[0076] 9 is a diagram illustrating an example of an annotation 40 realized by the history of skeletal information. In the example of FIG. 9, a representative image 50 is the last captured image 20 in an image sequence 10. Annotation 40-4 represents skeletal information extracted from each of the five captured images 20 other than the representative image 50.

[0077] Here, similarly to the history of the object region, the history of the skeleton information may be extracted from all the captured images 20 or from some of the captured images 20. In the latter case, it is possible to reduce the probability that pieces of skeleton information overlap with each other in the annotation 40.

[0078] <<Combination of Specific Examples>> Two or more of the annotations 40 of Specific Example 1 to Specific Example 3 described above may be combined. For example, the first generation unit 2060 combines the annotation 40 of Specific Example 1 with the annotation 40 of Specific Example 2. In this case, the object position history and the object area history are superimposed as the annotation 40 on the representative image 50.

[0079] 10 is a diagram illustrating an example of an annotation 40 realized by combining the object position history and the object area history. The annotation representative image 60 in Fig. 10 includes both the annotation 40-1 in Fig. 6 and the annotation 40-3 in Fig. 8.

[0080] Alternatively, for example, the first generating unit 2060 combines the annotation 40 of Specific Example 1 with the annotation 40 of Specific Example 3. In this case, the position history of the object and the skeleton information history are superimposed on the representative image 50 as the annotation 40.

[0081] 11 is a diagram illustrating an example of an annotation 40 realized by combining the object position history and the skeleton information history. The annotation representative image 60 in FIG. 11 includes both the annotation 40-1 in FIG. 6 and the annotation 40-4 in FIG. 9.

[0082] Alternatively, for example, the first generating unit 2060 combines the annotation 40 of specific example 2 with the annotation 40 of specific example 3. In this case, the history of the object region and the history of the skeleton information are superimposed on the representative image 50 as the annotation 40.

[0083] 12 is a diagram illustrating an example of an annotation 40 realized by combining the history of the object region and the history of the skeleton information. The annotation representative image 60 in FIG. 12 includes both the annotation 40-3 in FIG. 8 and the annotation 40-4 in FIG. 9.

[0084] Alternatively, for example, the first generating unit 2060 combines the annotation 40 of Specific Example 1, the annotation 40 of Specific Example 2, and the annotation 40 of Specific Example 3. In this case, the object position history, the object area history, and the skeleton information history are superimposed as the annotation 40 on the representative image 50.

[0085] 13 is a diagram illustrating an example of an annotation 40 realized by combining the object position history, the object area history, and the skeleton information history. The annotation representative image 60 in FIG. 13 includes the annotation 40-1 in FIG. 6, the annotation 40-3 in FIG. 8, and the annotation 40-4 in FIG. 9.

[0086] Here, the first generation unit 2060 may use a larger number of captured images 20 to generate the object position history than the number of captured images 20 to generate the object region history and the skeleton information history. For example, the first generation unit 2060 extracts captured images 20 from the image sequence 10 at a frequency of one captured image 20 per A images, and uses the extracted captured images 20 to generate an annotation 40 representing the object position history (A is a positive integer). Alternatively, the first generation unit 2060 may extract captured images 20 from the image sequence 10 at a frequency of one captured image 20 per B images, and uses the extracted captured images 20 to generate an annotation 40 representing the object region history and the skeleton information history (B is a positive integer). Here, B is a value greater than A.

[0087] According to this method, it is possible to include visual information of objects obtained from each of the multiple captured images 20 in the annotation representative image 60 while reducing the likelihood that object regions or skeletal information will overlap in the annotation representative image 60. Furthermore, according to this method, it is possible to include object position information in the annotation representative image 60 even from captured images 20 from which object regions or skeletal information have not been extracted. Therefore, it is possible to include a wealth of information about objects in one annotation representative image 60.

[0088] <<Type of annotation 40 superimposed on representative image 50>> The type of annotation 40 superimposed on the representative image 50 may be predetermined or selected. In the latter case, for example, the type of annotation 40 is specified by the user of the text generator 2000. Alternatively, for example, the type of annotation 40 may be dynamically determined, as described below.

[0089] <<Method for generating an annotation representative image 60 for each object>> When multiple objects are detected from the image sequence 10, the first generation unit 2060 may generate an annotation representative image 60 for each object. In this case, the first generation unit 2060 generates an annotation 40 for each object. Then, for each object, the first generation unit 2060 superimposes the annotation 40 generated for that object on the representative image 50 to generate an annotation representative image 60 for that object.

[0090] Here, a common representative image 50 may be used for all objects, or a different representative image 50 may be extracted for each object. In the latter case, for example, the first generation unit 2060 identifies, for each object, a time range in which the object appears in the image sequence 10. Then, for each object, the first generation unit 2060 extracts a representative image 50 from the image sequence 10 within the time range in which the object appears.

[0091] <Generation of Output Text 70: S110> The second generation unit 2080 uses the annotation representative image 60 to generate the output text 70, which is text related to the content of the annotation representative image 60 (S110). For example, the second generation unit 2080 generates a sentence that explains the content of the annotation representative image 60 as the output text 70. To generate the sentence that explains the content of the image, for example, a machine learning model that can handle images as input data can be used. Hereinafter, the machine learning model that generates the output text 70 will be referred to as a text generation model.

[0092] 14 is a diagram illustrating the operation of the text generation model 90. The text generation model 90 is configured to output output text 70 in response to input of an annotated representative image 60 and a prompt 100.

[0093] The prompt 100 is text that represents the content of a request made to the text generation model 90. For example, the prompt 100 is text that reads, "Please explain the content of the input image." When the prompt 100 and the annotation representative image 60 are input, the text generation model 90 outputs a sentence that explains the content of the annotation representative image 60 as the output text 70. The text generation model 90 can also be said to be "a model that returns an answer to the request (e.g., a question) indicated by the prompt 100."

[0094] Here, the text generation model 90 may be provided inside the text generator 2000 or outside the text generator 2000. In the latter case, the second generation unit 2080 transmits a request including the annotation representative image 60 and the prompt 100 to the text generation model 90 provided outside the text generator 2000. The second generation unit 2080 obtains the output text 70 as a response to this request.

[0095] Prompt 100 may also express a more specific request. For example, prompt 100 may include text requesting a determination of whether an object has performed a predetermined action. In this case, prompt 100 may include a sentence such as, "Please determine whether action X is being performed in the input images." Using such a prompt 100, it is possible to determine whether an object captured in image sequence 10 is performing a predetermined action.

[0096] Alternatively, for example, prompt 100 may include text requesting classification of an action performed by an object. In this case, prompt 100 may include a sentence such as, "Please tell me what action is being performed in the input images." By using such prompt 100, the action of the object captured in image sequence 10 can be classified.

[0097] Suppose multiple objects are captured in the image sequence 10, and information about a specific object is desired. In this case, annotated representative images 60 and prompts 100 are prepared so that output text 70 related to the specific object can be generated.

[0098] For example, the specific object is detected by the detection unit 2040. Furthermore, the first generation unit 2060 includes a mark representing the specific object (e.g., a frame representing a circumscribing rectangle of the object) as part of the annotation 40. The prompt 100 then represents a request for information about the object specified by the mark.

[0099] For example, suppose the specific object is a specific person. Furthermore, suppose the specific person is surrounded by a frame in the annotation representative image 60. In this case, for example, the prompt 100 may be a sentence such as, "Please identify the action performed by the person surrounded by the frame in the input image."

[0100] <<Prompt 100 Including Supplementary Explanation>> The prompt 100 may include an explanation (hereinafter, "supplementary explanation") regarding the annotated representative image 60. In this case, for example, the prompt 100 is composed of a portion representing a request to the text generation model 90 and a portion representing the supplementary explanation.

[0101] For example, the explanation may include an explanation regarding the definition of annotation 40. The following is an example of a prompt 100 that includes an explanation regarding the definition of annotation 40.

[0102] ・Request: Please explain the contents of the input image. ・Additional explanation: The red dots represent the object's position history.

[0103] Similarly, if the annotations 40 are the history of the object region, or if the annotations 40 represent skeletal information of the object, the prompt 100 will include additional explanations that define these annotations 40 .

[0104] As described above, when multiple objects are detected from the image sequence 10, the annotation 40 may have different features for each object. In this case, the annotation preferably includes an explanation of the correspondence between the object and the annotation 40. For example, the annotation may be something like, "The red dots represent the location history of the object with identifier 001. The blue dots represent the location history of the object with identifier 002."

[0105] In this case, it is preferable that the annotated representative image 60 includes an indication of an identifier for each object, which allows the text generation model 90 to easily identify the object specified in the prompt 100. The identifier is indicated, for example, on a frame surrounding the object.

[0106] The supplemental explanation is not limited to an explanation regarding the definition of the annotation 40. For example, suppose that a sequence of images 10 generated by each of a plurality of cameras is acquired, and an annotation representative image 60 is generated from each image sequence 10. In this case, for example, the supplemental explanation is an explanation regarding the camera corresponding to each annotation representative image 60.

[0107] The camera explanation is, for example, a sentence explaining the placement of the camera. For example, suppose there are cameras both inside and outside a building. Then, suppose a first annotated representative image 60 is generated from a sequence of images 10 generated by a camera inside the building. Also, suppose a second annotated representative image 60 is generated from a sequence of images 10 generated by a camera outside the building. In this case, for example, a supplementary explanation such as "The first image is an image taken inside the building. The second image is an image taken outside the building" is used.

[0108] The annotation is not limited to a sentence. For example, the text generation model 90 may be configured to accept an image representing the annotation in addition to the annotation representative image 60 and the prompt 100. The image representing the annotation may be, for example, a map image showing the placement of cameras.

[0109] The supplemental explanation may be included in the annotation representative image 60 instead of the prompt 100. For example, a map image may be included in the annotation representative image 60. In this case, the first generation unit 2060 generates the annotation representative image 60 by superimposing the annotation 40 and a map image showing the location of the camera that generated the representative image 50 on the representative image 50.

[0110] Fig. 15 is a diagram illustrating an example of an annotation representative image 60 that includes a map image. The annotation representative image 60 in Fig. 15 includes a map image 110. In Fig. 15, the map image 110 includes a mark 112 that indicates the position of the camera and a mark 114 that indicates the position of the building.

[0111] If the annotation representative image 60 includes a map image 110, the prompt 100 may include, as a supplemental explanation, an explanation of the map image 110. The explanation of the map image 110 includes the position of the map image 110 in the annotation representative image 60 and the meaning of each mark included in the map image 110.

[0112] <<Generation of Prompt 100>> The prompt 100 may be prepared in advance or may be input by the user. The text generator 2000 may be configured to allow the user to select a desired prompt 100 from a plurality of prepared prompts 100. In this case, the text generator 2000 provides, for example, a list of selectable prompts 100 to the terminal used by the user in a selectable manner. The user selects the prompt 100 they wish to use from the list displayed on the terminal. The text generator 2000 inputs the prompt 100 selected by the user into the text generation model 90. Note that the user may edit part of the selected prompt.

[0113] <<Selecting the Annotation 40 Using the Prompt 100>> The prompt 100 may be used to select the type of annotation 40 to be used by the first generator 2060. For example, the type of annotation 40 may be selected using a machine learning model (e.g., a neural network) trained to return an answer to a question. Hereinafter, this model is also referred to as an answer model.

[0114] For example, the first generator 2060 uses an answer model to execute a process of asking a question about which of a plurality of pieces of information is useful for executing a task. Here, the fulfillment of a request indicated in the prompt 100 can be regarded as a task. Furthermore, each type of annotation 40 can be regarded as information useful for executing the task.

[0115] Therefore, the first generation unit 2060 inputs a question to the answer model: "Of the multiple types of annotations 40, which type is useful for processing the request indicated in the prompt 100?" As a result, the first generation unit 2060 obtains an answer from the answer model indicating the type of annotation 40. The first generation unit 2060 generates, from the image sequence 10, annotations 40 of the type indicated in the answer obtained from the answer model, and superimposes them on the representative image 50.

[0116] The types of annotations 40 that can be selected may be provided in advance in the answer model, or may be input as part of the prompt.

[0117] <<Prior Knowledge Provided to Text Generation Model 90>> Prior knowledge may be provided to the text generation model 90. For example, the various supplementary explanations described above may be provided in advance as prior knowledge to the text generation model 90. The supplementary explanations provided as prior knowledge do not need to be included in the prompt 100.

[0118] Alternatively, for example, if a request to the text generation model 90 is fixed and predetermined, the content of the request may be provided to the text generation model 90 in advance as prior knowledge. This can also be said to mean that "the task to be executed by the text generation model 90 is specifically determined in advance." In this case, the only data input to the text generation model 90 may be the annotation representative image 60. In response to the input of the annotation representative image 60, the text generation model 90 executes the predetermined task regarding the annotation representative image 60 and outputs output text 70 representing the execution result.

[0119] <Output of Results> The text generator 2000 outputs the results of the processing in various ways. The text generator 2000 outputs at least the output text 70. In addition, for example, the text generator 2000 may further output information used in generating the output text 70, such as the representative image 50 and the annotated representative image 60.

[0120] The text generator 2000 may output information in a variety of ways. For example, the text generator 2000 may store the output text 70, etc. in an arbitrary storage unit. Alternatively, for example, the text generator 2000 may output the output text 70, etc. to a display device, etc., thereby displaying the output text 70 on the display device, etc. Alternatively, for example, the text generator 2000 may transmit the output text 70, etc. to another device.

[0121] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0122] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0123] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0124] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A text generation device comprising: an acquisition means for acquiring an image sequence consisting of a plurality of captured images; a detection means for detecting an object from each of the plurality of captured images; a first generation means for extracting a representative image from the image sequence and generating an annotation representative image by superimposing an annotation related to the object in the captured images other than the representative image on the representative image; and a second generation means for generating output text related to the object using the annotation representative image. (Supplementary Note 2) The text generation device according to Supplementary Note 1, wherein the second generation means generates the output text that describes the behavior of the object. (Supplementary Note 3) The text generation device according to Supplementary Note 1 or 2, wherein the first generation means superimposes the annotation, which indicates the position of the object in each of the plurality of captured images, on the representative image. (Supplementary Note 4) The text generation device according to Supplementary Note 1 or 2, wherein the first generation means superimposes an image region of the object detected from each of the plurality of captured images on the representative image as the annotation. (Supplementary Note 5) The text generator according to Supplementary Note 1 or 2, wherein the first generation means superimposes skeletal information of the object detected from each of the plurality of captured images on the representative image as the annotation. (Supplementary Note 6) The text generator according to Supplementary Note 1 or 2, wherein the second generation means generates the output text by inputting a prompt representing a request related to the object and the annotation representative image into a trained machine learning model and obtaining text from the machine learning model in accordance with the input, the machine learning model being trained to output text representing a response to the request in response to input of a prompt representing a request related to an image and the image. (Supplementary Note 7) The text generator according to Supplementary Note 6, wherein the prompt includes an explanation of the annotation. (Supplementary Note 8) The text generator according to Supplementary Note 6, wherein the second generation means inputs the annotation representative image, on which a map image representing the position of a camera that generated the image sequence is superimposed, into the machine learning model.(Supplementary Note 9) A text generation method executed by a computer, comprising: an acquisition step of acquiring an image sequence consisting of a plurality of captured images, a detection step of detecting an object from each of the plurality of captured images, a first generation step of generating an annotation representative image by extracting a representative image from the image sequence and superimposing annotations related to the object in the captured images other than the representative image on the representative image, and a second generation step of generating output text related to the object using the annotation representative image. (Supplementary Note 10) A program that causes a computer to execute: an acquisition step of acquiring an image sequence consisting of a plurality of captured images, a detection step of detecting an object from each of the plurality of captured images, a first generation step of generating an annotation representative image by extracting a representative image from the image sequence and superimposing annotations related to the object in the captured images other than the representative image on the representative image, and a second generation step of generating output text related to the object using the annotation representative image.

[0125] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 8 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 9 and 10 in the same dependency relationship as Supplementary Notes 2 to 8. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods.

[0126] This application claims priority based on Japanese Patent Application No. 2024-115482, filed on July 19, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0127] REFERENCE SIGNS LIST 10 Image sequence 20 Captured image 40 Annotation 50 Representative image 60 Annotated representative image 70 Output text 80 Object tracking information 82 Identifier 84 History 86 Time point 88 Object region information 90 Text generation model 100 Prompt 110 Map image 112 Mark 114 Mark 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Text generator 2020 Acquisition unit 2040 Detection unit 2060 First generation unit 2080 Second generation unit

Claims

1. A text generation device having: an acquisition means for acquiring an image sequence consisting of a plurality of captured images; a detection means for detecting an object from each of the plurality of captured images; a first generation means for extracting a representative image from the image sequence and generating an annotated representative image by superimposing annotations related to the object in the captured images other than the representative image onto the representative image; and a second generation means for generating output text related to the object using the annotated representative image.

2. The text generator according to claim 1, wherein said second generating means generates said output text representing a description of the behavior of said object.

3. A text generation device according to claim 1 or 2, wherein the first generation means superimposes the annotation indicating the position of the object in each of the plurality of captured images on the representative image.

4. A text generation device according to claim 1 or 2, wherein the first generation means superimposes an image area of ​​the object detected from each of the plurality of captured images onto the representative image as the annotation.

5. A text generation device according to claim 1 or 2, wherein the first generation means superimposes skeletal information of the object detected from each of the plurality of captured images onto the representative image as the annotation.

6. The text generation device of claim 1 or 2, wherein the second generation means generates the output text by inputting a prompt representing the content of a request regarding the object and the annotated representative image into a trained machine learning model and obtaining text from the machine learning model in response to the input, and the machine learning model is trained to output text representing a response to the request in response to input of a prompt representing a request regarding an image and the image.

7. The text generator of claim 6, wherein the prompt includes an explanation for the annotation.

8. A text generation device as described in claim 6, wherein the second generation means inputs the annotated representative image onto which a map image showing the position of the camera that generated the image sequence is superimposed into the machine learning model.

9. A text generation method executed by a computer, comprising: an acquisition step of acquiring an image sequence consisting of a plurality of captured images; a detection step of detecting an object from each of the plurality of captured images; a first generation step of generating an annotated representative image by extracting a representative image from the image sequence and superimposing annotations related to the object in the captured images other than the representative image onto the representative image; and a second generation step of generating output text related to the object using the annotated representative image.

10. A text generation method according to claim 9, wherein in said second generation step, said output text representing a description of the action of said object is generated.

11. A text generation method according to claim 9 or 10, wherein in the first generation step, the annotation indicating the position of the object in each of the plurality of captured images is superimposed on the representative image.

12. A text generation method according to claim 9 or 10, wherein in the first generation step, an image region of the object detected from each of the plurality of captured images is superimposed on the representative image as the annotation.

13. A text generation method according to claim 9 or 10, wherein in the first generation step, skeletal information of the object detected from each of the plurality of captured images is superimposed on the representative image as the annotation.

14. A text generation method as described in claim 9 or 10, wherein in the second generation step, a prompt representing the content of a request regarding the object and the annotated representative image are input into a trained machine learning model, and text is obtained from the machine learning model in response to the input, thereby generating the output text, and the machine learning model is trained to output text representing a response to the request in response to input of a prompt representing a request regarding an image and the image.

15. The text generation method of claim 14, wherein the prompt includes an explanation for the annotation.

16. A text generation method as described in claim 14, wherein in the second generation step, the annotated representative image on which a map image showing the position of the camera that generated the image sequence is superimposed is input to the machine learning model.

17. A program that causes a computer to execute the following steps: an acquisition step of acquiring an image sequence consisting of a plurality of captured images; a detection step of detecting an object from each of the plurality of captured images; a first generation step of extracting a representative image from the image sequence and generating an annotated representative image by superimposing annotations related to the object in the captured images other than the representative image onto the representative image; and a second generation step of generating output text related to the object using the annotated representative image.

18. The program according to claim 17, wherein in the second generating step, the output text is generated to represent an explanation of the behavior of the object.

19. The program according to claim 17 or 18, wherein in the first generating step, the annotation indicating the position of the object in each of the plurality of captured images is superimposed on the representative image.

20. A program according to claim 17 or 18, wherein in the first generation step, an image area of ​​the object detected from each of the plurality of captured images is superimposed on the representative image as the annotation.

Images