Summary generation device, summary generation method, and summary generation program

Abstract

This invention provides a summary generation device that can generate a summary text that shows more detailed actions of the subject within a video. [Solution] The summary text generation device 30 of the present invention includes: an acquisition unit that acquires video data obtained by imaging a subject 50 with a camera; a caption generation unit that generates captions explaining the actions of the subject 50 in the video on a frame-by-frame basis for the video data acquired by the acquisition unit; and a summary text generation unit that generates a summary text of the video based on the captions generated by the caption generation unit.

Description

Technical Field

[0001] The present invention relates to an abstract generation device, an abstract generation method, and an abstract generation program.

Background Art

[0002] With the advent of an aging society, the number of care facilities has been increasing. In care facilities, cameras for monitoring residents (target persons) may be installed in the rooms of the target persons. Videos of the target persons captured by the cameras are viewed by the staff of the care facilities or the like.

[0003] As a technology related to viewing videos captured by cameras, Patent Document 1 below discloses a technology for generating summary information of videos. The technology of Patent Document 1 generates arbitrary information obtained from the content of a video as summary information of the video captured by a surveillance camera and displays it on a display. According to this technology, a viewer of the video can easily grasp the content of the video by referring to the summary information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the above technology only displays the movement trajectory and residence time of the persons shown in the video as summary information, and is not sufficient as information for grasping the content of the video. In order to sufficiently grasp the content of the video, a summary text showing more detailed actions of the target person in the video is required.

[0006] The present invention has been made in view of the above-mentioned problems. Accordingly, the object of the present invention is to provide a summary text generation device, a summary text generation method, and a summary text generation program that can generate summary texts that show more detailed actions of a subject in a video. [Means for solving the problem]

[0007] The above objectives of the present invention are achieved by the following means.

[0008] (1) A summary generation device comprising: an acquisition unit that acquires video data obtained by imaging a subject with a camera; a caption generation unit that generates captions on a frame image basis for the video data acquired by the acquisition unit that describe the subject's actions in the video; and a summary generation unit that generates a summary of the video based on the captions generated by the caption generation unit.

[0009] (2) The summary generation device according to (1) above, wherein the summary generation unit inputs a prompt to the text generation model to instruct the generation of a summary, and generates a summary of the video.

[0010] (3) The summary text generation device according to (1) or (2) above, wherein the caption generation unit generates the caption using a learning model that has learned an image and the corresponding caption as training data.

[0011] (4) The summary text generation device according to (3) above, wherein the caption generation unit generates a caption for the specific frame image based on a caption corresponding to an image in the training data that is similar to the specific frame image of the video.

[0012] (5) The summary text generation device according to (3) above, wherein the caption generation unit combines a plurality of captions in the training data to generate a caption for a specific frame image of the video.

[0013] (6) A method for generating a summary text, comprising the steps of: (a) acquiring video data obtained by imaging a subject with a camera; (b) generating captions for each frame image of the video data acquired in step (a) that describe the actions of the subject in the video; and (c) generating a summary text of the video based on the captions generated in step (b).

[0014] (7) A summary generation program that causes a computer to perform the following steps: (a) acquire video data obtained by imaging a subject with a camera; (b) generate captions on a frame-by-frame basis for the video data acquired in step (a), describing the subject's actions in the video; and (c) generate a summary of the video based on the captions generated in step (b). [Effects of the Invention]

[0015] According to the present invention, it is possible to generate a summary text that shows more detailed actions of the subject within a video. [Brief explanation of the drawing]

[0016] The advantages and features provided by one or more embodiments of the present invention will be better understood from the following detailed description and accompanying drawings, which are for illustrative purposes only and are not intended to define any limitations of the present invention. [Figure 1] This diagram shows the overall configuration of the monitoring system. [Figure 2] This is a block diagram showing the schematic configuration of the detection device. [Figure 3] This is a block diagram showing the schematic configuration of the terminal device. [Figure 4] This is a diagram showing the schematic configuration of a server device. [Figure 5] This diagram shows the contents of the storage unit of the server device. [Figure 6] This is a flowchart showing the steps involved in the caption generation process. [Figure 7] This is a diagram showing an example of a frame image after caption generation processing. [Figure 8] This is a flowchart showing the procedure of summary sentence generation processing. [Figure 9] This is a diagram showing an example of a prompt for instructing the generation of a summary sentence. [Figure 10] This is a diagram for explaining summary sentence generation processing.

Mode for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the scope of the present invention is not limited to the disclosed embodiments.

[0018] FIG. 1 is a diagram showing the overall configuration of a monitoring system 1 to which a summary sentence generation device according to an embodiment of the present invention is applied.

[0019] As shown in FIG. 1, the monitoring system 1 includes a detection device 10, a terminal device 20, and a server device 30. The detection device 10, the terminal device 20, and the server device 30 are configured to be communicable via a network 40.

[0020] The detection device 10 is installed in the living room of the target person 50 in various facilities such as nursing facilities and hospitals. The terminal device 20 is used, for example, by staff such as nurses who care for the target person 50 or facility administrators. The server device 30 is an on-premises server installed within the facility premises or a cloud server using a commercial cloud service. The network 40 is configured by the Internet or an intranet.

[0021] <Detection device 10> FIG. 2 is a block diagram showing the schematic configuration of the detection device 10. The detection device 10 is installed as a sensor box on the ceiling or upper part of the wall of the living room of the target person 50.

[0022] As shown in Figure 2, the detection device 10 comprises a control unit 11, a communication unit 12, and a camera 13, which are interconnected by a bus.

[0023] The control unit 11 is composed of a CPU (Central Processing Unit) and memory such as RAM (Random Access Memory) and ROM (Read Only Memory), and controls each of the above parts and performs various calculation processes according to the program.

[0024] The communication unit 12 is an interface for communicating with other devices, and various wired or wireless communication interfaces are used.

[0025] Camera 13 captures images of the subject 50 from the ceiling or upper part of the wall of the living room and generates video data (image data) of the subject 50. Camera 13 is, for example, a near-infrared camera and captures a predetermined imaging area within the living room. Camera 13 captures the imaging area at a frame rate of, for example, 5 fps and generates video data.

[0026] The detection device 10 of this embodiment is configured to recognize predetermined actions of the subject 50 (such as getting up, getting out of bed, or falling) from video data obtained by imaging the subject 50 with the camera 13. When the detection device 10 recognizes a predetermined action of the subject 50, it outputs video data for one minute before and one minute after the occurrence of the action (a total of two minutes) to the server device 30 for recording. Note that the technology for recognizing predetermined actions (behaviors) of a person captured by the camera, and the technology for recording video data before and after an event occur, are known technologies, so a detailed explanation is omitted.

[0027] <Terminal device 20> Figure 3 is a block diagram showing the schematic configuration of the terminal device 20. The terminal device 20 is, for example, a PC (Personal Computer).

[0028] As shown in Figure 3, the terminal device 20 comprises a control unit 21, a storage unit 22, a communication unit 23, a display unit 24, and an input unit 25, which are interconnected by a bus. Note that, to avoid repetition in the explanation, the parts of the terminal device 20 that have the same functions as those of the detection device 10 will not be described.

[0029] The storage unit 22 consists of an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs and data.

[0030] The display unit 24 is, for example, a liquid crystal display, which displays various information.

[0031] The input unit 25 is equipped with a keyboard, numeric keypad, mouse, etc., and accepts input of various instructions and information.

[0032] The terminal device 20 of this embodiment is used, for example, by a user of the terminal device 20 (facility staff or administrator) to view video data obtained by capturing images of the subject 50 with the camera 13 of the detection device 10.

[0033] <Server device 30> Figure 4 is a block diagram showing the schematic configuration of the server device 30. The server device 30 corresponds to the summary text generation device of the present invention.

[0034] As shown in Figure 4, the server device 30 comprises a control unit 31, a storage unit 32, and a communication unit 33, which are interconnected by a bus. Note that the above-mentioned parts of the server device 30 have the same functions as the above-mentioned parts of the detection device 10 and the terminal device 20, so their descriptions are omitted.

[0035] Figure 5 is a diagram showing the contents of the storage unit 32 of the server device 30. As shown in Figure 5, the storage unit 32 of the server device 30 stores video data 100 of the subject 50. The video data 100 includes 2 minutes of video data obtained by imaging the subject 50 with the camera 13 of the detection device 10.

[0036] Furthermore, the storage unit 32 of the server device 30 stores programs corresponding to the acquisition unit 110, the caption generation unit 120, and the summary text generation unit 130. The acquisition unit 110 acquires video data obtained by imaging the subject 50 with the camera 13. The caption generation unit 120 generates captions (explanatory texts) for each frame image of the video data acquired by the acquisition unit 110, describing the actions of the subject 50 in the video. The caption generation unit 120 includes a learning model that has been trained using images and the corresponding captions as training data, and generates captions using this learning model. The summary text generation unit 130 generates a summary text of the video based on the captions generated by the caption generation unit 120. The summary text generation unit 130 includes a text generation model capable of generating a summary text from the caption information, and generates a summary text using this text generation model. The text generation model is a so-called generative AI (Artificial Intelligence), and is generated, for example, by fine-tuning a general-purpose text generation model. An example of a general-purpose text generation model is Copilot, provided by Microsoft. The functions of each of the above parts are performed by the control unit 31 executing the corresponding program.

[0037] Furthermore, the detection device 10, terminal device 20, and server device 30 may include components other than those described above, and may not include some of the components described above. For example, the detection device 10 may include other sensors such as a motion sensor or a microphone.

[0038] In the monitoring system 1 configured as described above, a summary text describing the content of a video is generated for the video data obtained by imaging the subject 50 with the camera 13 of the detection device 10. Specifically, first, for the video data obtained by imaging the subject 50 with the camera 13, multiple captions describing the actions of the subject 50 in the video are generated for each frame image. Then, a summary text of the video is generated based on the multiple captions generated for the video. The operation of the monitoring system 1 will be explained below with reference to Figures 6 to 10.

[0039] First, with reference to Figure 6, the operation of the server device 30 that generates captions for the video will be explained.

[0040] Figure 6 is a flowchart showing the procedure for the caption generation process performed by the server device 30. The process shown in the flowchart in Figure 6 is executed by the control unit 31 according to the program stored in the storage unit 32 of the server device 30.

[0041] (Step S101) First, the server device 30 acquires video data of the subject 50. More specifically, the server device 30 acquires video data of the subject 50 stored in the storage unit 32. The video data is 2 minutes (600 frames) of video data obtained by imaging the subject 50 with the camera 13 of the detection device 10.

[0042] (Step S102) Next, the server device 30 generates captions. More specifically, the server device 30 generates captions for each frame image of the video data acquired in step S101, describing the actions of the subject 50 in the video. In this embodiment, the server device 30 inputs the video data into a learning model that has learned the relationship between images and the captions corresponding to those images. The learning model, upon receiving the video data, recognizes the actions (features) of the subject 50 in each frame image and generates a caption describing those actions for each frame image.

[0043] (Step S103) The server device 30 then stores the caption and terminates the process. More specifically, the server device 30 stores the text information of the caption generated in step S102 and the frame number information of the frame image to which the caption is applied in the storage unit 32, associating them with the video data, and then terminates the process.

[0044] As described above, according to the flowchart shown in Figure 6, captions explaining the actions of the subject 50 in the video are generated on a frame-by-frame basis for the video data obtained by capturing the subject 50 with the camera 13 of the detection device 10. Specifically, for example, various captions explaining each action of the subject 50 in various scenes of the video are generated.

[0045] Figure 7 shows an example of a frame image 200 after the caption generation process. As described above, in the caption generation process, a caption describing the actions of subject 50 is generated for each frame image of the video data of subject 50.

[0046] As shown in Figure 7, the frame image 200 after the caption generation process displays a caption 210 overlaid on it that describes the actions of the subject 50. Specifically, for example, the caption 210 "The resident is sitting on the edge of the bed" is displayed overlaid on the bottom of the frame image 200 as a description of the actions of the subject 50 in the frame image 200 shown in Figure 7. Note that the actions of the subject 50 include not only the active actions of the subject 50 but also the passive actions of the subject 50 that are assisted by the facility staff.

[0047] As described above, the server device 30 generates captions for each frame of a video using a learning model that has been trained with images and their corresponding captions as training data. For a specific frame of a video, the learning model generates a caption for that frame based on the captions corresponding to similar images in the training data. Similarity between images means that the actions (features) of the people in the images are similar.

[0048] Training data is prepared, for example, by having an operator manually add captions (ground truth data) to frame images while viewing a video. The technique of generating a learning model with a predetermined function by feeding training data to a learning model such as a neural network and letting it learn is a well-known machine learning technique, so a detailed explanation will be omitted.

[0049] Furthermore, the learning model in this embodiment is configured to generate new captions by combining multiple captions within the training data. Specifically, for example, suppose the learning model has already learned two captions, "The resident is walking" and "The resident is brushing their teeth," and the action of the subject 50 in a specific frame image of the video is walking while brushing their teeth. In this case, if the learning model recognizes the two actions (features) "walking" and "brushing teeth" from the frame image, it will use a predetermined language model to combine the two captions and generate a new caption, "The resident is walking while brushing their teeth." With this configuration, when training the learning model, it is not necessary to provide the learning model with captions that explain multiple actions performed simultaneously by a person in the video as training data, thus reducing the effort required of the person preparing the training data.

[0050] Next, referring to Figures 8 to 10, the operation of the server device 30 that generates video summaries will be explained.

[0051] Figure 8 is a flowchart showing the procedure for the summary generation process performed by the server device 30. The process shown in the flowchart in Figure 8 is executed by the control unit 31 according to the program stored in the storage unit 32 of the server device 30.

[0052] (Step S201) First, the server device 30 inputs a prompt to the document generation model. More specifically, the server device 30 inputs a prompt 300 (see Figure 9) instructing the generation of a summary to the document generation model (generation AI) which is capable of generating a summary from caption information.

[0053] Figure 9 shows an example of a prompt 300 that instructs the generation of a summary. As shown in Figure 9, the prompt 300 instructs the text generation model to generate a summary that describes a series of actions of the subject 50 from multiple captions. In Figure 9, "frequency information" refers to the number of frame images to which a particular caption is applied.

[0054] (Step S202) Next, the server device 30 generates a summary. More specifically, the server device 30 generates a summary of the video based on the captions generated in the caption generation process shown in Figure 6. In this embodiment, the text generation model that receives a prompt in step S201 generates a summary of the video based on the information of multiple captions.

[0055] (Step S203) Then, the server device 30 stores the summary text and terminates the process. More specifically, the server device 30 stores the text information of the summary text generated in step S202 in the storage unit 32, associating it with the video data, and then terminates the process.

[0056] As described above, according to the flowchart shown in Figure 8, a summary of the video is generated based on multiple captions generated for the video. In this embodiment, for example, a summary is generated that describes a series of actions of the subject 50 in the video.

[0057] Figure 10 is a diagram illustrating the summary generation process. As described above, the summary generation process generates a summary of a video based on multiple captions generated for the video. Specifically, as shown in Figure 10, for example, a summary (right side of Figure 10) is generated by combining six captions (left side of Figure 10) that describe six actions of chronologically consecutive subjects 50.

[0058] With this configuration, a summary text is generated that shows more detailed actions of the subject 50 within the video, allowing the video viewer (user of the terminal device 20) to easily grasp the detailed content of the video. For example, when searching for a desired video from among multiple videos, the viewer can find the video from the summary text without having to watch the video.

[0059] In the embodiment described above, a summary sentence describing a series of actions of the subject 50 was generated from multiple captions generated for the video. However, the video summary sentence generated based on the captions is not limited to a summary sentence describing a series of actions of the subject 50. The video summary sentence generated based on the captions may, for example, be a summary sentence that shows an overview or trend of the subject 50's actions in the video. For example, a summary sentence such as, "The resident is often seen sitting on the edge of the bed, sitting in a wheelchair, and moving in front of the toilet or furniture. In particular, sitting in a wheelchair next to the bed and moving in front of the toilet is the most frequent action," may be generated.

[0060] In the monitoring system 1 of this embodiment, the content of the summary is changed by changing the content of the prompt input to the text generation model. The content of the prompt can be changed, for example, by the user of the terminal device 20 via the terminal device 20. With this configuration, the user of the terminal device 20 (video viewer) can easily generate a desired summary by changing the content of the prompt.

[0061] The present invention is not limited to the embodiments described above, and can be modified in various ways within the scope of the claims.

[0062] For example, in the embodiment described above, a model generated by fine-tuning a general-purpose text generation model was used as the text generation model that generates a summary from caption information. However, a model generated from scratch by training it with a large amount of data may also be used as the text generation model that generates a summary from caption information.

[0063] Furthermore, the above-described embodiment used the example of generating a summary text for a two-minute video. However, the videos for which a summary text can be generated are not limited to two-minute videos; summary texts can be generated for videos of various lengths.

[0064] The processing units in the flowcharts of the embodiments described above are divided according to the main processing content in order to facilitate understanding of each process. The present invention is not limited by how the processing steps are classified. Each process can be further divided into more processing steps. Also, one processing step may perform even more processes.

[0065] In the embodiments described above, the functions of each device may be implemented by other devices. For example, the function implemented by the detection device 10 may be implemented by the server device 30.

[0066] The means and methods for performing various processing in the summary generation device according to the above embodiment can be implemented by either a dedicated hardware circuit or a programmed computer. The program may be provided, for example, on a computer-readable recording medium such as a USB (Universal Serial Bus) memory or a DVD (Digital Versatile Disc)-ROM, or it may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a storage unit such as an HDD. Furthermore, the program may be provided as a standalone application software, or it may be incorporated into the software of the summary generation device as a function of the device.

[0067] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are for illustrative purposes only and are not limiting. The scope of the present invention should be interpreted in accordance with the language of the appended claims. [Explanation of symbols]

[0068] 1. Monitoring system, 10 detection devices, 11,21,31 Control Unit, 12, 23, 33 Communications Department, 13 cameras, 20 terminal devices, 22,32 memory section, 24 Display section, 25 Input section, 30 server devices, 40 networks, 50 target individuals.

Claims

1. An acquisition unit that acquires video data obtained by imaging a subject with a camera, A caption generation unit generates captions for each frame image that describe the actions of the subject in the video data acquired by the acquisition unit, A summary text generation unit generates a summary text of the video based on the captions generated by the caption generation unit, A summary generation device having the following features.

2. The summary generation device according to claim 1, wherein the summary generation unit inputs a prompt to a text generation model to instruct the generation of a summary, and generates a summary of the video.

3. The summary text generation device according to claim 1 or 2, wherein the caption generation unit generates the caption using a learning model that has learned an image and the corresponding caption as training data.

4. The summary text generation device according to claim 3, wherein the caption generation unit generates a caption for a specific frame image based on a caption corresponding to an image in the training data that is similar to a specific frame image of the video.

5. The summary text generation device according to claim 3, wherein the caption generation unit combines a plurality of captions in the training data to generate a caption for a specific frame image of the video.

6. (a) A step of acquiring video data obtained by imaging the subject with a camera, Step (b) involves generating captions for each frame image of the video data acquired in step (a) above, which describe the actions of the subject in the video. Step (c) generates a summary of the video based on the caption generated in step (b), A method for generating a summary text that has the following characteristics.

7. Procedure (a) for acquiring video data obtained by imaging a subject with a camera, The procedure (b) involves generating captions for each frame image of the video data obtained in the above procedure (a), which describe the actions of the subject in the video. The procedure (c) for generating a summary of the video based on the caption generated in the above procedure (b), A summary generation program that instructs a computer to perform the following actions.

Images