Tagging device, tagging method, and tagging program

Abstract

This invention provides a tagging device that can add search tags to videos that lack audio or subtitles. [Solution] The tagging device 30 of the present invention includes a storage unit that stores search values ​​for searching for videos, an acquisition unit that acquires video data obtained by imaging a subject 50 with a camera, an explanation unit that generates an explanation for the video data acquired by the acquisition unit, and a tagging unit that extracts a string corresponding to the search value from the explanation generated by the explanation unit and tags the video data.

Description

Technical Field

[0001] The present invention relates to a tagging device, a tagging method, and a tagging program.

Background Art

[0002] With the advent of an aging society, the number of nursing facilities is increasing. In a nursing facility, a camera for monitoring residents (target persons) may be installed in the resident's room. A video obtained by imaging a target person with the camera is recorded and viewed by staff of the nursing facility or the like.

[0003] In relation to this, Patent Document 1 below discloses a technique for attaching tags to a video in order to improve the searchability of the recorded video. The technique of Patent Document 1 extracts tags from the audio or telop information of a video according to a predetermined rule and attaches them to the video. According to the technique of Patent Document 1, it becomes possible to automatically tag a video.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, since the above technique extracts tags from the audio or telop information of a video, there is a problem that tags cannot be attached to a video without audio or telop.

[0006] The present invention has been made in view of the above problems. Therefore, an object of the present invention is to provide a tagging device, a tagging method, and a tagging program that can attach search tags to a video without audio or telop.

Means for Solving the Problems

[0007] The above objectives of the present invention are achieved by the following means.

[0008] (1) A tagging device comprising: a storage unit that stores search values ​​for searching for videos; an acquisition unit that acquires video data obtained by imaging a subject with a camera; an explanation unit that generates an explanation for the video data acquired by the acquisition unit; and a tagging unit that extracts a string corresponding to the search value from the explanation generated by the explanation unit and tags the video data.

[0009] (2) The tagging device according to (1) above, further comprising a caption generation unit that generates captions describing the actions of the subject in the video on a frame image basis, and the descriptive text generation unit that generates the descriptive text based on the captions generated by the caption generation unit.

[0010] (3) The tagging device described in (2) above, wherein the explanatory text generation unit generates the explanatory text by inputting a prompt to the text generation model that instructs the generation of the explanatory text.

[0011] (4) The tagging device according to (3) above, wherein the tagging unit inputs the prompt to the text generation model to further instruct the tagging of the video data, extracts the string from the description and tags the video data.

[0012] (5) The tagging device according to any one of (2) to (4) above, wherein the tagging unit further extracts the string from the caption and tags the video data.

[0013] (6) The tagging device according to (3) or (4) above, wherein the description generation unit generates the description containing the string by inputting the prompt instructing the text generation model to generate a description containing the string.

[0014] (7) The tagging device according to any one of (2) to (4) above, wherein the caption generation unit generates the caption using a learning model that has learned an image and the corresponding caption as training data.

[0015] (8) A tagging device according to any one of (1) to (4) above, further comprising a reception unit for receiving edits of the search values.

[0016] (9) The tagging device according to any one of (1) to (4) above, wherein the search value is the search value of at least one search item selected from among the presence or absence of assistance to the subject, the place where the subject is staying, the actions of the subject, and the object that the subject is acting on.

[0017] (10) A tagging method comprising the steps of: (a) acquiring video data obtained by imaging a subject with a camera; (b) generating a description of the video for the video data acquired in step (a); and (c) extracting a string corresponding to a search value for searching the video from the description generated in step (b) and tagging the video data.

[0018] (11) A tagging program that causes a computer to perform the following steps: (a) acquire video data obtained by imaging a subject with a camera; (b) generate a description of the video acquired in step (a); and (c) extract a string corresponding to a search value for searching the video from the description generated in step (b) and tag the video data. [Effects of the Invention]

[0019] According to the present invention, search tags can be added to videos that do not have audio or subtitles. [Brief explanation of the drawing]

[0020] The advantages and features provided by one or more embodiments of the present invention will be more fully understood from the following detailed description and the accompanying drawings, which are for illustrative purposes only and are not intended to define the limitations of the present invention. [Figure 1] It is a diagram showing the overall configuration of the monitoring system. [Figure 2] It is a block diagram showing the schematic configuration of the detection device. [Figure 3] It is a block diagram showing the schematic configuration of the terminal device. [Figure 4] It is a block diagram showing the schematic configuration of the server device. [Figure 5] It is a diagram showing the stored content of the storage unit of the server device. [Figure 6] It is a diagram showing an example of a video search screen. [Figure 7] It is a diagram showing an example of a video search result screen. [Figure 8] It is a flowchart showing the procedure of the caption generation process. [Figure 9] It is a diagram showing an example of a frame image after the caption generation process. [Figure 10] It is a flowchart showing the procedure of the tagging process. [Figure 11] It is a diagram showing an example of a prompt. [Figure 12A] It is a diagram for explaining the process of extracting a string from a caption. [Figure 12B] It is a diagram for explaining the process of extracting a string from a description text. [Figure 12C] It is a diagram showing an example of a string to be tagged to video data. [Figure 13] It is a diagram showing an example of a search condition editing screen. [Figure 14] It is a diagram for explaining the video search screen after the search condition editing process. [Figure 15A] It is a diagram for explaining the process of extracting a string from a caption. [Figure 15B]This diagram illustrates the process of extracting strings from a description. [Modes for carrying out the invention]

[0021] Embodiments of the present invention will be described below with reference to the drawings. However, the scope of the present invention is not limited to the disclosed embodiments.

[0022] Figure 1 shows the overall configuration of a monitoring system 1 to which a tagging device according to one embodiment of the present invention is applied.

[0023] As shown in Figure 1, the monitoring system 1 comprises a detection device 10, a terminal device 20, and a server device 30. The detection device 10, the terminal device 20, and the server device 30 are configured to communicate with each other via a network 40.

[0024] The detection device 10 is installed in the rooms of 50 target individuals within various facilities such as nursing homes and hospitals. The terminal device 20 is used, for example, by staff such as caregivers who care for the target individuals 50, or by facility administrators. The server device 30 is either an on-premise server installed on the facility's premises or a cloud server using a commercial cloud service. The network 40 consists of the internet or an intranet.

[0025] <Detection device 10> Figure 2 is a block diagram showing the schematic configuration of the detection device 10. The detection device 10 is installed as a sensor box on the ceiling or upper part of the wall of the room where the subject 50 lives.

[0026] As shown in Figure 2, the detection device 10 comprises a control unit 11, a communication unit 12, and a camera 13, which are interconnected by a bus.

[0027] The control unit 11 is composed of a CPU (Central Processing Unit) and memory such as RAM (Random Access Memory) and ROM (Read Only Memory), and controls each of the above parts and performs various calculation processes according to the program.

[0028] The communication unit 12 is an interface for communicating with other devices, and various wired or wireless communication interfaces are used.

[0029] Camera 13 captures images of the subject 50 from the ceiling or upper part of the wall of the living room and generates video data (image data) of the subject 50. Camera 13 is, for example, a near-infrared camera and captures a predetermined imaging area within the living room. Camera 13 captures the imaging area at a frame rate of, for example, 5 fps and generates video data.

[0030] The detection device 10 of this embodiment is configured to recognize predetermined actions of the subject 50 (such as getting up, getting out of bed, or falling) from video data obtained by imaging the subject 50 with the camera 13. When the detection device 10 recognizes a predetermined action of the subject 50, it outputs video data for one minute before and one minute after the occurrence of the action (a total of two minutes) to the server device 30 for recording. Note that the technology for recognizing predetermined actions (behaviors) of a person captured by the camera, and the technology for recording video data before and after an event occur, are known technologies, so a detailed explanation is omitted.

[0031] <Terminal device 20> Figure 3 is a block diagram showing the schematic configuration of the terminal device 20. The terminal device 20 is, for example, a PC (Personal Computer).

[0032] As shown in Figure 3, the terminal device 20 comprises a control unit 21, a storage unit 22, a communication unit 23, a display unit 24, and an input unit 25, which are interconnected by a bus. Note that, to avoid repetition in the explanation, the parts of the terminal device 20 that have the same functions as those of the detection device 10 will not be described.

[0033] The storage unit 22 consists of an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs and data.

[0034] The display unit 24 is, for example, a liquid crystal display, which displays various information.

[0035] The input unit 25 is equipped with a keyboard, numeric keypad, mouse, etc., and accepts input of various instructions and information.

[0036] The terminal device 20 of this embodiment is used, for example, by a user of the terminal device 20 (facility staff or administrator) to search and view video data obtained by capturing images of the subject 50 with the camera 13 of the detection device 10.

[0037] <Server device 30> Figure 4 is a block diagram showing the schematic configuration of the server device 30. The server device 30 corresponds to the tagging device of the present invention.

[0038] As shown in Figure 4, the server device 30 comprises a control unit 31, a storage unit 32, and a communication unit 33, which are interconnected by a bus. Note that the above-mentioned parts of the server device 30 have the same functions as the above-mentioned parts of the detection device 10 and the terminal device 20, so their descriptions are omitted.

[0039] Figure 5 is a diagram showing the contents of the storage unit 32 of the server device 30. As shown in Figure 5, the storage unit 32 of the server device 30 stores video data 110 and search condition data 120. The video data 110 includes multiple 2-minute video data obtained by imaging the subject 50 with the camera 13 of the detection device 10. Each video data is stored in association with information of a descriptive text that explains the content of the video and information of a tag for identifying the video. The tag information includes information of a string tagged to the video data. The search condition data 120 includes information of search values ​​(search keywords) for searching video data and information of a string corresponding to each search value.

[0040] Furthermore, the storage unit 32 of the server device 30 stores programs corresponding to the acquisition unit 131, the caption generation unit 132, the description generation unit 133, and the tagging unit 134. The acquisition unit 131 acquires video data obtained by imaging the subject 50 with the camera 13. The caption generation unit 132 generates captions for each frame image of the video data acquired by the acquisition unit 131, describing the actions of the subject 50 in the video. The caption generation unit 132 includes a learning model that has learned images and the corresponding captions as training data, and generates captions using this learning model. The description generation unit 133 generates a description (summary) of the video based on the captions generated by the caption generation unit 132. The description generation unit 133 includes a text generation model that can generate a description from the caption information, and generates the description using this text generation model. The tagging unit 134 extracts strings corresponding to search values ​​for searching videos from the description generated by the description generation unit 133 and tags the video data. The tagging unit 134 also extracts strings using the above-mentioned text generation model and tags the video data with those strings. The text generation model is a so-called generative AI (Artificial Intelligence), and is generated, for example, by fine-tuning a general-purpose text generation model. An example of a general-purpose text generation model is Copilot provided by Microsoft. The functions of each of the above-mentioned units are performed by the control unit 31 executing the corresponding programs.

[0041] Furthermore, the detection device 10, terminal device 20, and server device 30 may include components other than those described above, and may not include some of the components described above. For example, the detection device 10 may include other sensors such as a motion sensor or a microphone.

[0042] Next, with reference to Figures 6 and 7, the operation of the monitoring system 1 in searching for video data stored in the storage unit 32 of the server device 30 will be described. In the monitoring system 1, video data stored in the storage unit 32 of the server device 30 is searched using search values ​​(search keywords) that are stored in advance as video search conditions. The video data is tagged with strings corresponding to the search values.

[0043] Figure 6 shows an example of the video search screen 200. The video search screen 200 is displayed on the display unit 24 of the terminal device 20 based on display information transmitted from the server device 30. The display information is generated by the server device 30 based on the search value information stored as search condition data 120 in the storage unit 32 of the server device 30.

[0044] As shown in Figure 6, the video search screen 200 includes a search condition setting unit 210 and a search button 220. The search condition setting unit 210 includes multiple checkboxes for selecting search values ​​(search keywords) for searching for videos. Search values ​​are provided for each of the five search items: "time of day," "assistance provided," "action," "location," and "object."

[0045] The search item "Time of day" is a search item related to the time of day when the video was captured, and includes the search values ​​"Daytime" and "Nighttime". The search item "Assistance" is a search item related to whether or not assistance was provided to the subject 50, and includes the search values ​​"Yes" and "No". The search item "Actions" is a search item related to the actions of the subject 50, and includes the search values ​​"Getting up", "Sitting up", "Walking", "Getting dressed", "Grooming", and "Reading". The search item "Location" is a search item related to the place where the subject 50 was staying, and includes the search values ​​"Bed", "Chair", "Toilet", and "Wheelchair". The search item "Objects" is a search item related to the objects that the subject 50 was acting on, and includes the search values ​​"Wheelchair", "Walker", "TV", "Washbasin", "Clothes", and "Shoes".

[0046] The user of terminal device 20 selects a search value (search keyword) from among the multiple search values ​​(search keywords) displayed on the video search screen 200 to search for the desired video, and then presses (clicks) the search button 220. When the search button 220 is pressed, the server device 30 searches for video data by referring to the tag information associated with the video data. More specifically, the server device 30 searches among multiple video data for video data tagged with the string corresponding to the search value selected by the user of terminal device 20.

[0047] For example, if a user of terminal device 20 selects the search value "walking" for the search item "action," server device 30 searches for video data tagged with the strings "walking" or "walking" corresponding to the search value "walking." Also, for example, if a user of terminal device 20 selects the search value "yes" for the search item "caregiving," server device 30 searches for video data tagged with the strings "staff" or "assistance" corresponding to the search value "yes." Also, for example, if a user of terminal device 20 selects the search value "bed" for the search item "place," server device 30 searches for video data tagged with the string "bed." Also, for example, if a user of terminal device 20 selects the search value "walker" for the search item "object," server device 30 searches for video data tagged with the string "walker." Also, for example, if a user of terminal device 20 selects the search value "nighttime" for the search item "time of day," server device 30 searches for video data tagged with the string "nighttime."

[0048] In this embodiment, the video search screen 200 is configured to search for video data tagged with a string corresponding to any of the search values ​​selected by the user of the terminal device 20 (OR search). However, unlike this embodiment, the video search screen 200 may be configured to search for video data tagged with a string corresponding to all of the selected search values ​​(AND search).

[0049] Figure 7 shows an example of the video search results screen 300. The video search results screen 300 is displayed on the display unit 24 of the terminal device 20 based on display information transmitted from the server device 30.

[0050] The video search results screen 300 displays a list of video data tagged with a string corresponding to the search value (search keyword) selected by the user through the video search screen 200. As shown in Figure 7, the video search results screen 300 of this embodiment displays a list of pairs: a video playback screen 310 that displays the video data tagged with the string corresponding to the search value, and a description text 320 that explains the content of the video. The description text 320 is generated based on the description text information stored in the storage unit 32 of the server device 30 in association with the video data.

[0051] For example, suppose the search value "nighttime" or "walking" is selected through the video search screen 200, and three video data files tagged with strings corresponding to the search value "nighttime" or "walking" (such as "nighttime", "walking", "walking") are found. In this case, the video search results screen 300 displays a list of pairs of video playback screens 310 and descriptions 320 for the three found video data files.

[0052] As described above, according to the monitoring system 1 of this embodiment, a string corresponding to a pre-stored search value (search keyword) as a video search condition is tagged to the video data, and the video data is searched based on the tagged string. With this configuration, the user of the terminal device 20 can easily search for the desired video data by selecting a search value on the video search screen 200. In addition, the user of the terminal device 20 can easily understand the content of the video without viewing it by referring to the video description 320 displayed on the video search results screen 300.

[0053] The operation of the server device 30 that assigns search tags to video data will be explained below with reference to Figures 8 to 12.

[0054] Prior to tagging the video data, the server device 30 generates captions for each frame of the video data that describe the actions of the subject 50 in the video. Then, based on the captions generated for the video, the server device 30 generates a video description (summary) and tags the video data.

[0055] First, with reference to Figures 8 and 9, the operation of the server device 30 that generates captions for the video will be explained.

[0056] Figure 8 is a flowchart showing the procedure for the caption generation process performed by the server device 30. The process shown in the flowchart in Figure 8 is executed by the control unit 31 according to the program stored in the storage unit 32 of the server device 30.

[0057] (Step S101) First, the server device 30 acquires video data of the subject 50. More specifically, the server device 30 acquires 2 minutes (600 frames) of video data obtained by imaging the subject 50 with the camera 13 of the detection device 10.

[0058] (Step S102) Next, the server device 30 generates captions. More specifically, the server device 30 generates captions for each frame image of the video data acquired in step S101, describing the actions of the subject 50 in the video. In this embodiment, the server device 30 inputs the video data into a learning model that has learned the relationship between images and the captions corresponding to those images. The learning model, upon receiving the video data, recognizes the actions (features) of the subject 50 in each frame image and generates a caption describing those actions for each frame image.

[0059] (Step S103) The server device 30 then stores the caption and terminates the process. More specifically, the server device 30 stores the text information of the caption generated in step S102 and the frame number information of the frame image to which the caption is applied in the storage unit 32, associating them with the video data, and then terminates the process.

[0060] As described above, according to the flowchart shown in Figure 8, captions explaining the actions of the subject 50 in the video are generated for each frame image of the video data obtained by capturing the subject 50 with the camera 13 of the detection device 10. Specifically, for example, various captions explaining each action of the subject 50 in various scenes of the video are generated.

[0061] Figure 9 shows an example of a frame image 400 after the caption generation process. As described above, in the caption generation process, a caption describing the actions of subject 50 is generated for each frame image of the video data of subject 50.

[0062] As shown in Figure 9, the frame image 400 after the caption generation process displays a caption 410 overlaid on it that describes the actions of the subject 50. Specifically, for example, in the frame image 400 shown in Figure 9, the caption 410, "The resident is sitting on the edge of the bed," is displayed overlaid on the bottom of the frame image 400 to describe the actions of the subject 50. Note that the actions of the subject 50 include not only the subject 50's own actions but also the passive actions of the subject 50 that are assisted by the facility staff.

[0063] As described above, the server device 30 generates captions for each frame image of a video using a learning model that has been trained with images and their corresponding captions as training data. For a specific frame image of a video, the learning model generates a caption for that frame image based on the caption corresponding to an image in the training data that is similar to that frame image. The training data is prepared, for example, by an operator manually adding captions (ground truth data) to frame images while viewing the video. The technique of generating a learning model with predetermined functions by providing training data to a learning model such as a neural network is a well-known machine learning technique, so a detailed explanation is omitted.

[0064] Next, referring to Figures 10 to 12, the operation of the server device 30 that tags the videos will be explained.

[0065] Figure 10 is a flowchart showing the procedure for tagging performed by the server device 30. The process shown in the flowchart in Figure 10 is executed by the control unit 31 according to the program stored in the storage unit 32 of the server device 30.

[0066] (Step S201) First, the server device 30 inputs a prompt to the document generation model. More specifically, the server device 30 inputs a prompt 500 (see Figure 11) to the document generation model (generation AI) instructing it to generate an explanatory text and tag the video.

[0067] Figure 11 shows an example of prompt 500. As shown in Figure 11, prompt 500 instructs the text generation model to generate a video description (summary) based on multiple captions. In addition, prompt 500 instructs the model to extract predetermined strings ("staff," "assistance," "sitting," "standing," etc.) from the video description and multiple captions and tag the video data with them. In Figure 11, "frequency information" refers to the number of frame images to which a particular caption is applied. The strings listed at the bottom of Figure 11 ("staff," "assistance," "sitting," "standing," etc.) are strings corresponding to search values ​​(search keywords) and are generated based on the string information stored in the storage unit 32 as search condition data 120.

[0068] (Step S202) Next, the server device 30 generates a description. More specifically, the server device 30 generates a video description based on the captions generated in the caption generation process shown in Figure 8. In this embodiment, the text generation model, which receives a prompt in step S201, generates a video description based on the information of multiple captions.

[0069] (Step S203) Next, the server device 30 performs tagging. More specifically, the server device 30 extracts strings from the captions generated in the caption generation process and the explanatory text generated in step S202 that are identical to the strings stored in the storage unit 32 as search condition data 120, and tags the video data. In this embodiment, the text generation model that receives a prompt in step S201 extracts strings from the captions and explanatory text that are identical to the strings listed below the prompt 500, and tags the video data with the extracted strings.

[0070] (Step S204) The server device 30 then stores the description and tags and terminates the process. More specifically, the server device 30 stores the text information of the description generated in step S202 and the text information of the tagged strings in step S203 in the storage unit 32 in association with the video data, and terminates the process.

[0071] As described above, according to the flowchart shown in Figure 10, a video description is generated based on the multiple captions generated for the video, and a string corresponding to the search value is extracted from the caption and description and tagged to the video data.

[0072] Figures 12A to 12C are diagrams illustrating the tagging process. Figure 12A illustrates the process of extracting text from captions, and Figure 12B illustrates the process of extracting text from descriptions. Figure 12C shows an example of text tagged with video data.

[0073] As described above, in the tagging process, strings corresponding to the search value (search keyword) are extracted from the caption generated for each video frame image and the video description generated based on the caption, and then tagged to the video data. Specifically, for example, as shown in Figure 12A, the strings "bed" and "sitting" corresponding to the search value "bed" or "sitting / standing" are extracted from the caption "sitting on the edge of the bed" generated for the video frame image. Also, as shown in Figure 12B, the strings "bed," "sitting," "wheelchair," "toilet," and "movement" corresponding to the search value "bed," "sitting / standing," "wheelchair," or "toilet" are extracted from the video description. Then, as shown in Figure 12C, the strings "bed," "sitting," "wheelchair," "toilet," and "movement" extracted from the caption and description are tagged to the video data.

[0074] Furthermore, for the search items "Time of Day" and "Nighttime" in the video search screen 200 shown in Figure 6, the strings "Daytime" or "Nighttime" are not extracted from the caption and description. Instead, the strings "Daytime" or "Nighttime" are tagged to the video data based on the time information of the video data.

[0075] As described above, according to the monitoring system 1 of this embodiment, a video description is generated for video data obtained by imaging the subject 50 with the camera 13, and a string corresponding to the search value (search keyword) is extracted from the description and tagged. With this configuration, search tags can be added even to videos without audio or subtitles.

[0076] (Editing search criteria) The video search screen 200 can be edited as appropriate by the user of the terminal device 20. Figure 13 shows an example of the search condition editing screen 600. The search condition editing screen 600 is a screen for editing the video search screen 200 and accepts edits of search values ​​by the user of the terminal device 20. The search condition editing screen 600 is displayed on the display unit 24 of the terminal device 20 based on display information transmitted from the server device 30.

[0077] As shown in Figure 13, the search condition editing screen 600 displays the search values ​​stored in the storage unit 32 as search condition data 120. The search values ​​are displayed for each of the five search items: "time period," "assistance required," "action," "location," and "object." Of the five search items, an add button 610 is provided for adding search values ​​for the search items "action," "location," and "object."

[0078] The user of terminal device 20 presses the add button 610 in the search item "Action" to add new search values ​​"Grooming" and "Reading" to the existing search values ​​"Getting up," "Sitting / Being seated," "Walking," and "Getting dressed." The user of terminal device 20 also presses the add button 610 in the search item "Place" to add new search values ​​"Toilet" and "Wheelchair" to the existing search values ​​"Bed" and "Chair." The user of terminal device 20 also presses the add button 610 in the search item "Object" to add new search values ​​"Clothes" and "Shoes" to the existing search values ​​"Wheelchair," "Walker," "TV," and "Washbasin." After that, the user of terminal device 20 presses the update button 620. When the update button 620 is pressed, the server device 30 updates the search value information and string information stored in the memory unit 32.

[0079] As a result, a video search screen 200 is obtained in which six search values, enclosed in a thick border in Figure 14, have been newly added. Then, as shown in Figures 15A and 15B, strings corresponding to the newly added search values ​​are additionally extracted from the video's caption and description and tagged with the video data. With this configuration, users of the terminal device 20 can search for videos by adding desired search values, further improving the searchability of videos.

[0080] The present invention is not limited to the embodiments described above, and can be modified in various ways within the scope of the claims.

[0081] For example, in the embodiment described above, a text generation model extracted a string corresponding to the search value from the video description, etc. However, the method for extracting a string from the video description, etc. is not limited to the method using a text generation model. For example, the control unit 31 of the server device 30 may extract a string corresponding to the search value from the video description, etc. by executing a predetermined string extraction program.

[0082] Furthermore, in the embodiment described above, strings corresponding to the search value were extracted from the video description and multiple captions, and tagged with the video data. However, unlike the embodiment described above, strings may be extracted only from the video description, without extracting strings from the captions.

[0083] Furthermore, in the embodiments described above, the text generation model generated the description independently of the string corresponding to the search value. However, unlike the embodiments described above, the text generation model may generate the description so as to include the string corresponding to the search value. For example, by inputting a prompt to the text generation model instructing it to generate a description that includes the string corresponding to the search value if it is included in the caption, the text generation model can generate a description that includes the string corresponding to the search value. With such a configuration, since the string corresponding to the search value is always included in the description, the user of the terminal device 20 can easily understand from the video description that an appropriate search has been performed.

[0084] Furthermore, in the embodiments described above, a model generated by fine-tuning a general-purpose text generation model was used as the text generation model that generates explanatory text from caption information. However, a model generated from scratch by training it with a large amount of data may also be used as the text generation model that generates explanatory text from caption information.

[0085] Furthermore, the above-described embodiment used the example of adding tags to a 2-minute video file. However, the video file to which tags are added is not limited to 2-minute video files; tags can be added to video files of various lengths.

[0086] Furthermore, in the embodiment described above, pairs of video playback screens 310 for displaying videos and explanatory texts 320 describing the content of the videos were displayed in a list on the video search results screen 300. However, the configuration of the video search results screen 300 is not limited to the embodiment described above. For example, only video playback screens 310 may be displayed in a list on the video search results screen 300. Alternatively, strings tagged with video data (see Figure 12C) may be displayed on the video search results screen 300 along with the video playback screen 310.

[0087] The processing units in the flowcharts of the embodiments described above are divided according to the main processing content in order to facilitate understanding of each process. The present invention is not limited by how the processing steps are classified. Each process can be further divided into more processing steps. Also, one processing step may perform even more processes.

[0088] In the embodiments described above, the functions of each device may be implemented by other devices. For example, the function implemented by the detection device 10 may be implemented by the server device 30.

[0089] The means and methods for performing various processing in the tagging device according to the above embodiment can be implemented by either a dedicated hardware circuit or a programmed computer. The program may be provided, for example, on a computer-readable recording medium such as a USB (Universal Serial Bus) memory or a DVD (Digital Versatile Disc)-ROM, or it may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a storage unit such as an HDD. Furthermore, the program may be provided as a standalone application software, or it may be incorporated into the software of the tagging device as a function of the device.

[0090] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are for illustrative purposes only and are not limiting. The scope of the present invention should be interpreted in accordance with the language of the appended claims. [Explanation of symbols]

[0091] 1. Monitoring system, 10 detection devices, 11,21,31 Control Unit, 12, 23, 33 Communications Department, 13 cameras, 20 terminal devices, 22,32 memory section, 24 Display section, 25 Input section, 30 server devices, 40 networks, 50 target individuals.

Claims

1. A memory unit that stores search values ​​for searching for videos, An acquisition unit that acquires video data obtained by imaging a subject with a camera, A description generation unit generates a description for the video data acquired by the acquisition unit, A tagging unit extracts a string corresponding to the search value from the description generated by the description generation unit and tags the video data with it. A tagging device having the following features.

2. The system further includes a caption generation unit that generates captions describing the actions of the subject in the aforementioned video on a frame-by-frame basis. The tagging device according to claim 1, wherein the description generation unit generates the description based on the caption generated by the caption generation unit.

3. The tagging device according to claim 2, wherein the description generation unit generates the description by inputting a prompt instructing the generation of the description to the text generation model.

4. The tagging device according to claim 3, wherein the tagging unit inputs the prompt to the text generation model, which further instructs the tagging of the video data, to extract the string from the description and tag the video data.

5. The tagging device according to any one of claims 2 to 4, wherein the tagging unit further extracts the string from the caption and tags the video data.

6. The tagging device according to claim 3 or 4, wherein the description generation unit generates the description containing the string by inputting the prompt instructing the text generation model to generate a description containing the string.

7. The tagging device according to any one of claims 2 to 4, wherein the caption generation unit generates the caption using a learning model that has learned an image and a caption corresponding to the image as training data.

8. The tagging device according to any one of claims 1 to 4, further comprising a reception unit for receiving the editing of the search value.

9. The tagging device according to any one of claims 1 to 4, wherein the search value is the search value of at least one search item selected from among the presence or absence of assistance to the subject, the subject's location, the subject's actions, and the object that the subject is acting on.

10. (a) A step of acquiring video data obtained by imaging the subject with a camera, Step (a) above involves generating a video description for the video data obtained in step (a), Step (c) involves extracting a string corresponding to a search value for searching for a video from the description generated in step (b) above, and tagging the video data with it. A tagging method that includes [specific features / features].

11. Procedure (a) for acquiring video data obtained by imaging a subject with a camera, The procedure (b) involves generating a description of the video for the video data obtained in the above procedure (a), Step (c) involves extracting a string corresponding to a search value for searching for a video from the description generated in step (b) above, and tagging the video data with it. A tagging program that causes a computer to execute a tagging command.

Images