Behavior record creation support system, behavior record creation support method, and behavior record creation support program

Abstract

To assist photographers in creating a record of their actions. [Solution] One aspect of the present invention is a behavioral record creation support system comprising a data acquisition unit that acquires video image data captured by a camera device carried by the photographer, a request receiving unit that receives a request from a user terminal device to create a behavioral record of the photographer, a request processing unit that outputs to an external system photographer behavior information related to the photographer's behavior generated based on at least a portion of the video image data and request information requesting the creation of a behavioral record of behavior specified by the photographer, and acquires draft data of the photographer's behavior record output from the external system, and a transmission unit that transmits the draft data to the user terminal device.

Description

[Technical Field]

[0001] The present invention relates to a behavior record creation support system, a behavior record creation support method, and a behavior record creation support program. [Background technology]

[0002] For example, a work report creation device is known, as described in Patent Document 1. This work report creation device generates work behavior record data of a train crew member based on an estimation result of the train crew member's work behavior. In the work report creation device, a sensing data acquisition unit acquires sensing data obtained by sensing the train crew member's movements, a work behavior estimation unit processes the sensing data obtained by sensing the train crew member's movements and estimates the type of work behavior performed by the train crew member, and a work behavior record data generation unit generates the train crew member's work behavior record data based on the estimation result of the type of work behavior performed by the train crew member. The work report creation device is composed of a mobile device such as a smartphone carried by the train crew member and equipped with a sensor that senses the train crew member's movements, and a computer that acquires sensing data from the mobile device via wireless communication or the like. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-163196 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, for example, to prepare a business report, an activity record may be created from video images captured over a long period of time. However, when a videographer creates a record of his or her own activity, there is a problem in that even with the use of image recognition software, it is not possible to recognize what the videographer is doing.

[0005] If the person shooting the video is not visible in the video, it is possible to create a behavioral record manually. However, if you try to create a behavioral record while watching the video, the time required to review the video will increase in proportion to the length of the video. Furthermore, if the video is long, spanning several hours, the time required to review the video will increase, making it unrealistic to manually review the video and create a behavioral record. Another method that can be considered is to have the person filming the video record the person's actions in a memo, without using video, and then create a behavioral record based on the memo and memory. However, there is a risk that the notes will be missed, lost, misremembered, or forgotten. Furthermore, there is the problem that the effort required to take notes and create a behavioral record will be increased.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a behavior record creation support system, a behavior record creation support method, and a behavior record creation support program that can assist videographers in creating behavior records. [Means for solving the problem]

[0007] (1) One aspect of the present invention is a behavioral record creation support system comprising: a data acquisition unit that acquires video image data captured by a camera device carried by a photographer; a request receiving unit that receives a request from a user terminal device to create a behavioral record of the photographer; a request processing unit that outputs to an external system photographer behavior information related to the photographer's behavior generated based on at least a portion of the video image data and request information requesting the creation of a behavioral record of the behavior specified by the photographer, and acquires draft data of the photographer's behavior record output from the external system; and a transmission unit that transmits the draft data to the user terminal device.

[0008] (2) In one aspect of the present invention, the request processing unit may output a search query to a first external device for searching the video data and a time range that includes a specified image from the video data, obtain from the first external device time information of the relevant image indicating the time range of the relevant image searched from the video data based on the search query, convert the time information of the relevant image into a correspondence between the cameraman's actions and time, output the request information including the correspondence between the cameraman's actions and time to a third external device, and obtain data representing a draft of the cameraman's action record generated by the third external device.

[0009] (3) In one aspect of the present invention, the request processing unit may output a request to a fourth external device instructing the fourth external device to create an action record of the video data and the action specified by the photographer contained in the video data, thereby causing the fourth external device to output a search query to a first external device to search for video corresponding to the specified action contained in the video data, causing the fourth external device to obtain the time range of the relevant video searched from the video data by the first external device based on the search query, and obtaining time information of the relevant video indicating the time range of the relevant video from the fourth external device.

[0010] (4) In one aspect of the present invention, the data acquisition unit acquires frame images and time information of the frame images from the video data, and the request processing unit outputs the frame images to a fifth external device, acquires caption data indicating captions for the frame images output from the fifth external device, outputs the caption data to a third external device, acquires behavioral information of the photographer generated by the third external device, outputs the request information including the behavioral information of the photographer to the third external device, and acquires data representing a draft of the behavioral record of the photographer generated by the third external device.

[0011] (5) In one aspect of the present invention, the request processing unit may output audio contained in the video data to a second external device, acquire text data generated by the second external device, and output the request information including a correspondence between the text data and the time of the audio to the third external device.

[0012] (6) In one aspect of the present invention, the data acquisition unit may acquire location data of a camera device held by the photographer, and the request processing unit may output to the third external device a correspondence between time information of the action specified by the photographer and the location data.

[0013] (7) In one aspect of the present invention, the request processing unit may output location information of the camera device to a sixth external device, obtain address information indicating an address corresponding to the location information of the imaging device output from the sixth external device, and output the correspondence between the time information and the address information to the third external device instead of the correspondence between time information and location data.

[0014] (8) One aspect of the present invention is a behavior record creation support method including the steps of: a behavior record creation support device acquiring video image data captured by a camera device carried by a photographer; a behavior record creation support device receiving a request from a user terminal device to create a behavior record of the photographer; a behavior record creation support device outputting to an external system photographer behavior information related to the photographer's behavior generated based on at least a portion of the video image data and request information requesting the creation of a behavior record of the behavior specified by the photographer; a behavior record creation support device acquiring draft data of the photographer's behavior record output from the external system; and a behavior record creation support device transmitting the draft data to the user terminal device.

[0015] (9) One aspect of the present invention is a program that causes a computer of a behavioral record creation support device to execute the following steps: acquiring video data captured by a camera device carried by a photographer; accepting a request from a user terminal device to create a behavioral record of the photographer; outputting to an external system photographer behavior information related to the photographer's behavior generated based on at least a portion of the video data and request information requesting the creation of a behavioral record of the behavior specified by the photographer; acquiring draft data of the photographer's behavior record output from the external system; and transmitting the draft data to the user terminal device. [Effects of the Invention]

[0016] According to one aspect of the present invention, it is possible to assist a photographer in creating a behavior record. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing an example of a behavior record creation support system 1 according to a first embodiment. [Figure 2] FIG. 2 is a sequence diagram showing an example of a processing procedure of the behavior record creation support system 1 in the first embodiment. [Figure 3] 4 is an example of a display screen on the user terminal device 300 in the first embodiment. [Figure 4] 10 is a diagram showing the correspondence between the actions of a photographer, images captured during the photographer's actions, and search queries in the first embodiment. FIG. [Figure 5] FIG. 2 is a diagram showing an example of a screen for displaying a search query, search results, and images in a searched video in the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a prompt between the request processor 106 and the draft data creation device 230 in the first embodiment. [Figure 7] FIG. 10 is a diagram showing another example of a prompt between the request processor 106 and the draft data creation device 230 in the first embodiment. [Figure 8] FIG. 3 is a diagram showing an example of a display screen of an action record in the first embodiment. [Figure 9] FIG. 10 is a block diagram showing an example of a behavior record creation support system 1A according to a second embodiment. [Figure 10] FIG. 10 is a sequence diagram showing an example of a processing procedure of the behavior record creation support system 1A in the second embodiment. [Figure 11] FIG. 10 is a diagram showing an example of a prompt between a request processor 106A and a generating device 250 in the second embodiment. [Figure 12] FIG. 10 is a block diagram showing an example of a behavior record creation support system 1B according to a third embodiment. [Figure 13] FIG. 11 is a sequence diagram showing an example of a processing procedure of the behavior record creation support system 1B according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, a behavior record creation support system, a behavior record creation support method, and a behavior record creation support program to which the present invention is applied will be described with reference to the drawings. The behavioral record creation support system, behavioral record creation support method, and behavioral record creation support program embodying the present invention support the creation of behavioral record information that records the photographer's behavior using long videos shot by the photographer. The photographer, for example, captures long videos spanning several hours or numerous videos and still images during working hours, uploads the long videos to the behavioral record creation support system, and requests the creation of a behavioral record. In response, the behavioral record creation support system creates draft data for a behavioral record that chronologically describes the photographer's activities during working hours and presents it to the photographer, thereby supporting the photographer in creating the behavioral record. Details of the behavioral record creation support system are described below.

[0019] (First embodiment) FIG. 1 is a block diagram showing an example of a behavior record creation support system 1 according to the first embodiment. The behavior record creation support system 1 includes, for example, a behavior record creation support device 100, an external system 200, a user terminal device 300, a camera device 400, and a data storage device 500. The behavior record creation support device 100, the external system 200, the user terminal device 300, the camera device 400, and the data storage device 500 are each equipped with a communication interface (not shown in FIG. 1 ) such as a NIC (Network Interface Card) or a wireless communication module, and are connected via a communication network. The communication network includes, for example, the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), a cellular network, etc. The user in this embodiment is a person who creates an action record. The user wears, for example, a helmet equipped with a camera device 400 and takes moving images in which the user does not appear.

[0020] The user terminal device 300 is a smartphone or the like operated by a user. The user terminal device 300 starts up the camera device 400 by accepting a user operation. By accepting a user operation, the user terminal device 300 transmits a behavior record creation request to the behavior record creation support device 100 and receives draft data of the behavior record from the behavior record creation support device 100. The camera device 400 takes a picture by being operated by the user and generates video data. The camera device 400 uploads the generated video data to the data storage device 500.

[0021] In this embodiment, the user takes a photograph using the camera device 400, and the user terminal device 300 sends a request to create an action record based on the user's operation, but this is not limited to this, and the person who takes the photograph using the camera device 400 and the person who requests the creation of the action record may be different persons. Furthermore, in cases where the user terminal device 300 and the camera device 400 are integrated, the user terminal device 300 may upload video data.

[0022] The data storage device 500 stores moving image data uploaded from the camera device 400, and transmits the moving image data in response to a request from the action record creation support device 100 to the action record creation support device 100. The data storage device 500 may transmit information indicating the moving image data stored in the data storage device 500 to the user terminal device 300 in response to a request from the user terminal device 300, and may transmit moving image data specified by the user to the action record creation support device 100. Furthermore, the data storage device 500 may acquire, in addition to the moving image data, current frame image information of the camera device 400 included in the moving image, latitude and longitude information of the camera device 400, etc.

[0023] The behavior record creation support device 100 is an information processing device such as a server device that acquires various types of information and executes various types of information processing based on the acquired information. The behavior record creation support device 100 functions as a server device that controls the entire behavior record creation support system 1 including the internal components of the behavior record creation support device 100. The behavior record creation support device 100 includes, for example, a data acquisition unit 102, a request reception unit 104, a request processing unit 106, and a transmission unit 108. Functional units such as the data acquisition unit 102, the request reception unit 104, the request processing unit 106, and the transmission unit 108 are realized by a processor such as a CPU (Central Processing Unit) executing a behavior record creation support program stored in a program memory. Furthermore, some or all of these functional units may be realized by hardware such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array), or may be realized by a combination of software and hardware.

[0024] The data acquisition unit 102 acquires the video data and the position data captured by the camera device 400 held by the photographer. Note that acquiring the position data may be optional. The request receiving unit 104 receives a request from the user terminal device 300 to create a behavior record of the photographer. The request processing unit 106 outputs to the external system 200 photographer behavior information related to the photographer's behavior generated based on at least a part of the video data, text data indicating the audio in the video data, and request information requesting the creation of an action record of the behavior specified by the photographer, and obtains the draft data of the photographer's action record output from the external system 200. Note that the request information in this embodiment is information requesting an action record including text data indicating the audio in the video data, but is not limited to this and may also be information requesting the creation of an action record that does not include text data indicating the audio in the video data. The transmitting unit 108 transmits the draft data to the user terminal device 300 .

[0025] The external system 200 includes, for example, an image search device 210, a voice conversion device 220, a draft data creation device 230, and an address search device 240. Each of the image search device 210, the voice conversion device 220, the draft data creation device 230, and the address search device 240 is an information processing device that provides a service of performing various processes in response to a request from the behavior record creation support device 100 and transmitting the processing results as a response to the behavior record creation support device 100. Note that, although the voice conversion device 220 and the address search device 240 are included in this embodiment, the voice conversion device 220 and the address search device 240 are not essential components and may not be included in the external system 200.

[0026] The image search device 210 is a first external device that searches video data for a time range that includes video specified by a search query of the behavior record creation support device 100, and returns the search results to the behavior record creation support device 100. The image search device 210 may be, for example, a device that searches for videos or parts of videos that include audio or images highly relevant to the search query by vector search using a machine learning model. The machine learning model has a function of inputting images, text, etc., and outputting vectors that are more similar to each other the more relevant they are.

[0027] The function of the image search device 210 to "search (extract) a time range containing a specified video from video data" may be one of a group of image AI services that input an image and output various information. The group of image AI services may include a function (service) that outputs a caption when an image is input. Note that the image search device 210 is connected to the behavior record creation support device 100 via a communication network, but is not limited to this, and may be connected to the behavior record creation support device 100 locally.

[0028] The voice conversion device 220 is a second external device that receives voice data from the behavior record creation support device 100, converts the voice into text, and responds to the behavior record creation support device 100. The "convert voice to text" function may be one of a group of voice AI services that receive voice input and output various information. Note that the voice conversion device 220 is connected to the behavior record creation support device 100 via a communication network, but is not limited to this, and may be connected locally to the behavior record creation support device 100.

[0029] The draft data creation device 230 is a third external device that receives, from the action record creation support device 100, photographer action information related to the photographer's actions generated based on at least a portion of the video data, text data indicating the audio in the video data, and request information requesting the creation of an action record of the specified photographer's actions, and outputs draft data of the photographer's action record. The draft data of the photographer's action record is, for example, text data in which the video photographer's actions and audio are arranged in chronological order.

[0030] The draft data creation device 230 utilizes a machine learning model that has been trained to output a natural response when natural language is input, and outputs draft data as a natural response when request information is input in natural language. The machine learning model may be, for example, a large-scale language model (LLM), and may be locally connected to the behavior record creation support device 100. The draft data creation device 230 may be, for example, image processing software that arranges the photographer's behavior and voice in chronological order according to predetermined rules.

[0031] The address search device 240 is a sixth external device that receives latitude and longitude information from the activity record creation support device 100, converts it into address information indicating an address, and transmits the address information to the activity record creation support device 100 as a response. The address search device 240 may be one of a group of services for handling geographical information and may output the latitude and longitude information to one of the services. Furthermore, instead of converting it into address information, the address search device 240 may perform a process of converting the latitude and longitude into information in a format that is easy for the user to understand. When the request processing unit 106 acquires address information from the address search device 240, the request processing unit 106 may output the correspondence between time information and address information to the draft data creation device 230 instead of the correspondence between time information and location data.

[0032] FIG. 2 is a sequence diagram showing an example of a processing procedure of the behavior record creation support system 1 in the first embodiment. First, the user terminal device 300 attaches the camera device 400 and presses the recording button to transmit a request S100 to the camera device 400 to start recording. In response to the request S100, the camera device 400 starts recording and transmits video data and location information S102 to the data storage device 500. The camera device 400 may be attached to something the photographer is wearing, such as a helmet or breast pocket, and does not necessarily have to be attached to the user terminal device 300 that requests the creation of an activity record, such as a smartphone, tablet, or PC.

[0033] The data acquiring unit 102 transmits a location request S104 to the data storage device 500 to acquire location information. The data acquiring unit 102 may acquire location information by, for example, periodically transmitting the location request S104. The location request S104 may be a request to acquire GPS information active in the camera device 400 from the data storage device 500, and the data storage device 500 transmits a response including the latitude and longitude to the behavior record creation support device 100.

[0034] The user terminal device 300 transmits an action record creation request S106 to the action record creation support device 100 in response to a user operation, and the request receiving unit 104 receives the action record creation request S106. FIG. 3 shows an example of a display screen on the user terminal device 300 in the first embodiment. The user terminal device 300 accepts an operation to select video data as a screen for requesting creation of an action record, and accepts an operation to press the action record creation button. As a result, the user terminal device 300 transmits an action record creation request S106 including information indicating the video data to the action record creation support device 100.

[0035] The data acquisition unit 102 transmits a request S108 to the data storage device 500 to acquire the shooting start time and video data for the selected video. The data storage device 500 transmits the requested shooting start time and video data to the action record creation support device 100.

[0036] The request processing unit 106 transmits a request S110 including latitude and longitude information to the address searching device 240. The address searching device 240 converts the latitude and longitude information into an address and transmits the address information S112 to the action record creation support device 100. The request processing unit 106 may periodically acquire position information of the camera device 400 and output a request to the address searching device 240 to convert a position corresponding to the time of the video data into an address or the like that is easy for the user to understand. Note that the request processing unit 106 does not need to output the request S110 to the address searching device 240 when it is set not to cooperate with the address searching device 240.

[0037] Next, the request processing unit 106 outputs a request S114 including a search query and a request to search for video data and a time range containing the specified video from the video data to the image search device 210 (first external device), and obtains from the image search device 210 relevant video time information S116 indicating the time range of the specified video searched from the video data based on the search query.

[0038] FIG. 4 is a diagram showing the correspondence between the actions of a photographer, videos captured during the photographer's actions, and search queries in the first embodiment. The request processing unit 106 outputs a search query corresponding to the photographer's behavior specified by the user or a preset photographer's behavior. This allows the request processing unit 106 to obtain a time range that includes footage corresponding to the photographer's behavior. For example, if the photographer's behavior is "driving," the request processing unit 106 sends a search query of "Car steering wheel" to the image search device 210. This allows the request processing unit 106 to cause the image search device 210 to search for video data based on the search query and generate a time range that indicates when the "car steering wheel" is in view.

[0039] Image retrieval from a long video by the image retrieval device 210 is a process of searching all time ranges of video in which a specified object or action is captured from a long video, such as a process of searching all time ranges in which a car steering wheel is captured. The request processor 106 may transmit a search query that includes multiple search queries and indicates multiple videos. The image search device 210 repeats the video search as many times as the number of videos corresponding to each of the multiple search queries. For example, if three types of cameraman behaviors (conversation, construction attendance, and car movement) are to be extracted, repeating the search as many times as the number of video time ranges corresponding to each cameraman behavior means searching for video time ranges corresponding to each of the cameraman behaviors.

[0040] The request processing unit 106 inputs, for example, a search query "Car steering wheel" to the image search device 210 and requests a search for video corresponding to the photographer's actions to be extracted. The image search device 210 searches the video data for a time range in which a car steering wheel is shown in response to the search query. The time range in which a car steering wheel is shown is, for example, "01:01 to 01:40" or "03:10 to 03:55," and transmits this to the action record creation support device 100 as time range information for the video in which the car steering wheel is shown. This time range can be treated as the time during which the photographer was driving. The request processing unit 106 converts the playback time of the video to real time using the shooting start time. For example, if the shooting start time is "2024 / 7 / 1 10:00:00," the request processing unit 106 determines that the photographer was driving between "2024 / 7 / 11:01 to 11:40 and 13:10 to 13:55." The image search device 210 similarly searches for three types of video, for example, video showing a car steering wheel, video showing a construction site, and video of a person talking to a camera.

[0041] FIG. 5 is a diagram showing an example of a screen for displaying a search query, search results, and time ranges of images in the searched moving images in the first embodiment. The image search device 210 searches for images corresponding to the search query in a long video and outputs the search results to the behavior record creation support device 100, and the behavior record creation support device 100 may display a screen such as that shown in Figure 5 in response to a request from the user terminal device 300, for example.

[0042] Next, the request processing unit 106 outputs a request S118 including the audio included in the video data and a request to transcribe the audio to the audio conversion device 220 (second external device), and acquires the text data generated by the audio conversion device 220 as the transcription result S120. The audio conversion device 220 performs a process of transcribing the audio data of the long video.

[0043] The request processing unit 106 extracts audio data from the video data, divides it into multiple pieces, and transmits the divided audio data to the audio conversion device 220, which then transcribes each piece of divided audio data. As a result, the audio conversion device 220 creates transcription results including the content of the conversation every 10 minutes, for example, and creates a correspondence between the time every 10 minutes and the transcription results. The request processing unit 106 can create a correspondence between the actual date and time and the transcription results by correlating the time every 10 minutes with the actual date and time, taking into account the start time of filming.

[0044] The request processing unit 106 may output audio data from a long video data stream within a time range determined to be when the cameraman was speaking to the audio conversion device 220. This allows the audio conversion device 220 to create a transcription result of the audio from the time range when the cameraman was speaking. The request processing unit 106 can create a correspondence between the actual date and time and the transcription result. By the request processing unit 106 sending a portion of the audio data from the video data stream to the audio conversion device 220, the cost of receiving services from the audio conversion device 220 can be reduced.

[0045] Next, the request processing unit 106 outputs request information S122, which includes a correspondence between information indicating the photographer's behavior and time, a correspondence between the time of the photographer's behavior and location data, and a correspondence between the audio text data and the audio time, to the draft data creation device 230 (third external device, LLM), and obtains draft data S124 representing a draft of the photographer's behavior record generated by the draft data creation device 230.

[0046] FIG. 6 is a diagram showing an example of a prompt between the request processor 106 and the draft data creating device 230 in the first embodiment. The request processing unit 106 transmits to the draft data creation device 230 request information S122 including prompts such as, for example, "You are a supervisor who patrols a construction site" as the photographer's (user's) position, and "We will now create an activity record from the notes recording your daily activities in the following procedure: 1. Summarize the conversation, 2. Link the activities with the locations, ..." and "Please summarize the notes transcribed from the following conversations for each conversation in the format of the sample answers" as the request content and the activity record format. In response, the draft data creation device 230 transmits an output including draft data S124 including the answers and the conversations to the activity record creation support device 100.

[0047] The request processing unit 106 may acquire the draft data S124 by transmitting the request information S122 once, or may transmit the request information S122 multiple times to acquire multiple draft data S124. For example, the request processing unit 106 may first transmit the entire flow to the draft data creation device 230, transmit multiple pieces of request information S122 in stages in chronological order, and acquire the draft data S124 in chronological order. The request processing unit 106 may also transmit examples of draft data or past draft data for past request information to the draft data creation device 230, and request the draft data creation device 230 to create draft data S124 based on the past examples.

[0048] FIG. 7 is a diagram showing another example of a prompt between the request processor 106 and the draft data creating device 230 in the first embodiment. The request processor 106 requests a summary of the transcription results as shown in FIG. 6, and then outputs all information, such as the activity, conversation content, and location, to the draft data creation device 230 (LLM) as shown in FIG. 7, and requests that it be compiled into an activity record format. Because the LLM is not good at accurate arithmetic calculations, the request processor 106 converts the video capture start time to the video playback time into a date and time, and enters the converted time in a prompt to make the request. The request processor 106 also requests actions to be recorded in the activity record, which vary depending on the photographer, such as "getting on board, getting off, conversation, and construction supervision," as well as prompts such as "If the activity is a conversation, please write a summary of the conversation content in the conversation content field," as well as the start time of the capture, the time of the conversation, and an example of an activity record.

[0049] The request processing unit 106 transmits the draft data S126 acquired from the draft data creation device 230 to the user terminal device 300. The user terminal device 300 displays the draft of the action record. 8 is a diagram showing an example of a display screen of the activity record in the first embodiment. The user terminal device 300 can correct the draft of the activity record and add special notes in response to a user operation.

[0050] As described above, the behavior record creation support system 1 of the first embodiment acquires video data and location data captured by the camera device 400 carried by the photographer, and when a request to create an behavior record of the photographer is received from the user terminal device 300, it outputs to the external system 200 photographer behavior information related to the photographer's behavior generated based on at least a part of the video data, text data indicating audio in the video data, and request information requesting the creation of an behavior record of the behavior specified by the photographer, and can acquire the draft data of the photographer's behavior record output from the external system 200. This behavior record creation support system 1 can support the creation of the photographer's behavior record. The behavioral record creation support system 1 may be configured such that the external system 200 is made up of a single device, and as described above, the external system 200 does not have to be divided into three devices such as the image search device 210, the voice conversion device 220, and the draft data creation device 230, but may be divided into two devices, or even into four or more devices.

[0051] In the embodiment, when the external system 200 is divided into a first external device, a second external device, and a third external device, the request processor 106 may function as follows. The request processing unit 106 outputs a search query to the image search device 210 for searching for video data and a time range containing a specified video from the video data, and obtains from the image search device 210 relevant video time information indicating the time range of the specified video searched from the video data based on the search query. The request processor 106 outputs the audio included in the video data to the audio converter 220 and obtains the text data generated by the audio converter 220 . The request processing unit 106 outputs request information including a correspondence between information indicating the photographer's behavior and time, a correspondence between the time of the photographer's behavior and location data, and a correspondence between audio text data and the time of the audio to the draft data creation device 230, and acquires data representing a draft of the photographer's behavior record generated by the draft data creation device 230. As a result, the behavior record creation support system 1 can support the photographer in creating a behavior record in the same way as when the external system 200 is a single device.

[0052] (Second embodiment) The second embodiment will be described below, with the same parts as those in the first embodiment being given the same reference numerals. FIG. 9 is a block diagram showing an example of a behavior record creation support system 1A according to the second embodiment. The behavior record creation support device 100A of the behavior record creation support system 1A in the second embodiment differs from the first embodiment in that it does not directly use the image search device 210A of the external system 200A, but instead outputs a request to the generation device 250 (fourth external device) to search for a time range containing video data and a pre-set video, and the generation device 250 requests the image search device 210A to extract the time range of the relevant video.

[0053] The request processing unit 106A outputs a request to the generation device 250 to instruct it to search for a time range containing video data and video included in the video data that corresponds to a preset action of the photographer. In response, the request processing unit 106A causes the generation device 250 to output a search query to the image retrieval device 210A to search for specified video included in the video data, and causes the generation device 250 to acquire relevant video time information indicating the time range containing the specified video searched from the video data by the image retrieval device 210A based on the search query. The request processing unit 106A acquires the relevant video time information from the generation device 250.

[0054] FIG. 10 is a sequence diagram showing an example of a processing procedure of the behavior record creation support system 1A in the second embodiment.

[0055] The request processing unit 106A outputs a request S130 to the generation device 250 (fourth external device) to search for a time range that includes video data and video corresponding to a preset action of the photographer. The generation device 250 outputs a request S132 to the image retrieval device 210A to search for a time range that includes the specified video in response to the request S130. In response to the search request S132 for the time range that includes the specified video, the image retrieval device 210A generates corresponding video time information S134 that includes the frame image and time of the specified video and outputs it to the generation device 250. The generation device 250 outputs corresponding video time information S136 to the request processing unit 106A as a response to the request S130.

[0056] FIG. 11 is a diagram showing an example of a prompt between the request processor 106A and the generating device 250 in the second embodiment. The request processing unit 106A transmits a request S130 to the generation device 250, which includes, for example, video data and a prompt such as, "I will now create an activity record from footage of one day recorded by a camera attached to a person's helmet. Please note that the footage is from a first-person perspective, so the person for whom the activity record is to be created does not appear in the footage. Please watch the following video and tell me the playback times of scenes in which a person speaking directly to me, i.e., scenes in which the person for whom the activity record is to be created is having a conversation, in the following format: #Answer, 01:00-01:10, 02:20-01:25." In response, the generation device 250 generates a search query to search the image search device 210 for footage of a person speaking directly to me based on the request S130, and outputs a search request S132 including the search query. As a result, generation device 250 can output frame images (for example, 20 frames, S134) of a video showing a person speaking to the user from image retrieval device 210A to generation device 250, and output corresponding video time information S136 from generation device 250 to request processing unit 106A. Note that the time information of the frame images (corresponding scenes) may be extracted by image retrieval device 210A, but generation device 250 may also extract the time in the video data containing the frame images.

[0057] Furthermore, the request processing unit 106A may output video data and a question to the generation device 250. For example, when the photographer requests that the speed and distance on the vehicle's speedometer be included in the action record, the request processing unit 106A outputs a question such as "Please read the number on the odometer of the speedometer from the video data" to the generation device 250. In response to this, the generation device 250 may output a request to the image retrieval device 210A to extract frame images including a speedometer, read the number on the odometer of the speedometer from the frame images acquired from the image retrieval device 210A, and transmit the answer to the request processing unit 106A.

[0058] When there are multiple actions of the specified photographer, the request processing unit 106A creates a prompt for each video corresponding to the photographer's action and outputs the request S130 to the generation device 250, and the generation device 250 outputs a search request S132 for each specified video to the image retrieval device 210A. As a result, the image retrieval device 210A generates corresponding video time information S134 including the frame image and time of the specified video and outputs it to the generation device 250, and the request processing unit 106A obtains corresponding video time information S136 for each specified video from the generation device 250 for each video corresponding to the photographer's action.

[0059] Next, the request processing unit 106A outputs a request S138 including the voice included in the video data and a request to transcribe the voice to the voice conversion device 220 (second external device), and acquires the text data generated by the voice conversion device 220 as the transcription result S140. The voice conversion device 220 performs a process of transcribing the voice data of the long video.

[0060] The request processing unit 106A extracts audio data from the video data, divides it into multiple pieces, and transmits the divided audio data to the audio conversion device 220, which then transcribes each piece of divided audio data. As a result, the audio conversion device 220 creates a transcription result including the content of the conversation every 10 minutes, for example, and creates a correspondence between the time every 10 minutes and the transcription result. The request processing unit 106A can create a correspondence between the actual date and time and the transcription result by correlating the time every 10 minutes with the actual date and time, taking into account the start time of filming.

[0061] The request processing unit 106A may output audio data of a time range of long video data in which it is determined that the cameraman was talking to the audio conversion device 220. This causes the audio conversion device 220 to create a transcription result of the audio in the time range in which the cameraman was talking. The request processing unit 106A can create a correspondence between the actual date and time and the transcription result. By the request processing unit 106A sending a portion of the audio data of the video data to the audio conversion device 220, the cost of receiving services from the audio conversion device 220 can be reduced.

[0062] Next, the request processing unit 106A outputs request information S142, which includes a correspondence relationship between information indicating the photographer's behavior and time, a correspondence relationship between the time of the photographer's behavior and location data, and a correspondence relationship between the voice text data and the voice time, to the draft data creation device 230 (third external device, LLM), and acquires draft data S144 representing a draft of the photographer's behavior record created by the draft data creation device 230. The request processing unit 106A transmits the acquired draft data S144 to the user terminal device 300, and causes the user terminal device 300 to display the draft of the behavior record.

[0063] As described above, according to the second embodiment of the activity record creation support system 1A, the request processing unit 106A outputs a request to the generation device 250 to instruct the generation device 250 to search for video data and a time range containing video corresponding to a predetermined action included in the video data. This causes the generation device 250 to output a search query to the image retrieval device 210A to search for a specified video included in the video data. The generation device 250 then acquires relevant video time information indicating the specified video time searched from the video data based on the search query, and the relevant video time information can be acquired from the generation device 250. According to the activity record creation support system 1A, even if the request processing unit 106A does not set a search query in advance, the generation device 250 can output a search request to the image retrieval device 210A and cause the image retrieval device 210A to search for the specified video. This allows the activity record creation support system 1A to support the photographer in creating an activity record.

[0064] (Third embodiment) The third embodiment will be described below, with the same reference numerals used to designate the same parts as those in the above-described embodiment. FIG. 12 is a block diagram showing an example of a behavior record creation support system 1B according to the third embodiment. The external system 200B in the third embodiment differs from the above-described embodiments in that it includes a caption generation device 260 as a fifth external device. The caption generation device 260 is an information processing device having a function of generating captions for frame images included in video data transmitted from the behavior record creation support device 100B. A caption is text data that explains a frame image. The caption generation device 260 may be, for example, a machine learning model trained using still image data and information indicating a caption as training data. The machine learning model has a function of outputting information indicating a caption when a frame image is input.

[0065] In the action record creation support device 100B according to the third embodiment, the data acquisition unit 102B acquires frame images and time information for the frame images from video data. The request processing unit 106B outputs the frame images to the caption generation device 260 and acquires caption data indicating the captions for the frame images output from the caption generation device 260. The request processing unit 106B may also output the audio included in the video data to the audio conversion device 220 and acquire text data generated by the audio conversion device 220. Before creating the action record, the request processing unit 106B extracts the start time and end time of the action specified by the photographer from the captions of the chronologically ordered image frames. The request processing unit 106B may associate the caption with the action of the photographer on a rule basis, for example, determining that the photographer is driving if the caption contains the word "steering wheel." The request processing unit 106B outputs request information to the draft data creation device 230, which includes photographer behavior information related to the photographer's behavior created from caption data, a correspondence between the photographer's behavior, time information, and location information, and a correspondence between text data corresponding to the audio and time, and obtains data representing a draft of the photographer's behavior record created by the draft data creation device 230. When the draft data creation device 230 uses the LLM to generate data representing a draft, the request processing unit 106B supplements the fact that the subject for which an action record is to be created is not visible in the captured video, and outputs information specifying what caption should be associated with what action of the photographer in the first-person video to the draft data creation device 230. For example, the request processing unit 106B may send a message to the LLM by which the draft data creation device 230 writes, "The following memo is a caption generated for what was captured on the camera that filmed your daily actions from your helmet." Using the notes below, please summarize what you were doing and at what time, using the examples below. Please note that you will not appear in the video you took. Please include the start and end times for activities such as car travel, conversations, and construction supervision. If the interior of the car, such as the steering wheel or speedometer, is visible, you can assume that the car is moving. If you see someone facing the camera and talking, you can assume that a conversation is taking place. If a construction site is visible, please assume that construction is being supervised."

[0066] FIG. 13 is a sequence diagram showing an example of a processing procedure of the behavior record creation support system 1B in the third embodiment. First, the user terminal device 300 attaches the camera device 400 and presses the recording button to transmit a request S100 to the camera device 400 to start recording. In response to the request S100, the camera device 400 starts recording and transmits video data and location information S102 to the data storage device 500.

[0067] The data acquiring unit 102B acquires the location information by transmitting a location request S104 to the data storage device 500. The data acquiring unit 102B may acquire the location information by, for example, periodically transmitting the location request S104. The user terminal device 300 transmits an action record creation request S106 to the action record creation support device 100B in response to a user operation, and the request receiving unit 104 receives the action record creation request S106.

[0068] The data acquisition unit 102B transmits a request S108 to acquire the shooting start time and moving image data for the selected moving image to the data storage device 500. The data storage device 500 transmits the requested shooting start time and moving image data to the action record creation support device 100B.

[0069] The request processing unit 106B transmits a request S110 including the latitude and longitude information to the address search device 240. The address search device 240 converts the latitude and longitude information into an address, and transmits the address information S112 to the behavior record creation support device 100B.

[0070] Next, the request processing unit 106B outputs a request S150 to generate frame images included in the video data and captions for the frame images to the caption generation device 260, and acquires caption data S152 from the caption generation device 260. For example, if the frame image includes a monitor and a person operating the monitor next to it, the caption generation device 260 generates caption data such as "a man pointing at a screen."

[0071] Next, the request processing unit 106B outputs a request S118 to the speech conversion device 220, which includes a request to transcribe the audio and audio contained in the video data, and obtains the text data generated by the speech conversion device 220 as the transcription result S120.

[0072] The request processing unit 106B may output audio data from the long video data within a time range in which it is determined that the cameraman was active to the audio conversion device 220. This causes the audio conversion device 220 to create a transcription result of the audio within the time range in which the cameraman was talking. The request processing unit 106B creates a correspondence between the actual date and time and the transcription result. By the request processing unit 106B sending a portion of the audio data from the video data to the audio conversion device 220, the cost of receiving services from the audio conversion device 220 can be reduced. Furthermore, as described above, before creating the behavioral record, the request processing unit 106B may output an instruction S154 to the draft data creation device 230 to extract the behavior of the photographer from the caption, and obtain the results S156 of extracting the start time and end time of the behavior specified by the photographer from the captions of the image frames in chronological order from the output to the draft data creation device 230.

[0073] Next, the request processing unit 106B outputs request information including photographer behavior information related to the photographer's behavior created from the caption data, a correspondence between the photographer's behavior, time information, and location information, and a correspondence between text data corresponding to the audio and the time of the audio to the draft data creation device 230. Then, the request processing unit 106B acquires draft data S124 representing a draft of the photographer's behavior record created by the draft data creation device 230. The request processing unit 106B transmits the draft data S126 acquired from the draft data creation device 230 to the user terminal device 300. The user terminal device 300 displays the draft of the behavior record.

[0074] As described above, according to the behavior record creation support system 1B of the third embodiment, the request processing unit 106B outputs frame images to the caption generation device 260, obtains caption data indicating the caption of the frame image output from the caption generation device 260, outputs the caption data to the draft data creation device 230, obtains behavior information of the photographer, outputs request information including the behavior information of the photographer, correspondence between the photographer's behavior, time information, and location information, and correspondence between text data corresponding to audio and the time of the audio to the draft data creation device 230, and obtains data representing a draft of the behavior record of the photographer created by the draft data creation device 230. In this way, the behavior record creation support system 1B can support the creation of the behavior record of the photographer.

[0075] Although each embodiment and each variant have been described, these are merely examples and are not intended to limit the scope of the present invention. For example, one aspect of the present invention may be realized by combining any of the embodiments or variants, or a part of each embodiment or a part of each variant, with one or more other embodiments or one or more other variants.

[0076] The various processes described above relating to the behavior record creation support devices 100, 100A, and 100B may be performed by recording a program for executing each process of the behavior record creation support devices 100, 100A, and 100B in this embodiment on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system.

[0077] Note that the term "computer system" here may include hardware such as the OS and peripheral devices. Furthermore, if a WWW system is used, the term "computer system" also includes the homepage provision environment (or display environment). Furthermore, "computer-readable recording media" refers to storage devices such as flexible disks, magneto-optical disks, ROMs, and writable non-volatile memory such as flash memory, portable media such as CD-ROMs, and hard disks built into computer systems.

[0078] Furthermore, the term "computer-readable recording medium" also includes a storage medium that stores a program for a certain period of time, such as a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within a computer system that serves as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. The program may also be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium.

[0079] Here, the "transmission medium" for transmitting the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in a computer system.

[0080] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and the present invention also includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0081] 1, 1A, 1B Behavioral Record Creation Support System 100, 100A, 100B Behavioral record creation support device 102, 102B Data acquisition section 104 Request Reception Unit 106, 106A, 106B Request processing unit 108 Transmitter 200, 200A, 200B External Systems 210, 210A Image search device 220 Voice conversion device 230 Draft data creation device 240 Address Search Device 250 Generator 260 Caption Generator 300 User terminal device 400 Camera Equipment 500 Data storage device

Claims

1. a data acquisition unit that acquires video data captured by a camera device carried by a photographer; a request receiving unit that receives a request to create a behavior record of the photographer from a user terminal device; a request processing unit that outputs to an external system photographer behavior information related to the photographer's behavior generated based on at least a portion of the video data and request information requesting the creation of an action record of the photographer's designated behavior, and acquires draft data of the photographer's action record output from the external system; a transmitting unit that transmits the draft data to the user terminal device; A behavioral record creation support system that includes the above.

2. The request processing unit outputting to a first external device a search query for searching the video data and a time range in which a specified video is included from the video data, and obtaining from the first external device time information of the relevant video, which indicates a time range of the relevant video searched from the video data based on the search query; converting the time information of the video into a correspondence relationship between the photographer's actions and time; outputting the request information including the correspondence between the photographer's actions and time to a third external device, and acquiring data representing a draft of the photographer's action record generated by the third external device; The behavior record creation support system according to claim 1.

3. The behavior record creation support system of claim 2, wherein the request processing unit outputs a request to a fourth external device instructing the fourth external device to create a behavior record of the video data and the behavior specified by the photographer contained in the video data, thereby causing the fourth external device to output a search query to a first external device to search for video corresponding to the specified behavior contained in the video data, causing the fourth external device to obtain the time range of the relevant video searched from the video data based on the search query by the first external device, and obtaining time information of the relevant video from the fourth external device, indicating the time range of the relevant video.

4. the data acquisition unit acquires frame images and time information of the frame images from the video data; The request processing unit outputting the frame images to a fifth external device, and acquiring caption data indicating captions for the frame images output from the fifth external device; outputting the caption data to a third external device, and acquiring behavior information of the photographer generated by the third external device; outputting the request information including the behavior information of the photographer to a third external device, and acquiring data representing a draft of the behavior record of the photographer generated by the third external device; The behavior record creation support system according to claim 1.

5. The request processing unit outputting the audio included in the video data to a second external device, and acquiring text data generated by the second external device; outputting the request information including the correspondence between the text data and the time of the voice to the third external device; 5. The behavior record creation support system according to claim 2.

6. the data acquisition unit acquires position data of a camera device carried by a photographer; The request processing unit outputting, to the third external device, a correspondence between time information of the action designated by the photographer and position data; 5. The behavior record creation support system according to claim 2.

7. The request processing unit outputting the position information of the camera device to a sixth external device, and acquiring address information indicating an address corresponding to the position information of the camera device output from the sixth external device; outputting the correspondence relationship between the time information and the address information to the third external device instead of the correspondence relationship between the time information and the position data; The behavior record creation support system according to claim 6.

8. a step in which the behavior record creation support device acquires moving image data captured by a camera device carried by a photographer; a step of the action record creation support device receiving a request to create an action record of the photographer from a user terminal device; a step in which the behavior record creation support device outputs, to an external system, photographer behavior information related to the photographer's behavior generated based on at least a part of the video image data, and request information requesting the external system to create a behavior record of the photographer's specified behavior; a step in which the action record creation support device acquires draft data of the action record of the photographer output from the external system; a step of the action record creation support device transmitting the draft data to the user terminal device; A behavioral record creation support method including:

9. The computer of the behavior record creation support device acquiring video data captured by a camera device carried by a photographer; receiving a request to create a behavior record of the photographer from a user terminal device; outputting, to an external system, photographer behavior information relating to the photographer's behavior generated based on at least a portion of the video data, and request information requesting the external system to create a behavior record of the photographer's designated behavior; acquiring draft data of the photographer's behavior record output from the external system; transmitting the draft data to the user terminal device; A program that executes.

Images