Letter creation system, letter creation method, and program

Abstract

To provide a letter creation system, a letter creation method, and a program for easily creating high-quality letters including video.SOLUTION: A letter creation system 1 comprises a control unit 2 and a storage unit 3. The control unit 2 detects video frames in which a target person is smiling using AI from video data 5 stored in the storage unit 3, and extracts the video data 5 before and after the video frames in which the target person is smiling, thereby creating a provided video 7, and uses AI to create provided text 8 related to document data 6 and the provided video 7 stored in the storage unit 3 from the document data 6 and the provided video 7.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a letter creation system, a letter creation method, and a program that use AI to automatically create documents such as letters from video data and document data. [Background technology]

[0002] Conventionally, there is known a technique for using AI to recognize facial expressions of a target person and detect smiles in images of video data, etc. (for example, Non-Patent Documents 1 and 2). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "How does image recognition technology work? Types, latest use cases, and changes in the use of AI (deep learning)", [online], System Integrator Co., Ltd. [Retrieved July 18, 2023], Internet<URL:https: / / products.sint.co.jp / aisia-ad / blog / image-recognition-ai> [Non-Patent Document 2] "AI Image Recognition Solutions," [online], NEC Corporation [searched July 18, 2023], Internet<URL:https: / / jpn.nec.com / bv / hoso / ai_recognition.html> Summary of the Invention [Problem to be solved by the invention]

[0004] However, in residential care facilities such as nursing homes, for example, families have been unable to visit residents due to the effects of the COVID-19 pandemic. Meanwhile, recreational activities are held every day in care facilities, and reports are sometimes sent to families, but because it takes time, there was an issue of not being able to send out letters frequently. Under these circumstances, families could not easily find out about the daily status of residents.

[0005] The present invention has been made in consideration of the above circumstances, and provides a letter creation system, a letter creation method, and a program that enable users to easily create high-quality letters that include video. [Means for solving the problem]

[0006] In order to solve such problems, the present invention provides a letter creation system equipped with a control unit and a memory unit, and a letter creation method for creating letters using the same, in which the control unit uses AI to detect video frames in which a target person is smiling from video data stored in the memory unit, and creates a provided video by extracting video data before and after the video frames in which the target person is smiling, and the control unit uses AI to create provided text with content related to the document data and the provided video from the document data stored in the memory unit and the provided video.

[0007] In the letter creation system and the letter creation method, when creating the provided text, the provided text can be further created by AI from audio data contained in the provided video.

[0008] In the letter creation system and letter creation method, when creating the provided text, AI derives multiple top first keywords that are highly relevant to the keywords contained in the document data in each category of action verb words, state verb words, emotional adjective words, and noun words, and from the derived first keywords, AI derives top second keywords that are highly relevant to the provided video, and AI can create the provided text based on the derived second keywords.

[0009] In the letter creation system and the letter creation method, when detecting a video frame in which the target person is smiling, one video frame per any number of seconds can be used, and the remaining video frames can be skipped.

[0010] In the letter creation system and letter creation method, when detecting video frames in which the target person is smiling, the image size of the video data is divided, and video frames in which the target person is smiling can be detected using AI from each of the divided video data.

[0011] The letter creation system and the letter creation method can create a document that includes the provided text, a still image extracted from the video data, and an identification code that is an encrypted URL where the provided video is saved.

[0012] The present invention also provides a program for causing a computer to execute the letter creation method. Effect of the Invention

[0013] According to the letter creation system, letter creation method, and program of the present invention, it is possible to easily create high-value-added letters that include video content. These letters can be created automatically by AI, which can greatly reduce the labor required by care facility staff. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a letter writing system according to a preferred embodiment of the present invention; [Diagram 2] FIG. 2 is a schematic diagram showing an example of a configuration of a video frame. [Diagram 3] 10 is a schematic diagram for explaining an example of detection of a video frame in which a target person is smiling; FIG. [Figure 4] 1 is a schematic diagram for explaining an example of a method for dividing image sizes of moving image data. FIG. [Diagram 5] 13 is a flowchart showing a flow of detecting a smile and creating a video to be provided. [Figure 6] 13 is a flowchart showing the flow of a masking process. [Figure 7] FIG. 2 is a schematic diagram for explaining a means for creating a provided text. [Figure 8] FIG. 13 is a schematic diagram showing an example of a provided text; [Figure 9] FIG. 2 is a schematic diagram for explaining a means for creating a provided text using a first keyword and a second keyword. [Figure 10] FIG. 11 is a schematic diagram showing an example of a provided text created using a first keyword and a second keyword. [Figure 11] FIG. 13 is a schematic diagram showing an example of a letter document. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following description, and various modifications and changes are possible for those skilled in the art based on the gist of the invention described in the claims or disclosed in the description for carrying out the invention. Such modifications and changes are also included in the scope of the present invention.

[0016] <Overall Overview>

[0017] According to the letter creation system 1, letter creation method, and program 4 of the present embodiment shown in Fig. 1, staff at a residential care facility such as a nursing home can create letters to inform the residents' families of their recent situations. Since the letters are created automatically by AI, the labor of the staff at the care facility can be significantly reduced.

[0018] Recreational activities are held daily in nursing facilities. In the letter creation system 1, letter creation method, and program 4 of this embodiment, AI is used to detect video frames in which residents are smiling from video data 5 that captures such recreational activities, and a provided video 7 with many smiling faces is created. Also, AI is used to create a provided text 8 that describes how the residents are enjoying themselves from the provided video 7 and text data 6 that describes an overview of the recreational activities. The family of the resident can easily find out about the resident's recent situation by watching the provided video 7 and provided text 8.

[0019] 1, the letter creation system 1 includes at least a control unit 2 and a storage unit 3. The control unit 2 is, for example, at least one arithmetic processing device such as a CPU or a GPU, and executes a program 4 stored in the storage unit 3. The storage unit 3 is, for example, at least one memory device such as a RAM or a ROM, and stores the program 4 and data used by the letter creation system 1.

[0020] The letter creation system 1 is, for example, a computer, which may be operated directly by a user, or may be a server that communicates via a network such as the Internet. A program 4 causes the computer serving as the letter creation system 1 to execute the letter creation method of this embodiment, and the control unit 2 creates a provided video 7 and a provided text 8 using AI based on video data 5 and text data 6. The control unit 2 also creates the provided text 8, a still image 31, and a document 33 containing an identification code 32. The resident information 9 is used to identify the target person when detecting a smile.

[0021] <Creating videos to be provided>

[0022] To create the provided video 7, first, the control unit 2 uses AI to detect video frames 11 (see FIG. 3) in which the target person is smiling from the video data 5 stored in the storage unit 3. The video data 5 is, for example, footage of a recreational activity.

[0023] In order to detect video frames 11 in which a target person is smiling using AI, the control unit 2 performs face detection and target person identification using AI face recognition, and recognizes the facial expression of the target person using AI smile detection. For AI face recognition, for example, open source FaceNet or Insightface can be used. For AI smile detection, for example, open source Residual Masking Network can be used.

[0024] Conventionally, AI-based face recognition and smile detection are processed for all video frames of the video data 5. In this embodiment, when detecting a video frame 11 in which the target person is smiling, one video frame is used per any number of seconds, and the remaining video frames are skipped. For example, it is preferable to set it so that the user, a staff member of a nursing facility, can select to use one video frame per second, one per two seconds, ..., or one video frame per N seconds. Note that the number of seconds may be a decimal number rather than an integer, and can be any real number. For example, as shown in FIG. 2, when the video data 5 is 30 fps and one video frame is used per second, 29 frames are skipped, and the video frame 11 in which the target person is smiling is detected every 30 frames. This makes it possible to perform AI processing of face recognition and smile detection at high speed and with low load.

[0025] For example, AI smile detection estimates seven facial expressions: Neutral, Happy, Sad, Surprise, Angry, Fear, and Disgust.

[0026] Any method can be used to detect the video frame 11 in which the target person is smiling using AI, and various methods are possible. As an example, as shown in FIG. 3, when a plurality of consecutive video frames, for example, five frames, of "Happy" video frames in which the degree of smile based on the estimated facial expression is, for example, 90% or more are detected, the first video frame among the five frames is set as the video frame 11 in which the target person is smiling, and video data, for example, 15 seconds before and 15 seconds after the video frame 11 is extracted and used as the provided video 7. In this way, the control unit 2 can create the provided video 7 by detecting the video frame 11 in which the target person is smiling from the video data 5 stored in the storage unit 3 using AI, and extracting the video data 5 before and after the video frame 11 in which the target person is smiling.

[0027] An example of a method for detecting a video frame 11 in which a target person is smiling using AI, which is different from that shown in FIG. 3, will be described. First, the estimated degree of facial expression is listed for video frames in which AI-based smile detection has been performed throughout the entire video data 5. Next, from the list created, one or more video frames with the highest degree of smile are set as video frames 11 in which the target person is smiling. Then, video data, for example, 15 seconds before and 15 seconds after these video frames 11 are extracted and set as the provided video 7. In this way, the control unit 2 can create the provided video 7 by detecting a video frame 11 in which a target person is smiling from the video data 5 stored in the storage unit 3 using AI, and extracting video data 5 before and after the video frame 11 in which the target person is smiling.

[0028] In AI processing, for example, an image size of 1920×1080 (2K) is usually compressed to an image size of 300×300 to perform face recognition and smile detection. In this embodiment, as shown in FIG. 4, when detecting a video frame 11 in which a target person smiles, the image size of the video data 5 is divided into, for example, m×n, and the video frame 11 in which the target person smiles is detected by AI from each of the divided video data 51. The image size of the video data 5 can be set to, for example, a maximum of 7×4, and the edge width and overlap width should also be considered. This allows for flexible response to changes due to environmental factors such as weather and illuminance by changing the setting value of m×n. In addition, the accuracy of face recognition and smile detection is improved, detection omissions are prevented, and privacy can be reliably considered when, for example, a masking process described later is required.

[0029] FIG. 5 is a flowchart showing a flow of detecting a smile and creating the provided video 7.

[0030] First, in step S1, the control unit 2 acquires the video data 5 from the storage unit 3. In step S2, the control unit 2 performs face detection by AI facial recognition. In face detection, a certain number of samples are taken from all video frames and processed. Next, the processing is performed for each detected face. In step S3, the control unit 2 performs AI facial recognition to identify the target person by comparing the detected face with the resident information 9 stored in the storage unit 3. In step S4, the control unit 2 performs AI smile detection to recognize the facial expression of the identified target person. In step S5, if the facial expression determination is OK, that is, if the control unit 2 detects a video frame 11 in which the target person is smiling by AI, the process proceeds to step S6. In step S6, the control unit 2 extracts a still image 31 from the video data 5. In step S7, the control unit 2 extracts video data 5 before and after the video frame 11 in which the target person is smiling, and creates a provided video 7.

[0031] In step S8, a masking process such as mosaic processing is performed on residents who do not want to appear in the video. In this process, as shown in FIG. 6, the control unit 2 performs face detection in step S81 by AI face recognition, and identifies the target person in step S82. The control unit 2 performs masking such as mosaic processing on the identified target person in step S83. In order to perform the masking process in step S8 on video frames skipped when detecting video frame 11 in which the target person is smiling, an end determination is made in step S84, and the processes from steps S81 to S83 are repeated until the processes for all video frames are completed. Note that masking processes such as mosaic processing may be performed not only on residents who do not want to appear in the video, but also on all people other than the target person whose smiling face is detected. In this case, the processes from steps S81 to S83 are repeated on all people to be masked until the processes for all video frames are completed. If masking is not necessary, step S8 can be omitted.

[0032] In step S9, the control unit 2 stores the still image 31 and the provided video 7 for each target person in the storage unit 3. Then, in step S10, an end determination is made, and the processes in steps S3 to S9 are repeated until the processing for all people with detected faces is completed. The stored provided video 7 is used to create the provided text 8. Note that, instead of storing the data of the still image 31 and the provided video 7 in the storage unit 3, information on the provided video 7 and information output by the AI ​​may be stored in the storage unit 3, and the still image 31 and the provided video 7 may be created from this information when creating the provided text 8.

[0033] <Creating the provided text>

[0034] 7, the provided text 8 is created by AI from the document data 6 and the provided video 7 stored in the storage unit 3, with content related to the document data 6 and the provided video 7. The control unit 2 creates the provided text 8, which describes the target person enjoying the recreation described in the document data 6, by AI from the keywords described in the document data 6 that describes the outline of the recreation, and the video of the provided video 7 created from the video data 5 that captures the recreation. Since the provided video 7 is a video of the target person smiling and enjoying the recreation described in the document data 6, the provided text 8 is a text that describes the target person enjoying the recreation described in the document data 6.

[0035] Furthermore, when the control unit 2 creates the provided text 8, it may further create the provided text 8 by AI from the audio data 12 included in the provided video 7. In this case, the control unit 2 creates the provided text 8 by AI from the keywords described in the document data 6, the video of the provided video 7, and the conversation in the audio data 12, describing the target person enjoying the recreation described in the document data 6.

[0036] It is preferable to train the AI ​​to create only positive sentences, omitting negative emotions such as sadness, pain, hatred, resentment, jealousy, anger, anxiety, etc. This will make it easier to describe the subject enjoying recreational activities.

[0037] An example of the provided text 8 created from the document data 6 and the provided video 7 is shown in Fig. 8. As shown in Fig. 8, the document data 6 includes information such as the content and date of the recreational activity, and the provided text 8 is created based on the described keywords. In addition to template text, the provided text 8 includes text that describes how the target person is enjoying the recreational activity described in the document data 6.

[0038] For example, OpenAI's GPT can be used to create text from document data 6 using AI. For example, OpenAI's Clip can be used to create text from provided video 7 using AI. For example, OpenAI's Whisper can be used to create text from audio data 12 using AI.

[0039] FIG. 9 shows the configuration of the means for creating the provided text 8 using the first keyword 25 and the second keyword 26. In this creation means, when the control unit 2 creates the provided text 8, it derives a plurality of first keywords 25 that are highly relevant to the keywords included in the document data 6 in each category of action verb words 21, state verb words 22, emotion adjective words 23, and noun words 24 by AI, derives a second keyword 26 that is highly relevant to the provided video 7 from the derived first keywords 25 by AI, and creates the provided text 8 by AI based on the derived second keyword 26. This creation means creates the provided text 8 with contents related to the document data 6 and the provided video 7 from the document data 6 and the provided video 7 stored in the storage unit 3 by AI. In the figure, ten first keywords 25 and three second keywords 26 are derived in each category, but the number of these is not limited.

[0040] FIG. 10 shows an example of the provided text 8 created by the creation means using the first keyword 25 and the second keyword 26.

[0041] First, the control unit 2 uses AI to derive 10 top primary keywords 25 highly related to the keyword "chorus" contained in the document data 6 in each category of action verb words 21, state verb words 22, emotional adjective words 23, and noun words 24. The derived primary keywords 25 are as shown in FIG.

[0042] Next, the control unit 2 inputs the first keywords 25 and the provided video 7, and derives, by AI, three second keywords 26 in each category that are highly relevant to the provided video 7 from the derived ten first keywords 25. The provided video 7 used here is a video of multiple people singing in a circle. The derived second keywords 26 are as shown in FIG. 10.

[0043] Finally, the control unit 2 uses AI to create the provided text 8 based on a total of 12 second keywords 26. The created provided text 8 is as shown in Fig. 10. The provided text 8 created in this way includes, in addition to the template text, a text that describes the target person enjoying the recreation described in the document data 6, and includes keywords included in the document data 6 and a text that is highly relevant to the video of the provided video 7.

[0044] In addition, verification was also conducted using other keywords related to recreation, such as "gymnastics," "origami," and "spending time in a room," as well as related provided videos 7, and it was confirmed that appropriate provided sentences 8 could be created. Furthermore, verification was also conducted when two or five second keywords 26 were derived for each category, and it was confirmed that appropriate provided sentences 8 could be created.

[0045] <Creating a document>

[0046] As shown in FIG. 11, the control unit 2 may create a document 33 including the provided text 8, a still image 31 extracted from the video data 5, and an identification code 32 that is an encrypted URL where the provided video 7 is saved. The still image 31 may be created, for example, from a video frame 11 in which the target person is smiling. The identification code 32 may be, for example, a QR code (registered trademark). When creating the document 33, the staff of the care facility may be allowed to edit the provided text 8. Also, the staff of the care facility may be allowed to select the still image 31 at their discretion.

[0047] The staff of the care facility provide the created document 33 to the resident's family in paper form or electronic data. The resident's family can view the provided video 7 by reading the identification code 32 with a smartphone or the like and accessing the URL where the provided video 7 is saved.

[0048] <Summary of the embodiment>

[0049] As described above in detail, the letter creation system 1 and letter creation method of this embodiment are a letter creation system 1 equipped with a control unit 2 and a memory unit 3, and a letter creation method that creates a letter using the same, in which the control unit 2 uses AI to detect video frames 11 in which a target person is smiling from the video data 5 stored in the memory unit 3, and creates a provided video 7 by extracting video data 5 before and after the video frame 11 in which the target person is smiling, and creates a provided text 8 with content related to the document data 6 and the provided video 7 from the document data 6 stored in the memory unit 3 and the provided video 7, using AI.

[0050] With this configuration, letters can be automatically created using AI, significantly reducing the workload of care facility staff.

[0051] In addition, when creating the provided text 8, the control unit 2 further creates the provided text 8 from the audio data 12 included in the provided video 7 using AI.

[0052] This structure makes it possible to create a provided document 8 that more faithfully describes the resident's recreational activities.

[0053] In addition, when creating the provided text 8, the control unit 2 uses AI to derive multiple top first keywords 25 that are highly relevant to the keywords included in the document data 6 in each category of action verb words 21, state verb words 22, emotional adjective words 23, and noun words 24, and from the derived first keywords 25, derives top second keywords 26 that are highly relevant to the provided video 7 using AI, and creates the provided text 8 using AI based on the derived second keywords 26.

[0054] With this configuration, it is possible to create provided text 8 that is highly relevant to the keywords contained in the document data 6 and to the video of the provided video 7.

[0055] Furthermore, when detecting video frames 11 in which the target person is smiling, the control unit 2 uses one video frame per any number of seconds and skips the remaining video frames.

[0056] This configuration enables high-speed, low-load AI processing for facial recognition and smile detection.

[0057] In addition, when detecting video frames 11 in which the target person is smiling, the control unit 2 divides the image size of the video data 5 and uses AI to detect video frames 11 in which the target person is smiling from each of the divided video data 5.

[0058] This configuration enables flexible response to changes in the environment, while improving the accuracy of face recognition and smile detection, preventing missed detections.

[0059] The control unit 2 also creates a document 33 which includes the provided text 8, a still image 31 extracted from the video data 5, and an identification code 32 which is an encrypted URL where the provided video 7 is saved.

[0060] This configuration allows us to provide newsletters with higher added value, by allowing users to easily watch videos of residents having fun.

[0061] Moreover, the program 4 of this embodiment causes a computer serving as the letter creation system 1 to execute the letter creation method described above. [Explanation of symbols]

[0062] 1. Letter creation system 2. Control section 3 Storage section 4. Program 5. Video data 6 Document Data 7 Provided Videos 8 Provided text 9. Tenant Information 11 Video frames in which the subject is smiling 12 Audio data 21 Action verb words (categories) 22 State verb words (categories) 23 Words (categories) of emotional adjectives 24 Noun words (categories) 25 First Keyword 26 Second Keyword 31 Still Images 32 Identification Code 33 Documents

Claims

1. A letter creation system including a control unit and a storage unit, The control unit detects a video frame in which a target person is smiling from the video data stored in the storage unit using AI, and creates a provided video by extracting video data before and after the video frame in which the target person is smiling; The control unit creates a provided text having content related to the document data and the provided video from the document data stored in the storage unit and the provided video by AI. Letter creation system.

2. The letter creation system according to claim 1 , further comprising, when creating the provided text, creating the provided text using AI from audio data contained in the provided video.

3. When creating the above-mentioned provision text, Derive, by AI, a plurality of top first keywords highly related to the keywords included in the document data in each category of action verb words, state verb words, emotion adjective words, and noun words; Derive a second keyword having a high relevance to the provided video from the derived first keywords by AI; The letter creation system according to claim 1 , wherein the provided text is created by AI based on the derived second keyword.

4. 2. The letter creation system according to claim 1, wherein when detecting a video frame in which the target person is smiling, one video frame per any number of seconds is used and the remaining video frames are skipped.

5. The letter creation system of claim 1, wherein when detecting video frames in which the target person is smiling, the image size of the video data is divided, and video frames in which the target person is smiling are detected from each of the divided video data using AI.

6. The provided text; A still image extracted from the video data; and An identification code obtained by encrypting the URL where the provided video is stored; 2. The letter creation system according to claim 1, which creates a document including the above.

7. A letter creation method for creating a letter using a letter creation system having a control unit and a storage unit, comprising: A letter creation system including a control unit and a storage unit, The control unit detects a video frame in which a target person is smiling from the video data stored in the storage unit using AI, and creates a provided video by extracting video data before and after the video frame in which the target person is smiling; The control unit creates a provided text having content related to the document data and the provided video from the document data stored in the storage unit and the provided video by AI. How to create a letter.

8. The letter creation method according to claim 7, further comprising the step of: creating the provided text using AI from audio data included in the provided video when the provided text is created.

9. When creating the above-mentioned provision text, Derive, by AI, a plurality of top first keywords highly related to the keywords included in the document data in each category of action verb words, state verb words, emotion adjective words, and noun words; Derive a second keyword having a high relevance to the provided video from the derived first keywords by AI; The letter creation method according to claim 7 , wherein the provided text is created by AI based on the derived second keyword.

10. 8. The method for creating a letter according to claim 7, further comprising the steps of: detecting a video frame in which the target person is smiling by using one video frame per any number of seconds and skipping the remaining video frames.

11. The letter creation method according to claim 7, wherein, when detecting video frames in which the target person is smiling, the image size of the video data is divided, and video frames in which the target person is smiling are detected from each of the divided video data using AI.

12. The provided text; A still image extracted from the video data; and An identification code obtained by encrypting the URL where the provided video is stored; 8. The method for creating a letter according to claim 7, further comprising the step of creating a document including the above-mentioned.

13. A program for causing a computer to execute the letter creation method according to any one of claims 7 to 12.

Images