Condition evaluation device, condition evaluation method, and condition evaluation program

Abstract

This device provides a state evaluation system that estimates the detailed movements of a subject within a video, enabling the input of evaluation values ​​for a wider range of evaluation items. [Solution] The state evaluation device 30 of the present invention includes an acquisition unit that acquires video data obtained by imaging a subject 50 with a camera, an explanatory text generation unit that generates an explanatory text for the video data acquired by the acquisition unit, and an evaluation unit that estimates evaluation values ​​for evaluation items related to the state of the subject 50 based on the explanatory text generated by the explanatory text generation unit.

Description

Technical Field

[0001] The present invention relates to a state evaluation device, a state evaluation method, and a state evaluation program.

Background Art

[0002] The Barthel Index (BI) and ICF (International Classification of Functioning, Disability and Health) staging are effective evaluation indicators for evaluating the state of care recipients (target persons). In BI and ICF staging, evaluation values (scores) are input for a plurality of evaluation items based on the actions of the target person.

[0003] In relation to this, Patent Document 1 below discloses a technique for analyzing video data obtained by a camera installed in the living room of a target person and calculating input values (evaluation values) for a plurality of evaluation items. The technique of Patent Document 1 calculates the evaluation value of a specific evaluation item by detecting the skeleton of the target person captured in the video and estimating the actions of the target person. According to such a configuration, since the evaluation value is mechanically input for a specific evaluation item, the labor of staff for inputting evaluation values for a plurality of evaluation items is reduced.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the above technique has a problem that since it detects the skeleton of the target person captured in the video and estimates the actions of the target person, it cannot estimate the detailed actions of the target person, and there is a limit to the evaluation items for which evaluation values can be input.

[0006] This invention has been made in view of the above-mentioned problems. Accordingly, the object of this invention is to provide a state evaluation device, a state evaluation method, and a state evaluation program that can estimate the detailed movements of a subject in a video and enable the input of evaluation values ​​for a larger number of evaluation items. [Means for solving the problem]

[0007] The above objectives of the present invention are achieved by the following means.

[0008] (1) A state evaluation device comprising: an acquisition unit that acquires video data obtained by imaging a subject with a camera; an explanatory text generation unit that generates an explanatory text for the video data acquired by the acquisition unit; and an evaluation unit that estimates evaluation values ​​for evaluation items related to the state of the subject based on the explanatory text generated by the explanatory text generation unit.

[0009] (2) The state evaluation device according to (1) above, wherein the evaluation unit includes an estimation unit that estimates the subject's ability value with respect to a predetermined operational item from the description, and a calculation unit that calculates the evaluation value from the ability value estimated by the estimation unit.

[0010] (3) The state evaluation device according to (2) above, further comprising a storage unit that stores a state conversion table relating the ability value and the evaluation value, wherein the calculation unit calculates the evaluation value from the ability value by referring to the state conversion table.

[0011] (4) The state evaluation device according to (2) or (3) above, wherein the estimation unit estimates the ability value from the description using a learning model that has learned the description and the ability value corresponding to the description as learning data.

[0012] (5) The condition evaluation device described in any one of (1) to (3) above, wherein the evaluation items include evaluation items relating to the subject's activities of daily living or cognitive function.

[0013] (6) The condition evaluation device according to any one of (1) to (3) above, wherein the evaluation items include evaluation items for the Barthel Index or ICF staging.

[0014] (7) A state evaluation device according to any one of (1) to (3) above, further comprising a caption generation unit that generates captions describing the actions of the subject in the video on a frame image basis, wherein the explanatory text generation unit generates the explanatory text based on the captions generated by the caption generation unit.

[0015] (8) The state evaluation device according to (7) above, wherein the explanatory text generation unit generates the explanatory text by inputting a prompt to the text generation model that instructs the generation of the explanatory text.

[0016] (9) The state evaluation device according to (7) above, wherein the caption generation unit generates the caption using a learning model that has learned an image and a caption corresponding to the image as training data.

[0017] (10) A condition evaluation method comprising the steps of (a) acquiring video data obtained by imaging a subject with a camera, (b) generating a descriptive text for the video data acquired in step (a), and (c) estimating evaluation values ​​for evaluation items relating to the subject's condition based on the descriptive text generated in step (b).

[0018] (11) A state evaluation program that causes a computer to perform the following steps: (a) a step of acquiring video data obtained by imaging a subject with a camera; (b) a step of generating a descriptive text for the video data acquired in step (a); and (c) a step of estimating evaluation values ​​for evaluation items related to the subject's state based on the descriptive text generated in step (b). [Effects of the Invention]

[0019] According to the present invention, it becomes possible to input evaluation values for more evaluation items.

Brief Description of Drawings

[0020] The advantages and features provided by one or more embodiments of the present invention will be more fully understood from the following detailed description and the accompanying drawings, which are for illustrative purposes only and are not intended to define the limitations of the present invention. [Figure 1] It is a diagram showing the overall configuration of the monitoring system. [Figure 2] It is a block diagram showing the schematic configuration of the detection device. [Figure 3] It is a block diagram showing the schematic configuration of the terminal device. [Figure 4] It is a block diagram showing the schematic configuration of the server device. [Figure 5] It is a diagram showing the stored content of the storage unit of the server device. [Figure 6] It is a diagram for explaining video data. [Figure 7] It is a diagram showing an example of the state transition table. [Figure 8] It is a flowchart showing the procedure of the state evaluation process. [Figure 9] It is a diagram showing an example of the ability value estimated from the video description text. [Figure 10] It is a diagram showing an example of the final ability value. [Figure 11] It is a flowchart showing the procedure of the caption generation process. [Figure 12] It is a diagram showing an example of the frame image after the caption generation process. [Figure 13] It is a flowchart showing the procedure of the description text generation process. [Figure 14] It is a diagram showing an example of the prompt for instructing the generation of the description text.

Modes for Carrying Out the Invention

[0021] Embodiments of the present invention will be described below with reference to the drawings. However, the scope of the present invention is not limited to the disclosed embodiments.

[0022] Figure 1 shows the overall configuration of a monitoring system 1 to which a condition evaluation device according to one embodiment of the present invention is applied.

[0023] As shown in Figure 1, the monitoring system 1 comprises a detection device 10, a terminal device 20, and a server device 30. The detection device 10, the terminal device 20, and the server device 30 are configured to communicate with each other via a network 40.

[0024] The detection device 10 is installed in the rooms of 50 target individuals within various facilities such as nursing homes and hospitals. The terminal device 20 is used, for example, by staff such as caregivers who care for the target individuals 50, or by facility administrators. The server device 30 is either an on-premise server installed on the facility's premises or a cloud server using a commercial cloud service. The network 40 consists of the internet or an intranet.

[0025] <Detection device 10> Figure 2 is a block diagram showing the schematic configuration of the detection device 10. The detection device 10 is installed as a sensor box on the ceiling or upper part of the wall of the room where the subject 50 lives.

[0026] As shown in Figure 2, the detection device 10 comprises a control unit 11, a communication unit 12, and a camera 13, which are interconnected by a bus.

[0027] The control unit 11 is composed of a CPU (Central Processing Unit) and memory such as RAM (Random Access Memory) and ROM (Read Only Memory), and controls each of the above parts and performs various calculation processes according to the program.

[0028] The communication unit 12 is an interface for communicating with other devices, and various wired or wireless communication interfaces are used.

[0029] Camera 13 captures images of the subject 50 from the ceiling or upper part of the wall of the living room and generates video data (image data) of the subject 50. Camera 13 is, for example, a near-infrared camera and captures a predetermined imaging area within the living room. Camera 13 captures the imaging area at a frame rate of, for example, 5 fps and generates video data.

[0030] The detection device 10 of this embodiment is configured to recognize predetermined actions of the subject 50 (such as getting up, getting out of bed, or falling) from video data obtained by imaging the subject 50 with the camera 13. When the detection device 10 recognizes a predetermined action of the subject 50, it outputs video data for one minute before and one minute after the occurrence of the action (a total of two minutes) to the server device 30 for recording. Note that the technology for recognizing predetermined actions (behaviors) of a person captured by the camera, and the technology for recording video data before and after an event occur, are known technologies, so a detailed explanation is omitted.

[0031] <Terminal device 20> Figure 3 is a block diagram showing the schematic configuration of the terminal device 20. The terminal device 20 is, for example, a PC (Personal Computer).

[0032] As shown in Figure 3, the terminal device 20 comprises a control unit 21, a storage unit 22, a communication unit 23, a display unit 24, and an input unit 25, which are interconnected by a bus. Note that, to avoid repetition in the explanation, the parts of the terminal device 20 that have the same functions as those of the detection device 10 will not be described.

[0033] The storage unit 22 consists of an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs and data.

[0034] The display unit 24 is, for example, a liquid crystal display, which displays various information.

[0035] The input unit 25 is equipped with a keyboard, numeric keypad, mouse, etc., and accepts input of various instructions and information.

[0036] In this embodiment, the terminal device 20 is used, for example, by a user of the terminal device 20 (facility staff or administrator) to view video data obtained by imaging the subject 50 with the camera 13 of the detection device 10. The terminal device 20 is also used, for example, by a user of the terminal device 20 to view and edit evaluation values ​​of evaluation items (for example, BI evaluation items) related to the state of the subject 50 estimated from the video data.

[0037] <Server device 30> Figure 4 is a block diagram showing the schematic configuration of the server device 30. The server device 30 corresponds to the state evaluation device of the present invention.

[0038] As shown in Figure 4, the server device 30 comprises a control unit 31, a storage unit 32, and a communication unit 33, which are interconnected by a bus. Note that the above-mentioned parts of the server device 30 have the same functions as the above-mentioned parts of the detection device 10 and the terminal device 20, so their descriptions are omitted.

[0039] Figure 5 shows the contents of the storage unit 32 of the server device 30. As shown in Figure 5, the storage unit 32 of the server device 30 stores video data 110 and a state conversion table 120. The video data 110 includes multiple 2-minute video data obtained by imaging the subject 50 with the camera 13 of the detection device 10. Each video data is stored in association with text information of a descriptive text that explains the content of the video data. The state conversion table 120 is a table that associates the ability values ​​of the subject 50 with predetermined operation items and the evaluation values ​​of evaluation items related to the state of the subject 50. Details of the video data 110 and the state conversion table 120 will be described later.

[0040] Furthermore, the storage unit 32 of the server device 30 stores programs corresponding to the acquisition unit 131, the caption generation unit 132, the descriptive text generation unit 133, the estimation unit 134, and the calculation unit 135. The acquisition unit 131 acquires video data obtained by imaging the subject 50 with the camera 13. The caption generation unit 132 generates captions for each frame image of the video data acquired by the acquisition unit 131, describing the actions of the subject 50 in the video. The caption generation unit 132 includes a learning model that has learned images and the corresponding captions as training data, and generates captions using this learning model. The descriptive text generation unit 133 generates a descriptive text (summary) of the video based on the captions generated by the caption generation unit 132. The descriptive text generation unit 133 includes a text generation model that can generate a descriptive text from the caption information, and generates a descriptive text using this text generation model. The text generation model is a form of so-called generative AI (Artificial Intelligence), and is generated, for example, by fine-tuning a general-purpose text generation model. An example of a general-purpose text generation model is Copilot, provided by Microsoft. The estimation unit 134 estimates the ability scores of the subject 50 regarding predetermined action items from the explanatory text generated by the explanatory text generation unit 133. The estimation unit 134 includes a learning model that has been trained using explanatory text and the ability scores corresponding to said explanatory text as training data, and estimates the ability scores using this learning model. The calculation unit 135 calculates evaluation values ​​for evaluation items related to the state of the subject 50 from the ability scores of the subject 50 estimated by the estimation unit 134. The functions of each of the above units are performed by the control unit 31 executing the corresponding programs.

[0041] Furthermore, the detection device 10, terminal device 20, and server device 30 may include components other than those described above, and may not include some of the components described above. For example, the detection device 10 may include other sensors such as a motion sensor or a microphone.

[0042] Next, with reference to Figures 6 and 7, the video data 110 and the state conversion table 120 stored in the storage unit 32 of the server device 30 will be described in detail.

[0043] Figure 6 is a diagram illustrating the video data 110. As described above, the video data 110 includes multiple 2-minute video data obtained by imaging the subject 50 with the camera 13 of the detection device 10. As shown in Figure 6, each video 111 of the video data is accompanied by an explanatory text 112 that describes the content of the video 111, and each video data is stored in the storage unit 32 in association with the text information of the explanatory text 112. The video explanatory text 112 is generated by the generation AI described above. Details of the process for generating the video explanatory text 112 will be described later.

[0044] Figure 7 shows an example of a state conversion table 120. As described above, the state conversion table 120 is a table that associates the ability values ​​of the subject 50 with respect to predetermined operation items and the evaluation values ​​of evaluation items related to the state of the subject 50. As shown in Figure 7, the state conversion table 120 includes state rank information 121, ability value information 122, and evaluation value information 123.

[0045] The status rank information 121 is information indicating the status rank of the subject 50. In this embodiment, the status ranks are classified into nine ranks, from "Status 1" to "Status 9". "Status 1" indicates that the subject 50 is in the worst state, and "Status 9" indicates that the subject 50 is in the best state. A higher status rank indicates a better state for the subject 50.

[0046] Ability score information 122 is information that associates the status rank of subject 50 with combinations of subject 50's ability scores for multiple movement items. Multiple movement items include, for example, the movement item "Mobility" and the movement item "Transfer". The movement item "Mobility" includes, for example, five ability scores: "Walking Independently", "Walking with Support", "Walker", "Wheelchair", and "Assistance", and the movement item "Transfer" includes three ability scores: "Independent", "Supervision", and "Assistance". In Figure 7, for example, if subject 50's ability score for the movement item "Mobility" is "Wheelchair" and the ability score for the movement item "Transfer" is "Assistance", then subject 50's status rank will be "Status 4".

[0047] The evaluation value information 123 is information that associates the status rank of subject 50 with the evaluation values ​​of evaluation items related to subject 50's status, and includes care level information 1231 and BI information 1232. The care level information 1231 is information that associates the status rank of subject 50 with the care level. The BI information 1232 is information that associates the status rank of subject 50 with combinations of evaluation values ​​for 10 evaluation items of BI. For example, if subject 50's status rank is "Status 3", the evaluation value for the evaluation item "Care Level" will be "Care Level 4". Similarly, for example, if subject 50's status rank is "Status 3", the evaluation value for the BI evaluation item "Toilet Use" will be "Partial Assistance", and the evaluation value for the BI evaluation item "Stair Climbing" will be "Full Assistance".

[0048] The state conversion table 120 is created by referencing ability value data and evaluation value data for a large number of subjects (care recipients). More specifically, for a large number of subjects, ability value data for each subject is obtained from the video data of each subject by a process similar to the process of the present invention described later. On the other hand, for the same subjects, evaluation value data entered by staff, such as data from LIFE (Scientific Care Information System) provided by the facility or data on the level of care required, is obtained as evaluation value data for each subject. Then, taking into account the contents of the ability value data and evaluation value data obtained for a large number of subjects, the state conversion table 120 is manually created that associates the state rank with the ability value of each action item and the evaluation value of each evaluation item, and is stored in the storage unit 32.

[0049] Furthermore, the number of status ranks in the status conversion table 120 is not limited to 9, but is changed as appropriate according to the ability value information 122 and evaluation value information 123. For example, if the ability value information 122 consists of 3 action items, and each action item contains 3 ability values, the maximum number of status ranks will be 27.

[0050] In the monitoring system 1 of this embodiment, configured as described above, evaluation values ​​for evaluation items related to the state of the subject 50 are estimated from video data obtained by imaging the subject 50 with the camera 13. Specifically, the ability values ​​of the subject 50 for predetermined action items are estimated from the video description text 112 (see Figure 6) of the video data, and evaluation values ​​for evaluation items related to the state of the subject 50 are calculated from the ability values ​​of the subject 50.

[0051] The operation of the server device 30, which estimates evaluation values ​​for evaluation items related to the condition of subject 50 from the video descriptions, will be explained below with reference to Figures 8 to 10. In the following explanation, an example will be given in which evaluation values ​​for BI evaluation items are estimated from the descriptions of multiple videos corresponding to multiple video data acquired during a predetermined evaluation period. BI is one of the evaluation indices used to assess the activities of daily living (ADL) of subject 50.

[0052] Figure 8 is a flowchart showing the procedure for the state evaluation process. The process shown in the flowchart in Figure 8 is executed by the control unit 31 according to the program stored in the storage unit 32 of the server device 30.

[0053] (Step S101) First, the server device 30 reads the video description. More specifically, the server device 30 reads the text information of the description stored in the storage unit 32 in association with a single video data.

[0054] (Step S102) Next, the server device 30 estimates the subject's ability scores from the video description. More specifically, the server device 30 analyzes the text information of the description read in step S101 to estimate the subject's ability scores for predetermined operational items. In this embodiment, the server device 30 inputs the description into a learning model that has learned the relationship between the video description and the ability scores corresponding to that description. Upon receiving the description, the learning model recognizes characteristic terms in the description and outputs the corresponding operational items and ability scores.

[0055] Figure 9 shows an example of ability scores estimated from video descriptions. In Figure 9, ability scores are estimated for 11 action items, such as "walking," "standing," "sitting," "lying down," and "mobility," from six videos. As shown in Figure 9, for example, from the description of the video dated June 1, 2024, "After eating alone, the person walked to the entrance using a cane," an ability score of "cane" is estimated for the action item "walking." Also, from the same description, "After eating alone, the person walked to the entrance using a cane," an ability score of "independent" is estimated for the action item "eating."

[0056] It should be noted that the action items and ability scores are not limited to the examples shown in Figure 9, and various action items and ability scores can be used. For example, "total assistance" and "partial assistance" can be used as ability scores. Furthermore, the method for estimating ability scores from the video description is not limited to methods using learning models; for example, a predetermined term may be recognized by text mining, and an ability score may be estimated from that term by referring to a predetermined correspondence table. For example, if the term "walking" is recognized in the video description, but terms such as "cane," "walker," and "staff" are not recognized in the description, the ability score "independent" will be estimated for the action item "walking."

[0057] (Step S103) Next, the server device 30 determines whether or not there is other video data. More specifically, the server device 30 determines whether or not other video data acquired during a predetermined evaluation period (for example, June 1st to June 6th, 2024) is stored in the storage unit 32.

[0058] If it is determined that there is other video data (step S103: YES), the server device 30 returns to the process in step S101. As a result, the processes in steps S101 to S103 are repeated until it is determined that there is no other video data. On the other hand, if it is determined that there is no other video data (step S103: NO), the server device 30 moves on to the process in step S104.

[0059] (Step S104) If it is determined that there is no other video data (step S103: NO), the server device 30 determines the ability score of the subject 50. More specifically, the server device 30 determines the final ability score of the subject 50 based on the ability scores of the subject 50 estimated from the descriptions of multiple videos by repeating the processes in steps S101 to S103.

[0060] Figure 10 shows an example of the final ability scores for 50 subjects. The ability scores shown in Figure 10 are determined based on the ability scores shown in Figure 9. As shown in Figure 10, for example, for the action item "walking," the ability score "cane," estimated in Figure 9, is determined as the final ability score. Also, for the action items "dressing" and "grooming," for which two ability scores, "assistance" and "independence," were estimated in Figure 9, "assistance" is determined as the final ability score.

[0061] Furthermore, the final ability scores are not limited to the examples shown in Figure 10. For example, for the action items "dressing" and "grooming," ability scores such as "full assistance" and "partial assistance" may be used.

[0062] (Step S105) Next, the server device 30 identifies the status rank of the subject 50. More specifically, the server device 30 refers to the status conversion table 120 (see Figure 7) stored in the memory unit 32 and identifies the status rank of the subject 50 from the ability values ​​determined in step S104. For example, as shown in Figure 10, if the ability value for the movement item "mobility" is "wheelchair" and the ability value for the movement item "transfer" is "independent", then the status rank "Status 6" for the subject 50 is identified from the status conversion table 120 shown in Figure 7.

[0063] (Step S106) Next, the server device 30 calculates the evaluation values ​​for the subject 50. More specifically, the server device 30 refers to the state conversion table 120 and calculates the evaluation values ​​for the evaluation items related to the subject 50's state based on the state rank identified in step S105. For example, if the subject 50's state rank is "State 6", the server device 30 refers to the state conversion table 120 and calculates the subject 50's care level as "Care Level 3". The server device 30 also refers to the state conversion table 120 and calculates, for example, the evaluation value "Independent" for the BI evaluation item "Eating", and the evaluation value "Total Assistance" for the BI evaluation item "Climbing Stairs". The evaluation values ​​for the BI evaluation items are converted into scores (points) as needed. For example, the evaluation value "Independent" for the evaluation item "Eating" is converted into a score of "10 points".

[0064] (Step S107) The server device 30 then stores the evaluation values ​​of the subjects 50 and terminates the process. More specifically, the server device 30 stores the evaluation values ​​(scores) of each BI evaluation item calculated in step S106 into the storage unit 32 and terminates the process.

[0065] Furthermore, the evaluation values ​​stored in the memory unit 32 can be viewed and edited by facility staff, for example, via the terminal device 20. The staff can check the evaluation values ​​for each evaluation item for the subject 50 and correct the evaluation values ​​as necessary.

[0066] As described above, according to the flowchart shown in Figure 8, the ability scores of the subject 50 for a given action item are estimated from the video description. Then, the status rank of the subject 50 is identified from the ability scores of the subject 50, and the evaluation value of the BI evaluation item corresponding to that status rank is calculated.

[0067] This configuration allows for the automated input of evaluation values ​​for multiple evaluation items, reducing the workload for staff members. Furthermore, it eliminates inconsistencies in evaluations among staff, enabling objective and consistent assessments. Additionally, the detailed actions of the 50 subjects within the video are estimated from the video description, allowing for the input of evaluation values ​​for a greater number of evaluation items.

[0068] As described above, the server device 30 estimates the ability scores of the subject 50 from the video descriptions using a learning model that has been trained with video descriptions and corresponding ability scores as training data. The training data is prepared, for example, by having an operator manually input ability scores (correct data) for action items while viewing the descriptions. The technology itself, which generates a learning model with predetermined functions by providing training data to a learning model such as a neural network and training it, is a well-known machine learning technique, so a detailed explanation will be omitted.

[0069] The operation of the server device 30 that generates video descriptions will be explained below with reference to Figures 11 to 14. The server device 30 generates captions for each frame of the video data that describe the actions of the subject 50 in the video. Then, the server device 30 generates a video description (summary) based on the captions generated for the video.

[0070] First, with reference to Figures 11 and 12, the operation of the server device 30 that generates captions for the video will be described.

[0071] Figure 11 is a flowchart showing the procedure for the caption generation process performed by the server device 30. The process shown in the flowchart in Figure 11 is executed by the control unit 31 according to the program stored in the storage unit 32 of the server device 30.

[0072] (Step S201) First, the server device 30 acquires video data of the subject 50. More specifically, the server device 30 acquires 2 minutes (600 frames) of video data obtained by imaging the subject 50 with the camera 13 of the detection device 10.

[0073] (Step S202) Next, the server device 30 generates captions. More specifically, the server device 30 generates captions for each frame image of the video data acquired in step S201, describing the actions of the subject 50 in the video. In this embodiment, the server device 30 inputs the video data into a learning model that has learned the relationship between images and the captions corresponding to those images. The learning model, upon receiving the video data, recognizes the actions (features) of the subject 50 in each frame image and generates a caption describing those actions for each frame image.

[0074] (Step S203) The server device 30 then stores the caption and terminates the process. More specifically, the server device 30 stores the text information of the caption generated in step S202 and the frame number information of the frame image to which the caption is applied in the storage unit 32, associating them with the video data, and then terminates the process.

[0075] As described above, according to the flowchart shown in Figure 11, captions explaining the actions of the subject 50 in the video are generated on a frame-by-frame basis for the video data obtained by capturing the subject 50 with the camera 13 of the detection device 10. Specifically, for example, various captions explaining each action of the subject 50 in various scenes of the video are generated.

[0076] Figure 12 shows an example of a frame image 200 after the caption generation process. As described above, in the caption generation process, a caption describing the actions of subject 50 is generated for each frame image of the video data of subject 50.

[0077] As shown in Figure 12, the frame image 200 after the caption generation process displays a caption 210 overlaid on it that describes the actions of the subject 50. Specifically, for example, the caption 210 "The resident is sitting on the edge of the bed" is displayed overlaid on the bottom of the frame image 200 as a description of the actions of the subject 50 in the frame image 200 shown in Figure 12. Note that the actions of the subject 50 include not only the subject 50's own actions but also the passive actions of the subject 50 that are assisted by the facility staff.

[0078] As described above, the server device 30 generates captions for each frame image of a video using a learning model that has been trained with images and their corresponding captions as training data. For a specific frame image of a video, the learning model generates a caption for that frame image based on the caption corresponding to an image in the training data that is similar to that frame image. The training data is prepared, for example, by an operator manually adding captions (ground truth data) to frame images while viewing the video.

[0079] Furthermore, the learning model in this embodiment is configured to generate new captions by combining multiple captions within the training data. Specifically, for example, suppose the learning model has already learned two captions, "The resident is walking" and "The resident is brushing their teeth," and the action of the subject 50 in a specific frame image of the video is walking while brushing their teeth. In this case, if the learning model recognizes the two actions (features) "walking" and "brushing teeth" from the frame image, it will use a predetermined language model to combine the two captions and generate a new caption, "The resident is walking while brushing their teeth." With this configuration, when training the learning model, it is not necessary to provide the learning model with captions that explain multiple actions performed simultaneously by a person in the video as training data, thus reducing the effort required of the person preparing the training data.

[0080] Next, referring to Figures 13 and 14, the operation of the server device 30 that generates the video description will be explained.

[0081] Figure 13 is a flowchart showing the procedure for the explanatory text generation process performed by the server device 30. The process shown in the flowchart in Figure 13 is executed by the control unit 31 according to the program stored in the storage unit 32 of the server device 30.

[0082] (Step S301) First, the server device 30 inputs a prompt to the text generation model. More specifically, the server device 30 inputs a prompt 300 (see Figure 14) to the text generation model (generation AI) instructing it to generate an explanatory text.

[0083] Figure 14 shows an example of prompt 300. As shown in Figure 14, prompt 300 instructs the text generation model to generate a video description (summary) based on multiple captions. In Figure 14, "frequency information" refers to the number of frame images to which a particular caption is applied.

[0084] (Step S302) Next, the server device 30 generates a description. More specifically, the server device 30 generates a video description based on the captions generated in the caption generation process shown in Figure 11. In this embodiment, the text generation model, which receives a prompt in step S301, generates a video description based on the information of multiple captions.

[0085] (Step S303) Then, the server device 30 stores the explanatory text and terminates the process. More specifically, the server device 30 stores the text information of the explanatory text generated in step S302 in the storage unit 32, associating it with the video data, and then terminates the process.

[0086] As described above, according to the flowchart shown in Figure 13, a video description is generated based on multiple captions generated for the video.

[0087] Furthermore, according to the monitoring system 1 of this embodiment, evaluation values ​​for evaluation items related to the state of the subject 50 are estimated from the video description. With this configuration, detailed movements of the subject 50 in the video can be estimated, and it becomes possible to input evaluation values ​​for a larger number of evaluation items.

[0088] The present invention is not limited to the embodiments described above, and can be modified in various ways within the scope of the claims.

[0089] For example, in the embodiment described above, the state conversion table 120 was assigned strings such as "assistance" as evaluation values ​​for evaluation items related to the state of the subject 50. However, unlike the embodiment described above, the state conversion table 120 may be directly assigned BI scores as evaluation values ​​for evaluation items related to the state of the subject 50.

[0090] Furthermore, the above-described embodiment explained the case of calculating evaluation values ​​for BI evaluation items as an example. However, the evaluation items for which evaluation values ​​are calculated are not limited to BI evaluation items; evaluation values ​​may also be calculated for ICF staging evaluation items. When calculating evaluation values ​​for ICF staging evaluation items, the evaluation value information 123 in the state conversion table 120 is changed. In addition, the action items and ability values ​​are also changed as appropriate as necessary. Note that ICF staging is one of the evaluation indices for evaluating the activities of daily living and cognitive functions of the subject 50.

[0091] Furthermore, in the embodiments described above, a model generated by fine-tuning a general-purpose text generation model was used as the text generation model that generates explanatory text from caption information. However, a model generated from scratch by training it with a large amount of data may also be used as the text generation model that generates explanatory text from caption information.

[0092] Furthermore, in the embodiment described above, an example was given in which evaluation values ​​for evaluation items related to the condition of subject 50 were estimated from the explanatory text of a 2-minute video. However, the video used to estimate evaluation values ​​is not limited to a 2-minute video, and videos of various lengths can be used.

[0093] The processing units in the flowcharts of the embodiments described above are divided according to the main processing content in order to facilitate understanding of each process. The present invention is not limited by how the processing steps are classified. Each process can be further divided into more processing steps. Also, one processing step may perform even more processes.

[0094] In the embodiments described above, the functions of each device may be implemented by other devices. For example, the function implemented by the detection device 10 may be implemented by the server device 30.

[0095] The means and methods for performing various processing in the state evaluation device according to the above embodiment can be implemented by either a dedicated hardware circuit or a programmed computer. The program may be provided, for example, on a computer-readable recording medium such as a USB (Universal Serial Bus) memory or a DVD (Digital Versatile Disc)-ROM, or it may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a storage unit such as an HDD. Furthermore, the program may be provided as a standalone application software, or it may be incorporated into the software of the state evaluation device as a function of the device.

[0096] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are for illustrative purposes only and are not limiting. The scope of the present invention should be interpreted in accordance with the language of the appended claims. [Explanation of Symbols]

[0097] 1. Monitoring system, 10 detection devices, 11,21,31 Control Unit, 12, 23, 33 Communications Department, 13 cameras, 20 terminal devices, 22,32 memory section, 24 Display section, 25 Input section, 30 server devices, 40 networks, 50 target individuals.

Claims

1. An acquisition unit that acquires video data obtained by imaging a subject with a camera, A description generation unit generates a description for the video data acquired by the acquisition unit, An evaluation unit estimates evaluation values ​​for evaluation items related to the subject's condition based on the explanatory text generated by the explanatory text generation unit, A condition evaluation device having the following features.

2. The evaluation unit described above, An estimation unit that estimates the subject's ability value for predetermined operational items based on the above description, The state evaluation apparatus according to claim 1, further comprising a calculation unit that calculates the evaluation value from the capability value estimated by the estimation unit.

3. The system further includes a storage unit that stores a state conversion table relating the aforementioned ability values ​​and the aforementioned evaluation values, The state evaluation device according to claim 2, wherein the calculation unit calculates the evaluation value from the capability value by referring to the state conversion table.

4. The state evaluation device according to claim 2 or 3, wherein the estimation unit estimates the ability value from the description using a learning model that has learned the description and the ability value corresponding to the description as learning data.

5. The condition evaluation device according to any one of claims 1 to 3, wherein the evaluation items include evaluation items relating to the subject's activities of daily living or cognitive function.

6. The condition evaluation apparatus according to any one of claims 1 to 3, wherein the evaluation items include evaluation items for the Barthel Index or ICF staging.

7. The system further includes a caption generation unit that generates captions describing the actions of the subject in the aforementioned video on a frame-by-frame basis. The state evaluation apparatus according to any one of claims 1 to 3, wherein the explanatory text generation unit generates the explanatory text based on the caption generated by the caption generation unit.

8. The state evaluation device according to claim 7, wherein the explanatory text generation unit generates the explanatory text by inputting a prompt instructing the generation of the explanatory text to a text generation model.

9. The state evaluation device according to claim 7, wherein the caption generation unit generates the caption using a learning model that has learned an image and a caption corresponding to the image as training data.

10. (a) A step of acquiring video data obtained by imaging the subject with a camera, Step (a) above involves generating a video description for the video data obtained in step (a), Step (c) involves estimating evaluation values ​​for evaluation items related to the subject's condition based on the explanatory text generated in step (b), A method for evaluating the state of having a certain condition.

11. Procedure (a) for acquiring video data obtained by imaging a subject with a camera, The procedure (b) involves generating a description of the video for the video data obtained in the above procedure (a), A procedure (c) for estimating evaluation values ​​for evaluation items related to the subject's condition based on the explanatory text generated in the procedure (b) above, A state evaluation program that causes a computer to perform a certain action.

Images