Data collection system and program

Abstract

To provide a data collection system with which it is possible for information to be shared among people engaged in care business with regards to the state of a care receiver which is unknown to a care giver and it is possible to efficiently know the cause or meaning of it.SOLUTION: In a data collection system 10, the arrangement of personnel, etc., including a care giver, a care receiver, a care adjuster, and a care evaluator is registered to related person arrangement storage means in advance by care setting means 22, and care event information (moving-image and an annotation thereto) dispatched by the care giver or the care receiver is received by care event information collection means 21. Meanwhile, adjustment work on care event information by the care adjuster is assisted and substituted for and adjustment information (moving-image for evaluation request and an annotation thereto) is created by adjustment means 23, and the adjustment information is sent to the care evaluator and an annotation to the moving-image, etc. by the care evaluator is received by evaluation result collection means 26.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a data collection system and program configured by a computer that collects data related to the implementation of medical care, nursing, caregiving, assistance, or health care or similar care, and can be used, for example, when sharing information, mainly video, between multiple caregivers. [Background technology]

[0002] Various systems for collecting and sharing information related to care have been proposed. For example, a system using technology related to a chat function to support rapid decision-making in team medical care is known (see Patent Document 1). A typical team medical care system for making medical decision-making allows for many-to-many message exchange (chat) within the team, but for diseases where every minute counts, using chat is not always easy due to the hassle of typing characters. Therefore, the system described in Patent Document 1 uses the history of previously entered messages, the roles of members, and progress information to identify and display candidates for messages to be newly entered by specific members.

[0003] Also, a support system used when a supporter (such as a medical professional or caregiver) provides a service to a target person (such as a patient or a care recipient) is known (see Patent Document 2). Conventional systems have a problem in that information related to the service provided by the supporter (for example, information identifying the person who answered a call in a nurse call system) is not recorded. Therefore, the support system described in Patent Document 2 provides the supporter with a terminal equipped with functions such as display, communication, and photography, and inputs image information of the target person's liquid or other containing pack (such as an IV pack) via the terminal. The image information is compared with the previous image taken using machine learning, recorded, and linked to an electronic medical record.

[0004] Furthermore, a group management system aimed at reducing the workload of nurses is known (see Patent Document 3). In conventional systems, when a nurse wanted to refer to the medical information of a patient under their care, the nurse would designate the patient and read out the medical information. However, because the patients under a nurse's care change daily depending on the patient's condition and admission / discharge schedule, the workload of designating the patient was significant. Therefore, the group management system described in Patent Document 3 uses log information showing the access history of electronic medical record information by nurses belonging to a team to identify patients whose electronic medical record information has been accessed by the nurses belonging to the team, and adds the identified patients to the patient group for which the team is responsible for nursing.

[0005] Furthermore, in this application, video data is cut out. As a technology related to video cutout, for example, a monitoring device that deters violent acts in facilities is known (see Patent Document 4). The monitoring device described in Patent Document 4 detects the occurrence of a predetermined event (phenomenon) related to a subject. The predetermined event is an event that requires a response, such as a predetermined behavior of the subject or a nurse call (care call) by the subject (see paragraph

[0034] of Patent Document 4). Then, when the event determination unit determines that a subject event has occurred, the device transmits video for a predetermined period of time (e.g., several seconds to several minutes) including the time of the event to a server, or, when video is continuously transmitted to the server, cuts out video for a predetermined period of time including the time of the event (see paragraphs

[0055] ,

[0067] , and

[0115] of Patent Document 4).

[0006] Furthermore, a pediatric information sharing system is known that shares time-series record data including images related to a child's condition with medical professionals, and can accurately communicate the child's condition and state to medical professionals, thereby providing peace of mind to parents (see Patent Document 5). In this pediatric information sharing system, record data including images related to the child's condition is stored in a management server in association with dates on a calendar displayed on the screen of a user terminal, a list of record data corresponding to a date selected on the calendar is obtained from the management server and displayed on the screen, and record data selected from the list is obtained from the management server and displayed on the screen.

[0007] There is also known an electronic medical record system that allows for more appropriate care of hospitalized animals (see Patent Document 6). In this electronic medical record system, information regarding the execution time of tasks related to each management item during hospitalization is input in order to display a schedule of tasks related to the management item.

[0008] In this application, record fragments are created by performing pattern recognition processing on video data. Known related technologies include a technology that detects basic movement units of facial muscles (groups of facial muscles) called action units (AUs) and performs pattern recognition processing based on the combination of the presence or absence of these movements (see Non-Patent Document 1), a technology that detects movement in the gaze direction (see Non-Patent Document 2), a technology that detects movement in the face direction (see Non-Patent Document 3), and a technology that detects movement in the positions of skeletal and joint points (see Non-Patent Document 4).

[0009] Known implementation technologies include implementation technology related to action units (AUs) (see Non-Patent Document 5), implementation technology related to gaze direction (see Non-Patent Document 6), implementation technology related to face direction (see Non-Patent Document 7), implementation technology related to the positions of bones and joint points (see Non-Patent Documents 8 and 9), and implementation technology related to speech (see Non-Patent Document 10). [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2023-157873 [Patent Document 2] Patent Publication No. 2021-56988 [Patent Document 3] Japanese Patent Application Publication No. 2023-108448 [Patent Document 4] JP 2020-13185 A (Claim 10, paragraphs

[0034] ,

[0055] ,

[0067] , and

[0115] ) [Patent Document 5] Patent No. 7333058 Publication [Patent Document 6] Japanese Patent Application Publication No. 2022-60229 [Non-patent literature]

[0011] [Non-Patent Document 1] Zhiwen Shao, Zhilei Liu, Jianfei Cai, and Lizhuang Ma, “JAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention,” arXiv:2003.08834v3 [cs.CV] 24 Sep 2020. [Non-patent document 2] Ahmed A. Abdelrahman, Thorsten Hempel, Aly Khalifa, and Ayoub Al-Hamadi, “L2CS-NET: FINE-GRAINED GAZE ESTIMATION IN UNCONSTRAINED ENVIRONMENTS,” arXiv:2203.03339v1 [cs.CV] 7 Mar 2022. [Non-patent document 3] Thorsten Hempel, Ahmed A. Abdelrahman, and Ayoub Al-Hamadi, “6D ROTATION REPRESENTATION FOR UNCONSTRAINED HEAD POSE ESTIMATION,” arXiv:2202.12555v2 [cs.CV] 7 Nov 2022. [Non-patent document 4] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,” arXiv:1812.08008v2 [cs.CV] 30 May 2019. [Non-Patent Document 5] Zhiwen Shao, “PyTorch-JAANet,” [online], [Retrieved November 29, 2023], Internet<URL:https: / / github.com / ZhiwenShao / PyTorch-JAANet> [Non-patent document 6] Ahmed Abdelrahman, “L2CS-Net (Gaze Estimation),” [online], [Retrieved November 29, 2023], Internet<URL:https: / / github.com / Ahmednull / L2CS-Net> [Non-Patent Document 7] Thorsten Hempel, “6DRepNet (6-dimensional representation),” [online], [Retrieved November 29, 2023], Internet<URL:https: / / github.com / thohemp / 6DRepNet> [Non-patent document 8] OpenMMlab library, “mmpose (pose estimation),” [online], [Retrieved November 29, 2023], Internet<URL:https: / / github.com / open-mmlab / mmpose> [Non-Patent Document 9] OpenMMlab Library, “animal-pose-estimation,” [online], [Retrieved November 29, 2023], Internet<URL:https: / / github.com / topics / animal-pose-estimation> [Non-Patent Document 10] OpenAI, "Whisper," [online], [Retrieved November 29, 2023], Internet<URL:https: / / github.com / openai / whisper> Summary of the Invention [Problem to be solved by the invention]

[0012] In the system described in the aforementioned Patent Document 1, the information shared is limited to text messages, which means that the amount of information is insufficient and effective information sharing is not possible.

[0013] Furthermore, the support system described in the aforementioned Patent Document 2 has the problem that the objects to be photographed are limited to, for example, intravenous drip packs, and there is no flexibility in the objects to be monitored, making it difficult to deal with situations where caregivers need to efficiently share information about unknown events they encounter.

[0014] Furthermore, the group management system described in the aforementioned Patent Document 3 processes group management using log information that shows the history of electronic medical record information access by nurses belonging to the team, but it is not a system for sharing information about various events that occur to patients among nurses belonging to the team.

[0015] Furthermore, in the monitoring device described in Patent Document 4, an event determination unit determines the occurrence of an event in a subject and performs processing to cut out a video of a predetermined time period including the time of the event, whereas in the present application, a care provider (nurse, etc.) recognizes the occurrence of an event (the occurrence of a point of interest at which information is desired to be shared) in the care recipient (patient, etc.) and cuts out the video. For this reason, the monitoring device described in Patent Document 4 is a machine-based determination, making it difficult to obtain all the information that is desired to be shared, and also making it difficult to cut out video that takes into account the circumstances of each care provider (nurse, etc.).

[0016] Here, the "circumstances of individual caregivers (nurses, etc.)" means that the behavior of the care recipient about which they want to share information varies depending on the caregiver. For example, even if the caregiver notices an unusual behavior in the care recipient, the caregiver may not consider the time period during which the unusual behavior occurs to be a time of interest for information sharing if they already know the cause and meaning of the behavior from the care recipient's mother or other relevant source, or if it is the second or subsequent occurrence. However, the monitoring device described in Patent Document 4 makes a uniform determination using an event determination unit, and therefore is unable to cut out video according to the judgment of each individual caregiver.

[0017] Furthermore, the monitoring device described in Patent Document 4 detects the occurrence of a predetermined event (phenomenon) related to the subject, and is therefore unable to detect unknown events occurring to the subject. In contrast, the present application is able to detect events unknown to the caregiver (nurse, etc.) and share the information. Note that even if an event is unknown to the caregiver (nurse, etc.), the cause and meaning may be known to the mother of the care recipient (patient, etc.), and the present application makes it possible to utilize this to allocate the relevant parties.

[0018] Furthermore, whereas the monitoring device described in Patent Document 4 performs a process of cutting out video for a predetermined period of time, in the present application, the length of time for cutting out the video can be freely specified by the care provider (nurse, etc.).

[0019] In the present application, more appropriate adjustment information is created by performing a process of extracting video data using timing information for specifying a time point of interest input by a care provider (such as a nurse) to create a first record fragment, and then performing a process of creating a second record fragment from the first record fragment that includes a detection item change section, or a process of creating a second record fragment that includes both a detection item change section and a sensor data change section (see FIG. 19 described later). However, the monitoring device described in Patent Document 4 does not perform two-stage video extraction using such different processes.

[0020] Furthermore, the pediatric information sharing system described in the aforementioned Patent Document 5 uses a calendar display to store recorded data, including images related to the child's condition, on a management server, but does not collect information from the perspective of information sharing.

[0021] Furthermore, the electronic medical record system described in the aforementioned Patent Document 6 can display a schedule of tasks related to each management item during hospitalization, but does not collect information from the perspective of information sharing.

[0022] As described above, conventional systems attempt to solve some of the various issues related to care. However, no system has been proposed that solves the following problem: When a caregiver (e.g., nurse) provides care to a care recipient (e.g., patient), the caregiver may encounter an unknown behavior in the care recipient (a behavior in the care recipient that is not normally seen or heard, and whose cause or meaning is unknown) and be faced with the problem of not knowing how to deal with it. This behavior needs to be treated as an event and information shared among those involved in the care. Furthermore, rather than simply sharing information among those involved in the care, there is a need to efficiently understand the cause and meaning of the unknown behavior in the care recipient and use this information in subsequent care.

[0023] The object of the present invention is to provide a data collection system and program that allows those involved in the care to share information about the condition of a person receiving care, which is unknown to the care provider, and that allows the cause and meaning of the condition to be efficiently understood. [Means for solving the problem]

[0024] The present invention provides a computer-based data collection system for collecting data relating to the provision of medical, nursing, caregiving, assistance, or health care or similar care, comprising: a care setting means for receiving a setting input of the placement of people or living beings or alternative inanimate objects for assigning roles to the care participants, including a care provider who provides care, a care recipient who receives care from the care provider, a care coordinator who coordinates care event information related to the care, and a care evaluator who receives an evaluation request from the care coordinator, and storing the input in a participant placement storage means; a care event information collecting means for receiving input of care event information, which includes video data, still image data, or audio data showing the state of the care recipient recorded in a care scene in which the care provider provides care to the care recipient, and care provider transmitted information including the care provider's evaluation request intention as care provider annotation data to be added to the data; a care-related information storage means for storing care-related information including the care event information collected by the care event information collection means and additional information obtained thereafter; a coordination means for supporting or substituting a care coordinator's coordination work for the care event information stored in the care-related information storage means, the coordination means creating coordination information including video data, still image data, or audio data for an evaluation request, and coordinator annotation data for the evaluation request to be added to these data, and storing the created coordination information in the care-related information storage means as care-related information; an evaluation result collection means for transmitting the adjustment information stored in the care-related information storage means to an evaluator terminal operated by a care evaluator stored in the related party allocation storage means via a network, receiving evaluator annotation data including the evaluation results by the care evaluator who referenced the adjustment information from the evaluator terminal via the network, and associating the received evaluator annotation data with the adjustment information as care-related information and storing it in the care-related information storage means; an output means for executing an output process of displaying or transmitting the care-related information stored in the care-related information storage means on a screen to a terminal operated by a care provider, a care coordinator, a care evaluator, a care recipient, or other related person having reference authority, printing it out, outputting it as audio, transmitting it to another linked system, or writing it into a recording medium; The present invention is characterized by the following features.

[0025] Here, the "person" in "care recipient" includes not only humans but also pets, dolls, animals, plants, etc. Details of this will be described later in the section (Scope of those involved in care: care recipient, care provider, care coordinator, and care evaluator) in <Outline and terminology of data collection system 10> in the [Form for carrying out the invention].

[0026] Furthermore, "the deployment of personnel, living beings, or inanimate substitutes thereof" is an expansion of so-called "personnel deployment" to include non-human entities, and in the explanation of this application, it is sometimes referred to as "stakeholder deployment" or simply "deployment" when the surrounding text clearly indicates a stakeholder deployment. Of these, "living beings" include, for example, pets such as dogs and cats that are the subject of care, captive animals and plants, birds, and fish, and "stakeholder inanimate substitutes" include dolls and robots. Note that at least one role in a stakeholder deployment is assigned to a human, meaning that not all roles are assigned to non-humans. In that sense, even if some of the stakeholders include pets (non-humans), this can still be called a personnel deployment.

[0027] In the data collection system of the present invention, care participants (including care providers, care recipients, care coordinators, and care evaluators) are registered in advance in a participant allocation storage means along with their respective roles, and a care event information collection means collects video data, still image data, or audio data showing the appearance of the care recipient recorded in care scenes in which the care provider provides care to the care recipient, as well as provider annotation data to be added to these data (hereinafter referred to as "video data, etc."), and stores these as care event information in a care-related information storage means. Subsequently, an adjustment means assists or substitutes the care coordinator registered in the participant allocation storage means in adjusting the care event information, creating adjustment information (video data, etc. for evaluation requests and adjuster annotation data added to the data), which is stored as care-related information in the care-related information storage means. Furthermore, an evaluation result collection means sends the adjustment information to a care evaluator registered in the participant allocation storage means, receives evaluator annotation data including the care evaluator's evaluation results for the video data, etc., and stores this in the care-related information storage means as care-related information.

[0028] Therefore, starting from the transmission of video data and other related annotation data by the care provider (including the care provider's intent to request evaluation), information is shared with the care coordinator and care evaluator, and evaluation is achieved in this order. This order results in efficient information collection. Furthermore, since the information to be shared and evaluated is information that indicates the state of the care recipient, which is unknown to the care provider, the present invention does not capture the occurrence of a specific event, but rather allows for the collection of information that is not dependent on the type of event.

[0029] Therefore, in the present invention, information indicating the condition of the care recipient, which is unknown to the care provider, is transmitted in the order of the related party arrangement previously set in the related party arrangement storage means, and information is added in that order, so information sharing is not chaotic. Therefore, in order to obtain useful information, the present invention creates an efficient information flow, allowing the care provider to efficiently understand the cause and meaning of the condition of the care recipient, which is unknown to the care provider. This efficient understanding will be described in more detail in the <Effects of this embodiment> in the [Mode for carrying out the invention].

[0030] Furthermore, if care providers can obtain useful information from care evaluators and care coordinators about the condition of the care recipient, which is unknown to them, the quality of care can be improved and the safety of the care recipient can be ensured.In addition, if the care evaluator is the guardian (mother, etc.) of the care recipient, the care evaluator's peace of mind can be ensured, thereby achieving the above-mentioned objectives.

[0031] <Concerned Person Arrangement Mode Set in Concerned Person Arrangement Storage Means>

[0032] In addition, in the above-mentioned data collection system, The related party arrangement storage means The system can be configured to store the arrangements of care providers, care coordinators, and care evaluators when providing primary care and the arrangements of care providers, care coordinators, and care evaluators when providing secondary care for the same care recipient, in association with each other, with different arrangements of parties involved.

[0033] In this way, if the related party placement storage means is configured to be able to associate and store related party placements for primary care and secondary care (hereinafter sometimes referred to as "two-stage related party placement"), the related party placements for primary care and the related party placements for secondary care can be linked and stored according to the differences in the content of each of the primary care and secondary care, so that the setting and registration of the placement of related party personnel, etc. can be realized in an organized state, and an efficient flow of information can be formed in each of the primary care and secondary care, allowing effective information sharing and evaluation in each.

[0034] Furthermore, the above-mentioned stakeholder arrangement settings and registration can be realized in an organized state because, for example, while the positions and roles of the stakeholders will be interchangeable between the primary caregiver and secondary caregiver, the only thing that is always common between the primary caregiver and secondary caregiver is the care recipient, so if one were to simply register these roles in the system, there would be 4 x 2 - 1 = 7 possible roles, and a system that could assign 7 different roles would have to be constructed. However, in the present invention, the roles of each stakeholder can be registered in an organized state in a two-stage stakeholder arrangement.

[0035] Furthermore, in this invention, the intertwined contents of care are separated into primary care and secondary care, but the terms primary care and secondary care and the way of separating them (the act of separating them) are not existing but are unique to this invention. In other words, relating the intertwined contents of care after separating and organizing them into primary care and secondary care is not something that comes naturally or is an easy idea.

[0036] Specific examples of two-stage stakeholder allocation include the exemplary care shown below and stakeholder allocation for consent confirmation support care.

[0037] (Two-stage stakeholder arrangement for exemplary care) In the case where the related party arrangement storage means is configured to be able to store the related party arrangements of the primary care and secondary care in association with each other, Secondary care is model care provided to the care recipient by a person other than the primary care provider, who acts as a secondary care provider to serve as a reference for the primary care provider when providing primary care. The related party arrangement storage means includes: The person who will be the primary care provider will be remembered as the secondary care care assessor or care coordinator, or other party with reference authority. The person who will be the care provider for the primary care can adopt a configuration in which the model reference video data or still image data registered in the secondary care and stored in the care-related information storage means can be viewed in the role of the secondary care care evaluator or care coordinator, or other relevant person with the authority to view it.

[0038] Details of this two-stage arrangement of participants for exemplary care will be described later in the description of the <Configuration of the data collection server 20 / care setting means 22> in the [Form for implementing the invention], Figure 8, and Case J4 in Figure 41.

[0039] (Two-stage arrangement of stakeholders for consent confirmation supportive care) In the case where the related party arrangement storage means is configured to be able to store the related party arrangements of the primary care and secondary care in association with each other, Secondary care is supplementary care that confirms consent, in preparation for the primary care that will be provided later, by obtaining consent from the care recipient in advance for the implementation of primary care. The related party arrangement storage means includes: The person who will be the primary caregiver will be remembered as the secondary care evaluator. A configuration can be adopted in which the person who will be the primary care provider can, in the role of a secondary care care evaluator, refer to video data or still image data showing how consent has been obtained from the care recipient, which is registered in secondary care and stored in a care-related information storage means, confirm whether consent has been obtained, and use the confirmation results as secondary care evaluator annotation data.

[0040] Details of this two-stage arrangement of stakeholders for consent confirmation support care will be described later in the description of <Configuration of data collection server 20 / care setting means 22> in [Form for implementing the invention], Figures 9 and 42, Case J5.

[0041] (Personnel placement for educational care using dolls) In addition, although it is not a two-stage arrangement of stakeholders, it is possible to set and register the arrangement of stakeholders for educational care using dolls such as the following.

[0042] That is, in the above-mentioned data collection system, The related party arrangement storage means The system can be configured to store students who receive care training or practice as care providers, dolls that represent children, elderly people, or other people in need of care who are the subject of care training or practice as care recipients, teachers or instructors who provide care education as care coordinators, and guardians or caregivers of people in need of care as care evaluators.

[0043] Details of the placement of participants for educational care using this doll will be described later in the explanation of <Configuration of data collection server 20 / care setting means 22> in [Form for carrying out the invention] and in case J10 of Figure 47.

[0044] <Expanding the care event information collected>

[0045] (Configuration for accepting input of start care item information and start time of care scene) In addition, in the above-mentioned data collection system, The care setting instrument is The system is also configured to execute a process of accepting a setting input of care items indicating the type of care that the care provider may provide to the care recipient in each care situation or the type of care that the care provider plans to provide in each care situation; The means of collecting care event information are: It is desirable that the care event information be configured to also execute a process for accepting input of start care item information indicating which of the care items pre-set in the care setting means the care provider will now begin implementing, and the start time of the care scene, as information transmitted by the provider.

[0046] Here, "the type of care that may be implemented in each care situation" means the type of care (care item) that is expected to be implemented for each care recipient set and registered in the related party allocation storage means, among the many care types (care items) prepared as a care master, or among the many care types (care items) that exist in general. Also, "the type of care that is planned to be implemented in each care situation" means the type of care (care item) that is planned to be implemented for each care recipient set and registered in the related party allocation storage means (for example, included in a care session scheduled for a period of one day or half a day, etc.) among the many care types (care items) prepared as a care master, or among the many care types (care items) that exist in general.

[0047] It is not essential to input the "care scene end time."

[0048] In this way, when the system is configured to accept input of the start care item information and the start time of the care scene, the care setting means narrows down the care items, so the care provider (nurse, helper, etc.) can select from a limited number of options instead of selecting the type of care (care item) for the care scene to be started from an extremely large number of options, thereby reducing the workload of the care provider.

[0049] In addition, by collecting the start care item information and the start time of the care scene, this care provider behavior information can be used as data for creating an electronic medical record and data for calculating the degree of care necessity. Furthermore, the start care item information can be used in the interval detection process (see Figure 19 described later) by the adjustment means.

[0050] (Configuration for collecting information transmitted by the target person, such as video data, still image data, or audio data, and target person annotation data to be added to these data) Furthermore, in the case where the input of the above-mentioned start care item information and the start time of the care scene is accepted, The means of collecting care event information are: In addition to accepting input of information sent by the implementer that constitutes the care event information, as information sent by the target that constitutes the care event information, It is desirable that the system be configured to accept input of video data, still image data, or audio data showing the state of the care recipient recorded in a non-care situation other than the care situation, as well as the evaluation request intention of the care recipient or their input agent as subject annotation data to be added to these data.

[0051] Here, the "data entry agent" refers to, for example, the care recipient's mother or father, or a relative, friend, or acquaintance who visits the facility. Visitors to the facility may obtain authorization (login authority) to register video data and other data in the system when they visit, or may obtain authorization to operate buttons on their behalf (e.g., receive instructions from the facility to press a certain button in an emergency). Alternatively, they may obtain registration authorization and send files after recording (photographing or audio recording). The care provider (e.g., the nurse in charge) or the care coordinator (e.g., the nurse's supervisor) may be asked to register files of recorded (photographed or audio recorded) video data and other data. In this case, the registration is performed by the care recipient or their data entry agent, not by the care provider or care coordinator.

[0052] In addition, if the person receiving care is able to perform operations for recording video data (photography, audio recording) and operations for inputting subject annotation data, the person receiving care will perform these operations themselves, and if the person receiving care is unable to perform these operations themselves, an ``input proxy'' will perform these operations on their behalf.

[0053] In this way, when a configuration is adopted in which video data, still image data, or audio data, which are information transmitted by the care recipient, and the annotation data for the care recipient that is added to these data, are collected, it is possible to collect video data showing the state of the care recipient during such times, and the annotation data for the care recipient, even when the care provider (nurse, helper, etc.) is not near the care recipient, as there may be points in time when the care recipient needs to be informed. This makes it possible to collect more information about the care recipient, and therefore to provide more appropriate care.

[0054] (Configuration for accepting input of call time or target person needs information including call time and call intention) In the case where the above-mentioned target person transmitted information, such as video data, still image data, or audio data, and target person annotation data to be added to these data are collected, The means of collecting care event information are: In addition to accepting input of information sent by the implementer that constitutes the care event information, as information sent by the target that constitutes the care event information, It is desirable that the system be configured to accept input of subject needs information that includes the call time when the care recipient or their input agent presses a button on a call device or an input device that has the function of this call device to call the care provider, or that includes the call intention in addition to this call time.

[0055] In this way, if the system is configured to accept input of subject needs information including the call time, or the call time and the call intention, it is possible to collect subject needs information (call time, or the call time plus the call intention).By combining this with the practitioner behavior information (start care item information and start time of the care scene), it is possible to further enrich the data used to create electronic medical records and the data used to calculate the level of care necessity.

[0056] Therefore, for example, it is possible to record the time difference between the call time and the time when the care provider (nurse, etc.) starts providing care after being called by the care recipient or their input representative (care scene start time). Since the care provider (nurse, etc.) is not negligent in responding to the call due to work negligence, but is also engaged in caring for other care recipients, such time difference data can also be used to calculate the level of care need.

[0057] Note that while call devices (such as nurse call systems) have been around for some time, in this case the call device is not only used as a device with the original call function, but also the system collects the time of the call (or both the time of the call and the intention to call) made by the call device. Furthermore, operations for recording video data (photography, audio recording) and pressing buttons on a call device or an input device with similar functions are not necessarily linked, so these operations are not performed simultaneously. When these operations are performed simultaneously, they are described as separate inventions in this application.

[0058] <Video data extraction using adjustment methods>

[0059] (Configuration in which section detection processing is performed using status change detection items by adjustment means) In addition, in the above-mentioned data collection system, The care setting instrument is The intentions and emotional expressions of the care recipient that may be expressed in the care situation or that may be expressed in the care situation when a specific care item is performed among the care items that indicate the type of care, Accepting a setting input regarding a correspondence relationship with any one or a combination of two or more of the status change detection items for detecting a physical status change that can be recognized visually and audibly regarding the care recipient, including the movement of each of the care recipient's multiple facial muscles, the movement of the gaze direction, the movement of the face direction, the movement of the position of the skeletal and joint points, or the movement of other body parts of a human or living thing, or a change in the vocalization or cry of the care recipient, or a change in an intangible or tangible object emitted from the body of another human or living thing; The received correspondence relationship is stored in a status change detection item storage means in association with the identification information of the care recipient, or in association with the identification information of the care recipient and the identification information of the care item in a status change detection item storage means, The adjustment means is Using video data showing the state of the care recipient, a pattern recognition process is performed to detect a state change in a state change detection item stored in a state change detection item storage means in association with identification information of the care recipient who was the subject of the video data capture, or to detect a state change in a state change detection item stored in a state change detection item storage means in association with identification information of the care recipient and identification information of the care item performed while the video data was being captured, It is desirable that the system be configured to create a record fragment including a detection item change section in which a state change of a state change detection item is detected, and to add adjuster annotation data for an evaluation request to the created record fragment, and to execute a process of storing the record fragment and the adjuster annotation data for an evaluation request in a care-related information storage means as adjustment information.

[0060] Here, the "intangible or tangible object" in the "intangible or tangible object emanating from the body of a human or living thing" refers to an intangible object whose state change can be recognized from video data or audio data included in the video data, such as sounds such as vocalizations or cries (in the case of living things), or light such as the glow of a firefly. On the other hand, the "tangible object" refers to a tangible object whose state change can be recognized from video data, such as a nosebleed, runny nose, or excrement (in the case of living things not wearing pants or a diaper). Details of these will be described later in the section (Setting input regarding the correspondence between the intentions and emotional expressions of the care recipient and the state change detection items for detecting the physical state change of the care recipient that can be recognized visually and auditorily: Figure 19) in <Configuration of the data collection server 20 / care setting means 22> in the [Mode for carrying out the invention].

[0061] In this manner, when the adjustment means performs section detection processing (see FIG. 19 described later) using the status change detection items, the video data transmitted to the care evaluator as adjustment information can be made easier for the care evaluator to evaluate. For example, if the care evaluator is the guardian (mother, etc.) of the care recipient, the care evaluator may provide information to the facility or care coordinator, such as information about the care recipient's facial expressions and emotions, or information about the care recipient's facial expressions and emotions when providing care for a specific care item. If this information is stored in the status change detection item storage means, the adjustment means executes section detection processing according to the information provided by the care recipient's guardian (mother, etc.), creates record fragments, and transmits the record fragments (video data) to the care evaluator. In this case, if the care evaluator is the guardian (mother, etc.) of the care recipient who provided the information, the care evaluator will receive, as an evaluation target, record fragments (video data) that show the care recipient's appearance exactly as provided by the information, making evaluation easier. Furthermore, the record fragments (video data) and the evaluation results (evaluator annotation data) of the care evaluator (e.g., mother) can be used as a learning dataset (data and its labels) for constructing an initial model for pattern recognition by machine learning for the care recipient (determining parameters). The initial model here is a dedicated model for that care recipient (specific care recipient) that can be used only for that care recipient, and is a pattern recognition model for recognizing the intentions and emotions of the care recipient using changes in facial expressions, etc. (physical state changes that can be recognized in video data).

[0062] (Configuration in which section detection processing is performed using state change detection items after extraction processing using timing information for specifying a time point of interest by the adjustment means) Furthermore, in the case where the above-described adjustment means performs section detection processing using the state change detection items, The video data included in the information sent by the implementer is The video data is continuously recorded including a time point of interest indicating a time period when the caregiver determines that attention is required for the state of the care recipient, and time periods other than the time point of interest, The provider-sent information includes timing information for specifying the time point of interest that is added by the care provider as provider annotation data to the video data recorded including the time period other than the time point of interest, The adjustment means is It is desirable that the system be configured to extract data from the recorded video data, including time periods other than the time of interest, at an extraction position determined by timing information for specifying the time of interest to create a first record fragment, and then use this first record fragment to create a second record fragment including a section in which a status change in the status change detection item was detected, and also to add adjuster annotation data for an evaluation request to the created second record fragment, and to store the second record fragment and the adjuster annotation data for an evaluation request in a care-related information storage means as adjustment information.

[0063] Here, "continuously recording including a time point of interest indicating a time period determined to require attention and time periods other than this time point of interest" means that automatic extraction processing (processing to create a first record fragment) is not performed on video data in which only the time point of interest is recorded, and therefore this is intended to exclude such cases (see Figures 32 to 34 described later). Also, automatic extraction processing (processing to create a first record fragment) is performed on video data that includes time periods other than the time point of interest, but in such cases, there are cases of continuous recording (see Figures 20 and 21 described later) and cases in which the entire care scene is recorded (see Figures 30 and 31 described later), and therefore this is intended to include both of these. The same applies to other inventions.

[0064] In this manner, if the adjustment means performs an extraction process using timing information for specifying a time point of interest, and then performs a section detection process using a status change detection item (see Figure 19 described below), the adjustment means performs an extraction process using timing information for specifying a time point of interest to create a first record fragment, and then the adjustment means uses this first record fragment to perform a section detection process to create a second record fragment.Therefore, the range in which the adjustment means performs the section detection process on the video data collected by the care event information collection means 21 can be narrowed, thereby reducing the system load and shortening the processing time.

[0065] (Configuration in which the adjustment means performs section detection processing using sensor data) Furthermore, in a configuration in which section detection processing using state change detection items is performed after the extraction processing using timing information for specifying a time point of interest by the adjustment means described above, The means of collecting care event information are: The care event information is configured to receive input of sensor data output from a sensor that measures biological information of the care recipient, The adjustment means is The sensor data from the time period in which the first record fragment was extracted from the video data is used to determine whether the sensor data is larger or smaller than a predetermined threshold, or whether the sensor data is larger or smaller than a threshold set based on the past average value for the care recipient who was the subject of the video data.This can then detect a sensor data change section in which the sensor data value changes significantly, and create a second record fragment from the first record fragment that includes both the sensor data change section and the detection item change section.The system can also be configured to add adjuster annotation data for an evaluation request to the created second record fragment, and store the second record fragment and the adjuster annotation data for an evaluation request in a care-related information storage means as adjustment information.

[0066] Here, "sensor data" includes not only output data from various sensors that measure a single piece of biometric information, but also data obtained by inputting sensor data of multiple types of biometric information output from multiple types of sensors, executing a pattern recognition process to identify whether or not there is an abnormality, and outputting the recognition results (whether or not there is an abnormality, or the likelihood that there is an abnormality).

[0067] Furthermore, "determining whether the value is greater than or equal to a threshold" means determining whether the value is greater than or equal to an upper threshold, or whether the value is less than or equal to a lower threshold. An upper threshold is set when a large sensor data value indicates an abnormality, and a lower threshold is set when a small sensor data value indicates an abnormality. Depending on the sensor data, the threshold may be set both higher and lower.

[0068] Furthermore, the "threshold value determined based on the past average value" may be a threshold value using the standard deviation or variance value calculated from all past data, or may be a threshold value determined appropriately by the designer or developer of this system, or a care coordinator, etc.

[0069] When the adjustment means performs section detection processing using sensor data in this manner, a second record fragment including both the sensor data change section and the detection item change section can be created from the first record fragment. Therefore, when the care evaluator has information about the sensor data of the care recipient (for example, information that the pulse rate data increases when a certain facial expression changes in the care recipient), adjustment information (video data) that makes it easier for the care evaluator to evaluate can be transmitted. In addition, the adjustment information (video data, sensor data) and the care evaluator's evaluation results (evaluator annotation data) in this case can also be used as a learning dataset (data and its label) for building an initial model for pattern recognition specific to the care recipient.

[0070] (Configuration in which the adjustment means performs the cutting out process of the video data using timing information for specifying the point of interest input by the care provider) In addition, in the above-mentioned data collection system, The video data included in the information sent by the implementer is The video data is continuously recorded including a time point of interest indicating a time period when the caregiver determines that attention is required for the state of the care recipient, and time periods other than the time point of interest, The provider-sent information includes timing information for specifying the time point of interest that is added by the care provider as provider annotation data to the video data recorded including the time period other than the time point of interest, The timing information for designating a time point of interest is the elapsed time from the time point of interest to the time point at which the care provider designates the timing information, and the duration of the time point of interest. The adjustment means is It is desirable that the configuration be such that data for the time period of interest is extracted from recorded video data, including time periods other than the time period of interest, at an extraction position determined by timing information for specifying the time period of interest to create a record fragment, and that adjuster annotation data for an evaluation request is added to the created record fragment, and the record fragment and the adjuster annotation data for an evaluation request are stored as adjustment information in a care-related information storage means.

[0071] In this manner, when the adjustment means performs the video data extraction process using timing information for specifying the point of interest input by the care provider, the adjustment means can perform the video data extraction process using the "elapsed time from the point of interest to the time when the care provider specifies the timing information" and "duration of the point of interest" input by the care provider as timing information for specifying the point of interest.Therefore, a record fragment (video data) that accurately reflects the care provider's intention for the evaluation request can be created and sent to the care evaluator as adjustment information.

[0072] As already described in detail in the "Problem to be Solved by the Invention" section of the present application, when the caregiver recognizes the occurrence of an event for the care recipient (the occurrence of a point of interest at which the caregiver wishes to share information), the adjustment means cuts out video data, making it possible to collect all the information the caregiver wishes to share and to cut out video in consideration of the circumstances of each caregiver (whether the caregiver already knows the cause or meaning, etc.) In contrast, the monitoring device described in the above-mentioned Patent Document 4 uses an event determination unit to determine the occurrence of a subject event, making it difficult to obtain all the information the caregiver wishes to share and also making it impossible to cut out video data in consideration of the circumstances of each caregiver.

[0073] Furthermore, as already described in detail in the "Problem to be Solved by the Invention" section of this application, the present invention can capture events unknown to the caregiver and share the information. In contrast, the monitoring device described in Patent Document 4 detects the occurrence of a predetermined event (phenomenon) related to the subject, and therefore cannot capture unknown events occurring to the subject. Furthermore, while the present invention performs a video data extraction process by specifying the "length of time at the point of interest," the monitoring device described in Patent Document 4 can only extract video for a predetermined period of time.

[0074] (The timing information for specifying the time point of interest for video data extraction processing is input by the care recipient or their representative pressing a button on the call device or an input device with a call function.) In addition, in the case where the above-mentioned target person transmitted information, such as video data, still image data, or audio data, and target person annotation data to be added to these data are collected, The video data included in the target person's transmission information is The video data is continuously recorded including a time point of interest indicating a time period when the care recipient or their input agent determines that attention is required regarding the state of the care recipient, and a time period other than the time point of interest, The subject transmission information includes timing information for specifying the time point of interest that is assigned by the care recipient or their input representative as subject annotation data to the video data recorded including the time period other than the time point of interest, The adjustment means is The system is configured to execute a process of extracting data for the time period of interest from video data recorded including time periods other than the time period of interest at an extraction position determined by timing information for specifying the time period of interest to create a record fragment, and to add adjuster annotation data for an evaluation request to the created record fragment, and to store the record fragment and the adjuster annotation data for an evaluation request in a care-related information storage means as adjustment information, The means of collecting care event information are: In addition to accepting input of information sent by the implementer that constitutes the care event information, as information sent by the target that constitutes the care event information, The system accepts input of subject needs information including a call time when the care recipient or their input representative presses a button on a call device or an input operation device having the function of the call device to call the care provider, or including the call intention in addition to the call time; The configuration may be such that input of timing information for designating a time point of interest is accepted when the care recipient or their input representative presses a button on a call device or an input operation device.

[0075] In this configuration, when the care recipient or their input representative presses a button on a call device or an input device with a call function to input timing information for specifying a time point of interest for video data extraction processing, pressing a button on the call device or input device with a call function to call a care provider (nurse, helper, etc.) simultaneously calls the care provider and sends information for video data extraction processing showing the care recipient's condition (timing information for specifying a time point of interest) to the network. If possible, the button pressing operation can be performed by the care recipient themselves (patient, care recipient, etc.), or by an input representative such as the care recipient's mother who is visiting.

[0076] Because they are simultaneous and parallel, the evaluation request intention (e.g., a mother saying, "He looks like he's in pain. What's wrong?") and the call intention (e.g., a mother saying, "He looks like he's in pain, please come over") assigned to the captured video data, etc., are nearly identical. Note that "nearly identical" means that what the mother is thinking when pressing the button is nearly identical (the above "He looks like he's in pain..." is an example of what is going through her mind). In this invention, the call time is essential, but the call intention is not. Also, as will be described later with reference to FIG. 5, the evaluation request intention and the call intention as target person transmission information may be a single fixed signal. For example, a single fixed signal indicating the evaluation request intention (look here!) and a single fixed signal indicating the call intention (please come over, I want you to see!) may be acceptable. Furthermore, when the evaluation request intention and the call intention are input by voice, it does not mean that each intention is spoken separately, but a common voice input is acceptable.

[0077] <Communicating intentions through gestures or voice data>

[0078] (Configuration for converting gesture or voice data for conveying a rating request intention into rating request intention identification information or text data indicating the rating request intention) In addition, in the above-mentioned data collection system, The video data or audio data included in the implementer transmitted information includes gesture or audio data for conveying the evaluation request intention, which corresponds to the implementer annotation data, and The adjustment means is The configuration may be such that a process is executed in which gesture or voice data for transmitting a rating request intention is converted into rating request intention identification information or text data indicating the rating request intention.

[0079] Here, "evaluation request intention identification information" refers to information that identifies multiple evaluation request intentions that have been prepared in advance. For example, if evaluation request intentions such as "Look here!", "Listen to this!", "What does this mean?", "Is he happy?", and "Is he in pain?" have been prepared in advance, then "evaluation request intention identification information" refers to information that identifies these (such as an identification number or identification code).

[0080] This includes cases where a gesture is converted into evaluation request intention identification information, cases where a gesture is converted into text data, cases where voice data is converted into evaluation request intention identification information, and cases where voice data is converted into text data.

[0081] In this manner, when gestures or voice data for conveying an evaluation request intention are converted into evaluation request intention identification information or text data indicating the evaluation request intention, the gestures or voice data for conveying an evaluation request intention that correspond to the caregiver annotation data are included in the video data or voice data showing the state of the care recipient of interest, so the caregiver can input the evaluation request intention without having to operate a device to input the evaluation request intention.

[0082] (Configuration that integrates gesture conversion results and voice data conversion results) Furthermore, in the above-mentioned data collection system, The video data included in the implementer's transmitted information includes both gesture and voice data for conveying the evaluation request intention, which corresponds to the implementer's annotation data, and The adjustment means is The configuration may also be such that each of the gestures and voice data for conveying an evaluation request intention is converted into evaluation request intention identification information or text data indicating the evaluation request intention, and the gesture conversion results and the voice data conversion results are integrated to execute a process to obtain final evaluation request intention identification information or text data indicating the evaluation request intention.

[0083] In this way, when the gesture conversion result and the voice data conversion result are integrated, even if the intention of one conversion result is unclear, if some intention is obtained from the other conversion result, the other intention can be adopted. Note that if the two conversion results are completely different, the implementer annotation data may be generated as an unknown evaluation request intention, or the implementer annotation data may be generated as an implementation data in which both evaluation request intentions are written in parallel.

[0084] (Configuration for converting gesture or voice data for conveying a call intention into call intention identification information or text data indicating the call intention) In addition, in the case where the input of the target person needs information including the call time or the call time and the call intention is accepted, The target person needs information includes a call time and a call intention, The call intention is indicated by gestures or voice data for conveying the call intention contained in video data or voice data that is continuously recorded about the state of the care recipient, including not only care scenes but also time periods other than care scenes; The adjustment means is The configuration may be such that data from the time period before and after the call time when a button on the call device or an input operation device having the function of this call device is pressed is extracted from video data or audio data that is continuously recorded, including time periods other than care scenes, and a process is executed to convert the gesture or audio data for communicating the call intention contained in the extracted data into call intention identification information or text data indicating the call intention.

[0085] Here, "call intention identification information" refers to information that identifies multiple pre-prepared call intentions. For example, if pre-prepared call intentions include "I want to go to the toilet," "Please sit up," "I want to turn over in bed," "My affected area hurts," and "My head is itchy," then "call intention identification information" refers to information that identifies these (such as an identification number or code).

[0086] In this way, when gesture or voice data for communicating a call intention is converted into call intention identification information or text data indicating the call intention, the care recipient or their input representative can communicate the call intention using video data or voice data that is continuously recorded about the care recipient's condition, including time periods other than when care is being provided. Furthermore, because the call intention is converted into call intention identification information or text data, it can be used to create electronic medical records and calculate care needs (such as nursing needs).

[0087] Conventionally, there have been call devices (such as nurse call devices) that communicate a call intention by voice, but the voice input function is provided as a function of the call device. In contrast, in the present invention, the state of the care recipient is continuously recorded as video data or audio data regardless of whether or not there is a notable occurrence of the care recipient's behavior (in the case of continuous recording as shown in Figures 20 and 21 described below), and the video data or audio data is used to communicate the call intention by gesture or audio data. If gesture or audio data for communicating a call intention is included in video data or audio data that is continuously recorded, including time periods other than care scenes, it is not possible to determine from that alone which part (time point) of the video data or audio data contains the gesture or audio data for communicating a call intention. Therefore, data from the time period before and after the call time when a button on the call device or an input operation device having the function of the call device is pressed is extracted.

[0088] <Output method details>

[0089] (Configuration for outputting data for creating electronic medical records) In addition, in the case where the configuration is such that the input of the start care item information and the start time of the care scene is accepted, The output means is The care-related information stored in the care-related information storage means is used to execute a process of outputting data for creating an electronic medical record that can be used to create an electronic medical record. The data for creating electronic medical records includes at least: Start care item information and care scene start time to be placed in the electronic medical record; Information on the storage location of video data, still image data, or audio data showing the state of the care recipient, which is used as link information to be attached to the electronic medical record; It is desirable that the information includes annotation data including adjuster annotation data and evaluator annotation data that are added to video data, still image data, or audio data that are arranged near the link information.

[0090] When configured to output data for creating electronic medical records in this manner, the care-related information collected by this system can be used to create electronic medical records, thereby reducing the workload of care providers and other care-related personnel.

[0091] (Configuration for outputting data for calculating care need) Furthermore, in the case where the input of the target person needs information including the call time or the call time and the call intention is accepted, The output means is It is desirable that the system be configured to execute a process of outputting care need calculation data required to calculate nursing need, care need, or other care need using care-related information including the start care item information and care scene start time as information transmitted by the implementer stored in the care-related information storage means, and the call time as information transmitted by the subject, or the call time plus the call intention.

[0092] If the system is configured to output data for calculating care need in this manner, the care-related information collected by the system can be used to calculate care need, making it possible to build a more appropriate care system.

[0093] <Configuration for pre-setting the devices and apps to be used>

[0094] In addition, in the above-mentioned data collection system, The care setting instrument is The system is configured to receive setting inputs of devices and applications used by a care provider, a care recipient, or an input representative thereof, and to execute a process of storing the input setting information of the used devices and applications in a used device / application setting storage means; The means of collecting care event information are: It is desirable that the application be configured to be distributed via a network to devices used by the care provider, or the care recipient or their input agent, depending on the setting information of the device and application used stored in the device and application setting storage means.

[0095] When the device and application to be used are configured in advance in this way, when the application program to be used differs depending on the type of device used by the care provider, or the care recipient, or their input agent, and its usage conditions, it becomes possible to present a selection of available application programs with an appropriate narrowing down of options in response to a request from the care provider, or the care recipient, or their input agent, thereby reducing the workload of the care provider, or the care recipient, or their input agent, and preventing the inconvenience of acquiring an application program that is not suitable for the type of device used or its usage conditions.

[0096] <Configuration for controlling the timing of emergency signal transmission>

[0097] Furthermore, in the above-mentioned data collection system, The means of collecting care event information are: The implementer annotation data is configured to be capable of receiving an input of an emergency signal for requesting an urgent evaluation, The input means operated by the care provider is: It is desirable that the configuration be such that whether or not an emergency signal is sent can be controlled by the number of times a button is pressed, and that if a predetermined time has passed after the first pressing operation without a second pressing operation being performed, an emergency signal is sent to the care event information collection means, and that if a second pressing operation is performed within a predetermined time after the first pressing operation, the sending of the emergency signal is avoided.

[0098] This configuration for controlling the timing of emergency signal transmission ensures the basic function of enabling emergency signal transmission, since caregivers may need an urgent evaluation depending on their situation. Furthermore, because an emergency signal can be transmitted with a single press, i.e., emergency signal transmission is the default setting, the simplest operation is the operation for transmitting an emergency signal, thereby achieving a reasonable system configuration that allows for emergency response. Furthermore, a second press within a predetermined time period prevents the transmission of an emergency signal, thereby preventing the inconvenience of transmitting an emergency signal when it is not necessary, which would place unnecessary burdens on care coordinators and care evaluators.

[0099] <Configuration that allows care coordinators to respond to care providers>

[0100] In addition, in the above-mentioned data collection system, The adjustment means is When creating the coordination information, the care event information stored in the care-related information storage means is transmitted to a coordinator terminal operated by a care coordinator via a network, It is desirable that the system be configured to receive, in addition to the adjustment instructions from the care coordinator for creating adjustment information and the coordinator annotation data for requesting an evaluation from the care evaluator, advice from the care coordinator to the care provider or other response coordinator annotation data, and to execute a process of associating this response coordinator annotation data with the adjustment information as care-related information and storing it in the care-related information storage means.

[0101] Here, the fact that the care coordinator sends the care provider "the response coordinator annotation data as care-related information in association with the coordination information and stored in the care-related information storage means" does not mean that the response coordinator annotation data is coordination information. The response coordinator annotation data is simply stored in association with the coordination information and is not sent to the care evaluator, so it is non-coordination information.

[0102] In this way, if the system is configured to allow the care coordinator to respond to the care provider, there may be cases where the care coordinator can directly answer the care provider's questions or inquiries without necessarily requesting an evaluation from the care evaluator.A direct answer allows the necessary information to be conveyed to the care provider more quickly, making it possible to respond to such cases.

[0103] <Program invention>

[0104] The program of the present invention is for causing a computer to function as the data collection system described above.

[0105] The above program or a portion thereof may be recorded on a recording medium such as a magneto-optical disk (MO), compact disk (CD), digital versatile disk (DVD), flexible disk (FD), magnetic tape, read-only memory (ROM), electrically erasable and programmable read-only memory (EEPROM), flash memory, random access memory (RAM), hard disk drive (HDD), solid-state drive (SSD), or flash disk, and may be transmitted using a transmission medium such as a wired network (e.g., local area network (LAN), metropolitan area network (MAN), wide area network (WAN), the Internet, an intranet, or an extranet), a wireless communication network, or a combination thereof, or may be carried on a carrier wave. Furthermore, the above program may be a portion of another program, or may be recorded on a recording medium together with a separate program. [Effects of the Invention]

[0106] As described above, according to the present invention, information is collected starting from the transmission of video data etc. by the care provider and the provider's annotation data (including the care provider's intention to request evaluation), so that information indicating the condition of the care recipient that is unknown to the care provider can be shared and evaluated, and since the information is shared and evaluated in the order pre-set and registered in the related party allocation storage means, it has the effect of enabling the cause and meaning of the unknown condition of the care recipient to be efficiently learned. [Brief explanation of the drawings]

[0107] [Figure 1] 1 is a diagram showing the overall configuration of a data collection system according to an embodiment of the present invention; [Figure 2] FIG. 4 is an explanatory diagram of a flow of transmission of care event information collected by the data collection system of the embodiment. [Figure 3]FIG. 3 is an explanatory diagram of data and adjustment information (part 1) collected by the data collection system according to the embodiment. [Figure 4] FIG. 4 is an explanatory diagram of data and adjustment information (part 2) collected by the data collection system according to the embodiment. [Figure 5] FIG. 10 is an explanatory diagram of data and adjustment information (part 3) collected by the data collection system according to the embodiment. [Figure 6] FIG. 3 is a diagram showing the configuration of an album database according to the embodiment. [Figure 7] 10 is a view showing an example of a related party arrangement screen constituting the care setting screen of the embodiment. FIG. [Figure 8] 4 is a diagram showing the record configuration and a specific example (part 1) of a related party location table, which is the related party location storage means of the embodiment. FIG. [Figure 9] 10 is a diagram showing the record configuration and a specific example (part 2) of a related party location table, which is the related party location storage means of the embodiment. FIG. [Figure 10] FIG. 10 is an explanatory diagram of the setting of a care session according to the embodiment. [Figure 11] FIG. 10 is a view showing an example of a device / application use setting screen constituting the care setting screen of the embodiment. [Figure 12] 5A and 5B are diagrams showing a record configuration and a specific example of a care history table according to the embodiment. [Figure 13] 5A and 5B are diagrams showing a record configuration and a specific example of a state data management table according to the embodiment. [Figure 14] 4A and 4B are diagrams showing the configuration and a specific example of an evaluation and appending table according to the embodiment. [Figure 15] 4A and 4B are diagrams showing record configurations of a customer table and a service provider table according to the embodiment. [Figure 16] FIG. 10 is a view showing an example of an adjuster screen in the embodiment. [Figure 17] FIG. 10 is a view showing an example of an evaluator screen according to the embodiment. [Figure 18] FIG. 10 is a view showing an example of a care record display screen according to the embodiment. [Figure 19] 5A to 5C are explanatory diagrams illustrating a section detection process performed by the adjustment unit according to the embodiment. [Figure 20]FIG. 10A is a diagram showing the device layout in the case of continuous recording according to the embodiment; [Figure 21] FIG. 10 is an explanatory diagram of the collection of video data or audio data in the device arrangement (A) in the case of continuous recording in the embodiment. [Figure 22] FIG. 10 is a view showing an example of a care provider input screen (compatible with continuous photography) in the device arrangement (A) of the embodiment. [Figure 23] FIG. 10 is an exemplary diagram showing a dedicated input operation device having one physical button operated by a care provider, a care recipient, or their input agent in the device arrangement (A) of the embodiment. [Figure 24] FIG. 10 is an exemplary diagram of a dedicated input operation device having two physical buttons operated by a caregiver in the device arrangement (A) of the embodiment. [Figure 25] FIG. 10 is an illustrative view of an input operation device that can be used as both a calling device (selectable) and a call device operated by the care recipient or their input agent in the device arrangement (A) of the embodiment. [Figure 26] FIG. 10 is an illustrative diagram of an input operation device (mode-switchable) that can also be used as a call device operated by a care provider, a care recipient, or their input agent in the device arrangement (A) of the embodiment. [Figure 27] FIG. 10 is an exemplary diagram of an input operation device that is also used as a call device (concurrent type) operated by the care recipient or their input agent in the device arrangement (A) of the embodiment. [Figure 28] FIG. 10 is an exemplary diagram of a call device operated by a care recipient or their input agent in the device arrangement (A) of the embodiment. [Figure 29] FIG. 10 is an explanatory diagram of the flow of data collection processing in device layout (A) in the case of continuous recording according to the embodiment. [Figure 30] FIG. 10B is a diagram showing the device arrangement in the case of recording the entire care scene according to the embodiment. [Figure 31] FIG. 10 is an explanatory diagram of the collection of video data or audio data in the device arrangement (B) in the case of recording the entire care scene in the embodiment. [Figure 32] FIG. 10 is a diagram showing a device arrangement (C) in the case of a time-point-limited recording according to the embodiment. [Figure 33]FIG. 10 is an explanatory diagram of collection of moving image data or audio data in device layout (C) in the case of time-point limited recording in the embodiment. [Figure 34] FIG. 10 is a view showing an example of a care provider input screen (compatible with focused time-point limited photography) in the device arrangement (C) of the embodiment. [Figure 35] FIG. 10 is a diagram showing the device arrangement (D) in the case of file transmission according to the embodiment. [Figure 36] FIG. 10 is a view showing an example of a file registration screen in the device layout (D) according to the embodiment. [Figure 37] FIG. 10 is an explanatory diagram of the flow of data collection processing in device layout (D) in the case of file transmission according to the embodiment. [Figure 38] FIG. 10 is an explanatory diagram of Example J1 (utilization in monitoring care in a sick child daycare center) of the embodiment. [Figure 39] FIG. 10 is an explanatory diagram of Example J2 (education of mothers and nursery staff on emergency response in a sick child nursery) of the embodiment. [Figure 40] FIG. 10 is an explanatory diagram of Case J3 (understanding the difficult wishes of a hospitalized seriously ill child) in the embodiment. [Figure 41] FIG. 10 is an explanatory diagram of case J4 (a mother wants to share information about her child with staff) in the embodiment. [Figure 42] FIG. 10 is an explanatory diagram of Case J5 (decision-making and judgment on treatment during remote medical care) in the embodiment. [Figure 43] FIG. 10 is an explanatory diagram of Case J6 (medical support tailored to the needs of children) of the embodiment. [Figure 44] FIG. 10 is an explanatory diagram of Case J7 of the embodiment (in home nursing, mothers and nurses share issues and information and grow together). [Figure 45] FIG. 10 is an explanatory diagram of Case J8 (information sharing in electronic medical records) of the embodiment. [Figure 46] FIG. 10 is an explanatory diagram of case J9 (school) of the embodiment. [Figure 47] FIG. 10 is an explanatory diagram of a case J10 (medical professionals and parent education) of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0108] An embodiment of the present invention will be described below with reference to the drawings. Fig. 1 shows the overall configuration of a data collection system 10 of this embodiment, and Fig. 2 shows the flow of transmission of care event information collected by the data collection system 10. Figs. 3 to 5 are explanatory diagrams of the data and adjustment information collected by the data collection system 10. 6 is a diagram of the album database 33, FIG. 7 is an example of a related party arrangement screen 400 constituting the care setting screen, FIG. 8 and FIG. 9 are diagrams showing the record configuration and specific examples of the related party arrangement table 33B, which is a related party arrangement storage means, FIG. 10 is an explanatory diagram of setting a care session, FIG. 11 is an example of a used device / used application setting screen 600 constituting the care setting screen, FIG. 12 to FIG. 14 are diagrams showing the record configuration and specific examples of the care history table 33F, the status data management table 33G, and the evaluation / addition table 33H, FIG. 15 is a diagram showing the record configuration of the customer table 33L and the service provider table 33M, FIG. 16 to FIG. 18 are example diagrams of an adjuster screen 700, an evaluator screen 800, and a care record display screen 900, and FIG. 19 is an explanatory diagram of the section detection processing by the adjustment means 23. Furthermore, Figures 20 to 29 show device layouts (A) for continuous recording, Figures 30 and 31 show device layouts (B) for recording the entire care scene, Figures 32 to 34 show device layouts (C) for recording limited to a specific time point, and Figures 35 to 37 show device layouts (D) for file transmission. Figures 38 to 47 show various examples.

[0109] <Overview and terminology of Data Collection System 10>

[0110] The data collection system 10 is a system for collecting data related to the implementation of medical care, nursing care, care for the elderly, assistance, or health care, or similar care. The terms used in the description of this data collection system 10 have the following meanings.

[0111] (Scope of care covered by Data Collection System 10)

[0112] "Care" includes medical care, nursing, caregiving, assistance, or health care, as well as similar care. Examples of similar care include care for pets such as dogs and cats by pet sitters, care for babies by babysitters, health education care in which parents, caregivers, school teachers, professionals, and other related parties work together to exchange and share information about children, care for animals, plants, fish, birds, etc., provided through communication between zookeepers and facility managers before and after relocation (e.g., animals and birds from a zoo, fish from an aquarium, or plants from a botanical garden), care when new animals, plants, fish, birds, etc. are added to such facilities, and physical care for athletes, provided through communication between related parties before and after transfers (e.g., from high school to college, or from college to a corporate team).

[0113] (Meanings of basic terms related to care)

[0114] "Type of care" refers to, for example, medical care, nursing, caregiving, assistance, health care, pet sitting care, babysitting care, health education care, care for raising animals, plants, birds, fish, etc., and physical care for athletes.

[0115] A "care item" is a type of care, and indicates, for example, toileting, bathing, suction, injections, intravenous drips, medication, walks, meals, snacks, tooth brushing, exercise, recreation, etc. These "care items" belong to a "care type," as different items exist or are prepared for each of the above "care types." However, care items with the same name (name) are often used in common across multiple care types; for example, the care item "toileting" exists in common across many care types.

[0116] A "care situation (care scene)" is a time period from the start to the end of an action or behavior by a caregiver toward a care recipient, and is a unit of care separated by time. Each "care situation" has the above-mentioned "care item (care type)" as an attribute, so different "care items" result in different "care situations." For example, if a care item called "breakfast" is performed between 8:00 and 8:50, and a care item called "tooth brushing" is performed between 8:50 and 9:00, the care is performed continuously from 8:00 to 9:00, but this is not one "care situation" but two consecutive "care situations."

[0117] A "care session" indicates multiple "care items" and the order in which they are scheduled to be performed by a caregiver in charge of a certain care recipient during a certain period (e.g., a period of any length, such as half a day in the morning, a day, three hours, or a week). Therefore, multiple "care situations" exist within a "care session," but different "care items" do not need to be performed in all "care situations" within a single "care session." For example, the same "care item" may be performed multiple times within a single "care session," such as toileting, bathing, toileting, snacks, etc. However, in this example, the first and second toileting visits are the same "care item," but are different "care situations" on the timeline, and therefore are assigned different care situation IDs. Furthermore, a "care session" may specify not only the order in which the "care items" are performed, but also their start times (care scene start times), in which case, in the above example, the start time of the first toilet care scene, the start time of the bathing care scene, the start time of the second toilet care scene, the start time of the snack care scene, etc. It is not incorrect to say that the care scene start times are the start times of the care items, but because the same "care item" may be repeated multiple times, as in the above example, it is more accurate to think of them as the start times of the care scenes, which are units of care separated by time.

[0118] A "care theme" is a topic or subject that indicates the content of care to address the cause of a care recipient's hospitalization or institutionalization, such as "Nursing Tokkyo Goro's Gastric Ulcer." Each care theme is assigned a case ID. For example, if the same care recipient was hospitalized last year for a broken arm and this year for a gastric ulcer, they are assigned different care themes and therefore different case IDs. In this case, if the care recipient is the same, different care themes can be linked by the care recipient's identification information (in this embodiment, customer ID; see FIG. 15 ). Furthermore, if the care recipient is receiving or has received care at multiple facilities, businesses, etc., and the customer ID is a customer ID within the facility, business, etc. (in this embodiment, facilities, businesses, etc. are identified by care location ID; see FIG. 12 ), the care recipient can be linked by the My Number or, in the case of a pet, by microchip identification information (see FIG. 15 ). The customer ID may be identification information that can identify both the facility, business, etc. and the individual. Furthermore, in the event of a recurrence of an illness, the facility, business, or care coordinator can choose whether to continue the previous case ID and care theme, or to assign a new case ID to a different care theme, such as "Nursing Tokkyo Saburo's illness (recurrence)." If the case ID and care theme are continued and the related party placement has changed (for example, if the doctor or nurse in charge has changed), the settings of related party placement storage means, related party placement table 33B (see Figures 7 to 9), will be changed.

[0119] "Primary care" and "secondary care" are the main and secondary care provided for the same care recipient, respectively. The allocation of personnel (e.g., personnel allocation) for "primary care" and "secondary care" is linked by the same case ID. Therefore, "primary care" and "secondary care" share the same "care theme." Furthermore, "primary care" and "secondary care" may each have "care sessions," "care items," and "care situations." While "primary care" exists for all "care types," "secondary care" does not necessarily have to be set for some "care types." Examples of "secondary care" include model care, supplementary care with consent confirmation, temporary home care, and educational care. However, model care, supplementary care with consent confirmation, etc., which are typically registered as "secondary care," may be registered as "primary care" if the primary purpose of collecting videos and their annotation data is to share or exchange information about the care itself (or, alternatively, they may be registered without distinction between primary and secondary care). Therefore, what constitutes "primary care" and what constitutes "secondary care" can be freely determined by the facility, business, etc., or the care coordinator, depending on the purpose of data collection, and they can also freely decide whether to have two levels of participants, "primary care" and "secondary care," or a single level of participants. Therefore, care described as "secondary care" in the claims of this application can also be care with a single level of participants, without any distinction between "primary care" and "secondary care."

[0120] A "care location" is a care site where a caregiver provides care to a care recipient, such as a hospital, welfare facility, nursing care facility, short-stay nursing home, pet hotel, home, assisted living facility (serviced housing for the elderly), school, etc. In the description of this application, these may be represented by "facility" or "facility, etc." In addition, when online care (online medical consultation, etc.) is provided, a site connecting the location of the caregiver and the location of the care recipient, or a screen of a device such as a personal computer on which the caregiver and care recipient are displayed, may be considered the "care location." In the case of online care, if the care recipient's appearance on the screen is recorded (photographed or audio recorded), this becomes video data, still image data, and audio data, which are information transmitted by the caregiver.

[0121] (Scope of those involved in care: care recipient, care provider, care coordinator, and care assessor)

[0122] "Care recipients" include, for example, humans such as disabled children, infants, sick people, and elderly people (primarily weak people who require care from others, but also strong people who wish to improve their physical functions, such as athletes, who wish to become stronger), pets kept in homes or stores, and animals, plants, fish, and birds kept in facilities such as zoos, botanical gardens, and aquariums. "Care recipients" also include inanimate objects (dolls, robots) that play the role of these humans or living creatures. Furthermore, "care recipients" also include able-bodied individuals who pose as weak individuals in order to train caregivers. Note that when it is difficult to distinguish between individual creatures, or when there is no point in distinguishing between them, such as multiple tropical fish kept in an aquarium or multiple small birds kept in a birdcage, the term "care recipient" refers to a group of creatures rather than each individual creature.

[0123] "Care providers" include, for example, medical professionals (nurses, doctors, medical equipment technicians, etc.), childcare workers, caregivers (helpers, care workers, etc.), sitters (babysitters, pet sitters, etc.), zookeepers at zoos, botanical gardens, aquariums, etc., and robots that fulfill these roles. Furthermore, when creating video data or still image data for model reference in secondary care, there are cases where those who could serve as care evaluators for primary care, such as guardians such as parents of disabled children or infants, or pet owners, become "care providers" for secondary care (model care) and create video data, etc.

[0124] "Care coordinators" include supervisors and managers of facilities, businesses, etc., and healthcare professionals (including the healthcare professionals themselves and their superiors, if the healthcare professionals who provide care are involved in creating the coordination information after the care provider has provided care). Note that as the "coordination means" creates more and more automated processes, the human work of "care coordinators" will decrease or even disappear. In this case, the system will replace almost all of the coordination work of the human "care coordinators." However, no matter how much the "coordination means" automates the process, the "care coordinators" will still need to confirm and make selections. Therefore, the "care coordinators" are registered in the participant allocation table 33B (see Figures 7 to 9), which serves as the participant allocation storage means. Note that the annotation data automatically added to video data, etc. by the system is also referred to as "coordinator annotation data" because it replaces the work of the "care coordinator."

[0125] "Care evaluators" include parents and other guardians of disabled children and infants, pet owners, guardians, guardianship supervisors, the person receiving care who understands the person best, and professionals such as doctors.

[0126] "Other related parties" include those who input data, those who reference data, and those who fall into both categories, and all of these related parties are registered in related party allocation table 33B (see Figures 7 to 9), which is related party allocation storage means. For example, a relative, friend, acquaintance, etc. who visits a facility, etc. may be given the authority to register video data, etc. showing the state of the care recipient, or a barber who frequents a facility, etc. may be given the authority to reference video data, etc., so that he or she can view video data, etc., showing the care recipient's unhappy behavior when the care recipient's ear is touched. Furthermore, in the claims of this application, the term "other related parties with reference authority" refers to those who at least have reference authority, and does not exclude those who have input authority.

[0127] (Identifying information of care participants)

[0128] In this embodiment, the identification information of the care recipient, care provider, care coordinator, care evaluator, and other related parties is referred to as an "ID." This ID is identification information in a broad sense, and may be a combination of not only numbers and symbols but also characters (various characters such as the alphabet, hiragana, katakana, kanji, and Greek letters). The same applies to identification information of other information (e.g., case, care location, care session, care situation, care item, etc.), and in this embodiment, it is referred to as an "ID" (e.g., case ID, care location ID, care session ID, care situation ID, care item ID, etc.).

[0129] In this embodiment, care participants are divided into customers and service providers, and identification information (ID) is assigned to each of them, as shown in the customer table 33L and the service provider table 33M in Fig. 15. Note that the identification information for participants is not limited to this division, and in short, it is sufficient to be able to identify each individual, each animal, or each bird (as mentioned above, in the case of living creatures, there may be one group per group).

[0130] Specifically, in this embodiment, the care recipient is identified by a client ID, and the care provider and care coordinator are generally identified by a service provider ID, although in the case of secondary care, they may be identified by a secondary client ID (an ID associated with the client ID). If the care evaluator is the care recipient's guardian (mother, father), sibling, etc., they are identified by the secondary client ID, and if they are a doctor, nurse, etc., they are identified by the service provider ID. Furthermore, other related parties, such as a barber or care taxi driver who frequents the facility, are identified by a service provider ID; if they are a friend or acquaintance who visits, they are identified by a secondary client ID; and if a photography collaborator is requested by the facility, they are identified by a service provider ID, and if requested by the care recipient, they are identified by a secondary client ID.

[0131] More specifically, in this embodiment, if the customer ID of the patient who is the recipient of care is, for example, 20230001-00 (however, the 00 is optional), the customer auxiliary IDs assigned to the patient's guardians (mother, father) and siblings, etc. will be branch numbers that visually indicate a link to the patient's customer ID (= 20230001-00, however, the 00 is optional), such as 20230001-01 (mother), 20230001-02 (father), 20230001-03 (older brother), 20230001-04 (younger brother), etc. If a patient's guardian (mother, father) or siblings become ill and become a patient themselves, they will be assigned a separate customer ID (for example, 20230008-00, 20230009-00, 20230010-00, 20230011-00, etc., although the 00 is optional).

[0132] In the present invention, the existence of a "primary caregiver" and a "secondary caregiver" results in a two-stage participant arrangement (see FIGS. 7 to 9), and the same person may be assigned with different roles and positions. For example, a patient's guardian (such as the patient's mother) may serve as both the care evaluator for the primary caregiver and the care provider for the secondary caregiver. In such a case, in this embodiment, the auxiliary customer ID assigned to the guardian (such as the mother) is used for setting and registering the role and position of the care provider in either of the two stages of participant arrangement. For this reason, in this embodiment, the care provider is not always considered to be in the role of the service provider and the service provider ID is not used when setting and registering the care provider. In some cases, the role of the care provider may be played by a client (such as the mother), and in such cases, the auxiliary customer ID is used for setting and registering the participant arrangement.

[0133] If the mother or father of a patient (child) currently hospitalized at a hospital has been employed as a nurse or doctor at the hospital, or has been a nurse or doctor at the hospital from the beginning, the mother or father will also be assigned a service provider ID. Therefore, if they are involved in the care of their own child, they will log in to the system using their customer ID, and if they are involved in the care of other patients as a professional, they will log in to the system using their service provider ID.

[0134] In addition, when the patient's (child's) mother or father becomes ill, the mother or father is assigned a customer ID. Therefore, when the mother or father is caring for the sick child, they log in to the system using the customer ID, and when they want to refer to their own care-related information, they log in to the system using their own customer ID.

[0135] Furthermore, if a child who was a patient leaves the hospital, grows up, and later becomes involved in caring for his or her mother or father, the child will be assigned a supplementary customer ID.When the child logs into this system, they will not use the customer ID that was previously assigned to them (the customer ID that the child had when he or she was a patient), but will use the supplementary customer ID to become involved in caring for their mother or father, who has now become the recipient of care.

[0136] In the case of self-care, the person performing the self-care (e.g., patient) will be both the care provider and the care recipient, but regardless of their role or position, the customer ID of the person performing the self-care will be used when allocating relevant parties or logging into this system.

[0137] Furthermore, if the same person works at different facilities, they will be assigned different service provider IDs. Similarly, if the same person or the same living thing (such as a pet) receives services at different facilities, they will be assigned different customer IDs. While linking these is not essential, in this embodiment, they are linked by My Number (for people) or microchip identification number (for pets, etc.) (see FIG. 15).

[0138] In addition, a "care provider" may, after some time has passed, become a "care coordinator" and coordinate data. In this case, the care provider logs in using the care coordinator's identification information. In this embodiment, if the care provider and the care coordinator are the same person, there is no distinction between the care provider's identification information and the care coordinator's identification information; both are the person's service provider ID, so there is no need to perform processing such as single sign-on.

[0139] For convenience, the doll is also assigned the customer ID of the person being cared for. Different customer IDs may be assigned to different types of dolls (e.g., IDs indicating the type of doll, such as nursing doll, care doll, Little Ann doll, etc.), or the same customer ID may be assigned to all types of dolls.

[0140] (Location of equipment used in collecting data on care delivery)

[0141] The data collection format and the equipment used vary depending on the care location, such as a facility or business (e.g., a hospital, a nursing care facility, a pet hotel, a home, a school, etc.). The data collected by the data collection system 10 includes video data, still image data, and audio data recorded (photographed or recorded) by the care provider, the care recipient, or their input agent, and the arrangement of the equipment used and the flow of the data collection process vary depending on the recording format. Note that in the description of this application, when referring to a conceptual care location rather than a specific care location among various care locations such as a hospital, a nursing care facility, a pet hotel, a home, a school, etc., the terms "facility" or "facility, etc." may be used.

[0142] In this embodiment, there are provided equipment arrangements (A) for continuous recording (24 hours, or with on / off) shown in Figures 20 to 29, equipment arrangements (B) for recording the entire care scene shown in Figures 30 and 31, equipment arrangements (C) for recording limited to a specific time point shown in Figures 32 to 34, and equipment arrangements (D) for file transmission shown in Figures 35 to 37.

[0143] Here, the time of interest is the time when the care provider, the care recipient, or their input agent (such as the care recipient's mother or other guardian, or a relative, friend, or acquaintance who has come to visit) wants to record (photograph or audio) the state of the care recipient and share it with related parties, or contact related parties. However, in this application, if the recorded data is still image data, the time of interest means one time point, but if it is video data or audio data, the time of interest refers to a set of such desired times (time period).

[0144] (Device layout for continuous recording (24 hours or with on / off): Figures 20 to 29)

[0145] The device layout (A) for continuous recording (24 hours a day, or with on / off) shown in Figures 20 to 29 is the device layout when recording (photography and audio) is always performed with real-time transmission, regardless of whether a care scene is occurring or whether a point of interest has occurred. Continuous recording here includes 24-hour recording 365 days a year, and recording with cameras and microphones turned on and off.

[0146] The latter case of on / off refers to cases where recording (photography, audio recording) with a camera or microphone is not performed, for example, at night, on holidays, or when no one is present. Such cases are also considered to be continuous recording in the present application. Therefore, continuous recording in the present application does not only mean 24-hour recording (photography, audio recording), but also means recording (photography, audio recording) that is not limited to the care scene (when care is being provided) and is not limited to a specific time point. In the case of continuous recording with on / off, on / off control may be automatic (for example, controlled by time or date, or controlled by the presence or absence of people, etc.) or manual (for example, when a facility manager, security guard, etc. performs the switching operation). The difference from device layout (B) described below is that in device layout (A), the care provider does not perform the on / off operation at the care delivery location (where care is provided) at their own will or discretion, but rather the on / off operation is performed by a facility manager, security guard, etc. It should be noted that the equipment that is turned on / off in this equipment arrangement (A) is solely for recording (photographing, audio recording) the condition of the person receiving care, so even if that equipment is turned off, it does not necessarily mean that recording equipment for other purposes in the facility (for example, infrared cameras for night security) will also be turned off.

[0147] In this equipment layout (A), as will be described in detail later, images can be taken using fixed cameras 50 installed at fixed positions within the facility, cameras mounted on robots that circulate within the facility, or mobile cameras 51 such as cameras that circulate along guide rails within the facility (see FIG. 2). However, these fixed cameras 50 and mobile cameras 51 cannot be turned on / off frequently by caregivers; rather, they are fixed cameras 50A and mobile cameras 51A that are turned on / off by facility managers or security guards (see FIG. 20). This differs from the fixed cameras 50B and mobile cameras 51B (see FIG. 30) used in the next equipment layout (B). In addition, in equipment layout (A), recording (photography and audio) can also be performed using a web camera 52A or a microphone 52B connected to a personal computer (PC) 52 installed within the facility (see FIGS. 2 and 20).

[0148] In the case of equipment arrangement (A), when a care provider provides care to a care recipient in a state or position that is captured by a camera that is constantly recording, the care provider or the care recipient will be deemed to be recording the care of their own volition, regardless of who is responsible for installing or managing the camera.

[0149] (Device layout for recording the entire care scene (B): Figure 30, Figure 31)

[0150] The device arrangement (B) for recording the entire care scene shown in Figures 30 and 31 is the device arrangement when recording (photographing and audio recording) with real-time transmission is started by the caregiver's operation at the start of or before the start of a care scene, regardless of whether or not a point of interest occurs (when the point of interest has not yet arrived). Therefore, unlike the device arrangement (A) for continuous recording (24 hours a day, or with on / off), device arrangement (B) is the device arrangement when recording (photographing and audio recording) only the care scene (when care is being provided). However, in the strict sense, it is not necessary to record only the care scene; in short, the idea is to record for the time being because care is being provided, even though it is unclear whether a point of interest will occur.

[0151] In this equipment layout (B), as will be described in detail later, images can be captured using fixed cameras 50 installed at fixed locations within the facility and mobile cameras 51, such as cameras attached to robots that orbit the facility and receive on-site human commands (voice or button operation) to start capturing images (see Figure 2). However, these fixed cameras 50 and mobile cameras 51 are fixed cameras 50B and 51B that can be frequently turned on and off by caregivers (see Figure 30), and are not turned on and off by facility managers or security guards. This differs from the fixed cameras 50A and mobile cameras 51A used in the above-mentioned equipment layout (A) (see Figure 20). Furthermore, in equipment layout (B), recording (photography and audio) can also be performed using web cameras 52A and microphones 52B connected to personal computers (PCs) 52 installed within the facility, as long as they can be frequently turned on and off by caregivers (see Figures 2 and 30).

[0152] (Device layout for time-of-interest limited recording (C): Figures 32 to 34)

[0153] The device configuration (C) for recording limited to a time point of interest shown in Figures 32 to 34 is a device configuration when recording (photographing and audio recording) is performed in real time transmission limited to a time point of interest by operation of the care provider, the care recipient, or their input representative. The operation to start recording (photographing and audio recording) is performed after recognizing the occurrence of the time point of interest, and when the time point of interest ends, an operation to end recording (photographing and audio recording) is usually performed unless forgotten. This is because the premise is that only the time point of interest is recorded, so automatic clipping processing by the adjustment means 23 is not performed. If the termination operation is forgotten and unnecessary portions are recorded, the care coordinator can manually clip out the necessary portions (remove the unnecessary portions).

[0154] In this equipment layout (C), as with the above-described equipment layout (B), images can be taken using a fixed camera 50B and a mobile camera 51B that can respond to frequent on / off operations by the caregiver (see Fig. 32). Also, in equipment layout (C), if a web camera 52A and a microphone 52B connected to a personal computer (PC) 52 installed in the facility can respond to frequent on / off operations by the caregiver, recording (photography and audio) can be performed using these devices (see Figs. 2 and 32).

[0155] Furthermore, in device arrangement (C), only the point of interest is recorded (photographed and audio recorded), so the recording does not last very long. Therefore, recording (photographed and audio recorded) may be performed using a camera serving as the photographing means 41 and a microphone serving as the audio recording means 42 of a portable device 53 such as a smartphone or tablet operated by the care provider, or a camera serving as the photographing means 41 and a microphone serving as the audio recording means 42 of a portable device 54 such as a smartphone or tablet operated by the care recipient or their input representative (see Figures 32 and 2). Recording (photographed and audio recorded) may also be performed using a camera (not shown) serving as the photographing means 41 and a microphone serving as the audio recording means 42 of an input operation device 55 (see Figures 32 and 2).

[0156] (Device layout for file transmission (D): Figures 35 to 37)

[0157] The device arrangement (D) for file transmission shown in Figures 35 to 37 is a device arrangement when a video file, still image file, or audio file created by recording (photographing, audio recording) a time point of interest through the operation of a care provider, or the care recipient, or their input representative, is transmitted after the file is created. Therefore, as with the device arrangement (C) described above, the data in the file is data that records only the time point of interest, and it is assumed that no extra data is recorded. If a lot of extra data were recorded in the file, others (related parties) might not be able to recognize which part of the file is the time point of interest, so the person who recorded (photographed, audio recorded) should remove the extra parts from the file before transmitting it to the care event information collection means 21.

[0158] In this device arrangement (D), as in the case of the device arrangement (C) described above, only the point of interest is recorded (photographed and audio recorded), so the recording does not last very long. Therefore, recording (photographing and audio recorded) may be performed using a camera serving as the imaging means 41 and a microphone serving as the audio recording means 42 of a portable device 53 such as a smartphone or tablet operated by the care provider, or a camera serving as the imaging means 41 and a microphone serving as the audio recording means 42 of a portable device 54 such as a smartphone or tablet operated by the care recipient or their input representative (see Figures 35 and 2).

[0159] In addition, in equipment layout (D), if a webcam 52A or microphone 52B connected to or built into a personal computer (PC) 52 installed in the facility can be used to record (photograph and audio) if the webcam 52A or microphone 52B can be turned on / off frequently by the care provider (see Figure 35 and Figure 2).

[0160] (Data and adjustment information collected by data collection system 10: Figures 3 to 5)

[0161] As shown in Figures 3 to 5, the data collection system 10 collects care-related information, which includes care event information that occurs at the care location where care is provided (including when the care provider is not present), and additional information obtained thereafter.

[0162] In Figure 3, care event information includes provider-sent information (see Figure 4 for details) that is generated as a result of the care provider's actions and behaviors related to care, and recipient-sent information (see Figure 5 for details) that is generated based on a request from the care recipient or their input agent.

[0163] On the other hand, the additional information obtained thereafter includes, for video data, still image data, and audio data recorded (photographed or audio) by the care provider, the care recipient, or their input agent, coordinator annotation data (included in the coordination information transmitted to the care evaluator) assigned by the care coordinator to request an evaluation from the care evaluator, coordinator annotation data (non-coordination information) assigned by the care coordinator to respond to the care provider, coordinator annotation data (non-coordination information) assigned by the care coordinator to respond to the care recipient, evaluator annotation data (including the evaluation results) assigned by the care evaluator, and additional annotation data. Note that because coordination information is the coordination information referred to in the claims of this application that is transmitted from the care coordinator to the care evaluator, all coordinator annotation data assigned by the care coordinator other than the coordinator annotation data for requesting an evaluation from the care evaluator is non-coordination information.

[0164] Additional annotation data includes annotation data added by the care provider, annotation data added by the care recipient, annotation data (non-coordinated information) added by the care coordinator (from the second time onwards), and annotation data added by the care evaluator (from the second time onwards).

[0165] In addition, care-related information does not include annotation data attached to specific video data, still image data, or audio data, but also includes comments on the care as a whole (comments from the care provider, care recipient, care coordinator, and care evaluator).

[0166] Furthermore, the care-related information includes sensor data output from various sensors that measure the biological information of the care recipient. This sensor data is continuously streamed (packetized and transmitted) to the data collection server 20, and thus, like video data and audio data, record fragments are automatically extracted and created in the data collection server 20. These record fragments may also serve as adjusted sensor data (adjustment information) for an evaluation request from the care coordinator to the care evaluator. Furthermore, when the care coordinator creates a new record fragment (second record fragment) by further extracting this record fragment (first record fragment), the second record fragment may also serve as adjusted sensor data (adjustment information) for an evaluation request. Furthermore, when performing section detection processing using sensor data (section detection processing for extracting key parts at a time of interest from video data or audio data) (see Figure 19), the results of the section detection processing used to extract the key parts (whether or not an abnormal value was detected in the magnitude judgment processing using a threshold, the type of sensor in which the abnormal value was detected (sensor type ID), and in addition to these, the duration and time period of the sensor data change section, etc.) may also serve as coordination information for the care coordinator's evaluation request to the care evaluator.

[0167] In Figure 4, the "provider-sent information" that constitutes the care event information includes provider behavior information (start care item information, care scene start time, care scene end time), video data (video data including audio data or video data not including audio data) showing the state of the care recipient that the care provider wants the care provider to pay attention to, still image data, and audio data (audio data recorded independently, not audio data within the video data), and provider annotation data added to these data by the care provider.

[0168] Video data and audio data showing the state of the care recipient may be continuously transmitted to the data collection server 20 via streaming (packetized and transmitted) or may be transmitted as a file. When the data collection server 20 receives streaming data, the data collection server 20 automatically creates a record fragment, which may serve as coordinator annotation data (coordination information) for the care coordinator's evaluation request to the care evaluator. When the care coordinator creates a new record fragment (second record fragment) by further extracting the record fragment (first record fragment) or performs section detection processing (section detection processing for extracting key parts at a time point of interest from video data or audio data) to create a record fragment (second record fragment) (see FIG. 19 ), the second record fragment may also serve as coordinator annotation data (coordination information) for the evaluation request. When the data collection server 20 receives a file, the file may be the original file, or a file containing video data or audio data that has been adjusted and processed by the data collection server 20 may serve as the coordination information for the care coordinator's evaluation request to the care evaluator.

[0169] The provider annotation data also includes the care provider's evaluation request intention, timing information for specifying the point of interest by the care provider in the equipment arrangement (A) of Figure 20 and the equipment arrangement (B) of Figure 30 (the time of sending the timing information for specifying the point of interest, the elapsed time from the base point of the point of interest specified by the care provider to the time of sending, and the duration of the point of interest specified by the care provider), the start time and end time of the point of interest automatically specified by the care provider's start and end operations for recording (photographing, audio recording) in the equipment arrangement (C) of Figure 32, and meta information including the recording date and time (photographing date and time, audio recording date and time) attached to the file created by the care provider through recording (photographing, audio recording) in the equipment arrangement (D) of Figure 35.

[0170] The form in which the care provider's evaluation request intention is transmitted may be text data (text data entered by the care provider using keyboard input), a selection signal for the evaluation request intention (information indicating the evaluation request intention selected by the care provider from multiple evaluation request intention options) or a single fixed signal indicating the evaluation request intention, a gesture included in video data showing the care recipient's appearance, audio data included in video data showing the care recipient's appearance, a combination of gestures and audio data included in video data showing the care recipient's appearance, audio data indicating the evaluation request intention included in audio data showing the care recipient's appearance (audio data recorded independently, not audio data within the video data), or audio data indicating the evaluation request intention (audio data from a different audio source than the video data and audio data showing the care recipient's appearance, i.e., an audio source prepared not for recording the care recipient's appearance but for inputting the care provider's evaluation request intention).

[0171] The single fixed signal indicating an evaluation request intention is a signal (information) when the input means 46 is configured to only be able to transmit, for example, "pay attention here (look here, listen to this)." Therefore, this signal is intended to request attention simply by transmitting video data, still image data, or audio data showing the state of the care recipient, or by transmitting timing information for specifying the point of interest to request extraction. In other words, this signal means that it has not simply been transmitted, and can be considered an identification signal for distinguishing video data, etc. showing the state of the care recipient from other data transmitted to the data collection server 20 (for example, facial photograph data at the time of admission, scanned data of health insurance card, etc.).

[0172] Furthermore, when the care provider's evaluation request intention is received as gesture or voice data, the data collection server 20 converts it into text data or evaluation request intention identification information, as will be described in detail later.

[0173] In Figure 5, the "subject transmission information" that constitutes the care event information includes video data (video data including audio data or video data not including audio data) showing the appearance of the care recipient that the care recipient or their input agent wants the care recipient to pay attention to, still image data, and audio data (not audio data within the video data, but audio data recorded independently), as well as subject annotation data added to these data by the care recipient or their input agent, and subject needs information.

[0174] The video and audio data showing the condition of the care recipient is the same as in the case of the provider-sent information in Figure 4, with the only difference being that the person who recorded the data is the care provider in Figure 4, and the person who recorded the data is the care recipient or their input agent in Figure 5.

[0175] The subject annotation data also includes the evaluation request intention of the care recipient or their input agent, timing information for specifying the point of interest by the care recipient or their input agent in the equipment arrangement (A) of Figure 20 and the equipment arrangement (B) of Figure 30 (the transmission time of the timing information for specifying the point of interest, the elapsed time from the base point of the point of interest specified by the care recipient or their input agent to the transmission time, and the duration of the point of interest specified by the care recipient or their input agent), the start time and end time of the point of interest automatically specified by the start and end operations of recording (photographing, audio recording) by the care recipient or their input agent in the equipment arrangement (C) of Figure 32, and meta information including the recording date and time (photographing date and time, audio recording date and time) attached to the file created by the care recipient or their input agent through recording (photographing, audio recording) in the equipment arrangement (D) of Figure 35.

[0176] The manner in which the care recipient or their input agent sends their evaluation request intention is the same as the manner in which the care provider sends their evaluation request intention in Figure 4, with the only difference being that the person who inputs the evaluation request intention is the care provider in Figure 4 and the care recipient or their input agent in Figure 5.

[0177] The care recipient needs information includes the call time when the care recipient or their input agent pressed the button on the call device and the call intention when the button on the call device was pressed (such as a care item requested by the care recipient or their input agent). The call intention may be transmitted as a selection signal for the call intention by pressing a button (information indicating the call intention selected by the care recipient or their input agent from multiple call intention options), a single fixed signal indicating the call intention, or voice data indicating the call intention uttered simultaneously with or immediately after pressing the button (which is later converted into text data or call intention identification information). The meaning of the single fixed signal indicating the call intention is almost the same as that of the single fixed signal indicating the evaluation request intention, and is a signal (information) when the input means 46 is configured to only be able to communicate, for example, "Please come."

[0178] <Overall configuration of data collection system 10>

[0179] In Figure 1, the data collection system 10 performs various processes related to the collection of care-related information, including care event information that occurs at the care location where care is provided (including cases where the care provider is not present) and additional information obtained thereafter, and is equipped with a data collection server 20 that stores various data required for these processes.

[0180] In addition, the care location (facility, home, etc.) is equipped with devices connected via a network to the data collection server 20, including an imaging means 41, a recording means 42, and a sensor 43, which are status data acquisition means for acquiring the status data of the care recipient (appearance, audible state, biometric information), a facility recorder 44 and a connection device 45 permanently installed in the facility, an input means 46 for accepting various input operations by the care provider, or the care recipient or their input agent, and a display means 47 for the care provider, or the care recipient or their input agent to view various screens.

[0181] Furthermore, the data collection server 20 is connected via a network to a coordinator terminal 60 operated by a care coordinator, an evaluator terminal 70 operated by a care evaluator, an electronic medical record creation system 80, a care need management system 81 such as a nursing need management system or a care need management system, and a care provider call system 82 such as a nurse call system.

[0182] Here, the network connecting the data collection server 20 and various devices is an external network mainly composed of the Internet, but it can also be a combination of the Internet and an internal network such as a LAN or an intranet, and it does not matter whether it is wired or wireless, or even a combination of wired and wireless; in short, it is sufficient if it can transmit information at a certain speed between multiple points (regardless of the distance).

[0183] The data collection server 20 is composed of one or more computers and includes a care event information collection means 21, a care setting means 22, an adjustment means 23, an output means 24, an album registration means 25, an evaluation result collection means 26, a care-related information storage means 30, and an application storage means 35.

[0184] The output means 24 includes an album viewing means 24A, an electronic medical record creation data transmission means 24B, and a care necessity calculation data transmission means 24C.

[0185] The care-related information storage means 30 includes a temporary storage means 31 and an album storage means 32, and the album storage means 32 includes an album database 33 and a file storage means .

[0186] As shown in FIG. 6, the album database 33 is composed of a case table 33A, a related party allocation table 33B which is a related party allocation storage means, a care item master table 33C, a care session setting table 33D, a used device / used application setting table 33E which is a used device / used application setting storage means, a care history table 33F, a status data management table 33G, an evaluation / addition table 33H, a call history table 33K, a customer table 33L, a service provider table 33M, an authority table 33N, and a status change detection item table 33P which is a status change detection item storage means.

[0187] Of the above, the means 21 to 26 for executing various processes are realized by a central processing unit (CPU) provided inside the computer constituting the data collection server 20, one or more programs that define the operation procedures of this CPU, and working memories such as main memory and cache memory. Details of each of these means 21 to 26 will be described later.

[0188] Furthermore, non-volatile memories such as hard disk drives (HDDs) and solid state drives (SSDs) can be used as the care-related information storage means 30 and the application storage means 35. Furthermore, as a data storage format, at least a portion of the data stored in the album database 33 may be stored as files. Details of these storage means 30 (31 to 34) and 35 will be described later. Note that some of the data stored in the care-related information storage means 30 may be protected from data tampering by using distributed ledger technology (DLT) such as blockchain.

[0189] (Details of the photographing means 41, the recording means 42, and the sensor 43, which are the status data acquisition means)

[0190] The imaging means 41 is a means for capturing video or still image data of the state (visual condition) of the care recipient. Specifically, as shown in FIG. 2, this imaging means 41 may include, for example, a fixed camera 50 permanently installed at a fixed position within the facility and a mobile camera 51 that moves within the facility. Examples of the mobile camera 51 include a mobile camera attached to a robot (including a robot accompanying a caregiver) that moves around the facility, and a mobile camera that moves along a guide rail attached to the ceiling or the like within the facility. While these fixed cameras 50 and mobile cameras 51 are not typically installed in homes, they may be installed in, for example, a housing facility for the elderly with support services (senior housing) that can be considered a facility. The number of cameras is not limited to the fixed camera 50 and the mobile camera 51. One or more cameras may be installed per care recipient, or one camera may be installed per multiple care recipients.

[0191] These fixed cameras 50 and mobile cameras 51 are not designed to accommodate frequent on / off operations by caregivers, but include fixed cameras 50A and mobile cameras 51A that are turned on / off by facility managers, security guards, etc., and fixed cameras 50B and mobile cameras 51B that can accommodate frequent on / off operations by caregivers. The former fixed cameras 50A and mobile cameras 51A are used in the equipment layout (A) of Fig. 20, and the latter fixed cameras 50B and mobile cameras 51B are used in the equipment layout (B) of Fig. 30 and the equipment layout (C) of Fig. 32.

[0192] 2, the photographing means 41 may include a web camera 52A connected to or built into a personal computer (PC) 52 installed in the facility, a camera of a mobile device 53 such as a smartphone or tablet operated by the care provider, a camera of a mobile device 54 such as a smartphone or tablet operated by the care recipient or their input representative, and a camera (not shown) installed in an input operation device 55. Note that although input operation devices 55A, 55B, 55C, 55D, and 55E shown in FIGS. 23 to 27 are given as examples of input operation device 55, there may also be other input operation devices 55 equipped with cameras.

[0193] The recording means 42 is a means for recording the state of the care recipient (what it sounds like) and obtaining recorded data, and may record audio data included in video data or may record only audio data. Audio includes human vocalizations as well as the cries of animals such as pets.

[0194] Specifically, as shown in Figure 2, the recording means 42 include a microphone 52B connected to or built into a personal computer (PC) 52 installed within the facility, a microphone on a portable device 53 such as a smartphone or tablet operated by the care provider, a microphone on a portable device 54 such as a smartphone or tablet operated by the care recipient or their input representative, a microphone installed on an input operation device 55 (not shown in Figure 2), and a microphone 56C on a calling device 56 (see Figure 28).

[0195] In this embodiment, the microphones provided in the input operation device 55 include microphone 55A3 of dedicated input operation device 55A having one physical button as shown in FIG. 23, microphone 55B4 of dedicated input operation device 55B having two physical buttons as shown in FIG. 24, microphone 55C4 of input operation device 55C which is also used as a calling device (selectable type) as shown in FIG. 25, microphone 55D3 of input operation device 55D which is also used as a calling device (mode switching type) as shown in FIG. 26, and microphone 55E3 of input operation device 55E which is also used as a calling device (simultaneous type) as shown in FIG. 27.

[0196] The sensor 43 is a sensor that is equipped in medical equipment, nursing care facilities, etc. and measures biological information of the care recipient, and the sensor data output from this sensor 43 includes, for example, pulse rate (PR) data, heart rate (HR) data, electrocardiogram waveform data, electroencephalogram data, cerebral blood flow data, pulse wave data, etc. In addition, in the present application, biological information is understood in a broad sense, and therefore the sensor 43 also includes, for example, a perspiration sensor that measures the amount of perspiration, an incontinence sensor that measures the amount of incontinence (a leakage sensor, a urination notification sensor, a diaper sensor, a bedwetting sensor), an odor sensor that measures flatulence, bad breath, body odor, etc.

[0197] Furthermore, the sensor 43 includes not only various sensors that measure a single piece of biometric information (broadly defined) as described above, but also sensors that input sensor data of multiple types of biometric information output from multiple types of sensors, perform pattern recognition processing to identify whether or not there is an abnormality, and output the recognition result (whether or not there is an abnormality, or a likelihood indicating the probability of there being an abnormality).

[0198] (Details of the facility recorder 44 and connection device 45, which are equipment within the facility)

[0199] As shown in Fig. 2, the facility recorder 44 is a device that accumulates and records video data continuously captured by fixed cameras 50 and mobile cameras 51 installed within the facility, and is similar to a recorder for a so-called surveillance camera. Therefore, the recorded video data is stored for a certain period of time, such as three months, and then deleted starting with the oldest data. As shown in Fig. 2, the video data continuously captured by the fixed cameras 50 and mobile cameras 51 is sometimes transmitted via the facility recorder 44 to the care event information collection means 21 of the data collection server 20 via the network, and sometimes transmitted to the care event information collection means 21 via the network without passing through the facility recorder 44.

[0200] The connection device 45 is a device having the function of an edge device that transmits data to a network (for example, a function of converting communication protocols), and is configured by a computer. As shown in FIG. 2, the connection device 45 receives, via wired or wireless communication, video data output from the fixed camera 50 and the mobile camera 51 (including when it is transmitted via the facility recorder 44) and sensor data output from the sensor 43, and executes processing to transmit the received video data and sensor data, together with time information (date and time information including year, month, day, hour, minute, and second) and device identification information (identification information of the fixed camera 50, the mobile camera 51, and the sensor 43), via the network to the care event information collection means 21 of the data collection server 20. The time information may be provided by the connection device 45, or may be output from the fixed camera 50, the mobile camera 51, or the sensor 43.

[0201] As shown in FIG. 2, video data, still image data, and audio data may be transmitted to the care event information collection means 21 of the data collection server 20 via a network without going through the connection device 45. In such cases, a personal computer (PC) 52, a mobile device 53 such as a smartphone or tablet operated by the care provider, and a mobile device 54 such as a smartphone or tablet operated by the care recipient or their input agent execute a process of transmitting the video data, etc., together with time information (date and time information in the form of year, month, day, hour, minute, and second) and device identification information (identification information of the PC 52 and the mobile devices 53 and 54) to the care event information collection means 21 of the data collection server 20 via the network. However, in this application, the explanation will be given assuming that the imaging means 41 and the sound recording means 42 execute the transmission process to the network, including both cases where the transmission process to the network is executed without going through the connection device 45 and cases where the transmission process to the network is executed via the connection device 45. 1 and 2, connection device 45 is shown outside of photographing means 41 and sound recording means 42, but photographing means 41 and sound recording means 42 will be understood in a broad sense, and the functions of connection device 45 will be described as being included in photographing means 41 and sound recording means 42 (see FIGS. 20, 30, 32, and 35). The same is true for sensor data; while connection device 45 is shown outside sensor 43 in FIGS. 1 and 2, connection device 45 will be shown outside sensor 43, and the functions of connection device 45 will be described as being included in sensor 43 (see FIGS. 20, 30, 32, and 35).

[0202] (Input means 46 and display means 47 installed or brought to the care location)

[0203] The input means 46 is a means for inputting data from the care event information excluding video data, still image data, and audio data showing the state of the person being cared for, and accepts input by keystroke input, operation using a pointing device such as a mouse, operation using a touch panel, etc., and transmits the input data to the care event information collection means 21 of the data collection server 20 via the network.

[0204] The process of transmitting input data to the care event information collection means 21 of the data collection server 20 via the network is executed by a device that realizes the input means 46. As shown in FIG. 2 , the transmission process to the network is executed by a personal computer (PC) 52, a portable device 53 such as a smartphone or tablet operated by the care provider, a portable device 54 such as a smartphone or tablet operated by the care recipient or their input agent, an input operation device 55, and a calling device 56. In the present application, the input means 46 is broadly defined and described as including not only the function of accepting input (e.g., the functions of a mouse, keyboard, touch panel, etc.) but also the function of transmitting to the network. Therefore, as already detailed in the description of the connection device 45, the photographing means 41 and the sound recording means 42 are also broadly defined. Therefore, the PC 52, the portable devices 53 and 54, and the input operation device 55 constitute the input means 46 as well as the photographing means 41 and the sound recording means 42.

[0205] The data input by the input means 46 and transmitted to the care event information collection means 21 includes provider behavior information (start care item information, care scene start time, care scene end time), provider annotation data (including the care provider's evaluation request intention and timing information for the care provider to specify a point of interest), recipient annotation data (including the care recipient's or their input agent's evaluation request intention and timing information for the care recipient's or their input agent's specification of a point of interest), and recipient needs information (the call time when the care recipient or their input agent pressed the button on the call device, and the call intention when the care recipient or their input agent pressed the button on the call device). Furthermore, identification information (service provider ID, customer ID, or customer auxiliary ID) of the care provider who entered the data, or the care recipient or their input agent, is also transmitted to the care event information collection means 21 together with these data.

[0206] Furthermore, as shown in FIG. 1, the input means 46 requests the download of an application program to be used in the device that realizes the input means 46, and also receives and installs the application program related to the download request stored in the application storage means 35 that is transmitted from the care event information collection means 21 of the data collection server 20 via the network.

[0207] Specifically, as shown in Figure 2, the input means 46 include, for example, a personal computer (PC) 52 (having the function of accepting input via a keyboard, mouse, touch panel, etc. and the function of transmitting to a network) installed in a facility or at home, a portable device 53 such as a smartphone or tablet operated by the care provider (having the function of accepting input via a touch panel, etc. and the function of transmitting to a network), a portable device 54 such as a smartphone or tablet operated by the care recipient or their input representative (having the function of accepting input via a touch panel, etc. and the function of transmitting to a network), an input operation device 55 (having the function of accepting input via physical push buttons, etc. and the function of transmitting to a network), and a calling device 56 (having the function of calling the care provider and the function of transmitting to a network).

[0208] The display means 47 is realized by a device having, for example, a liquid crystal display, and is installed at the care site (facility, home, etc.) or carried around so that the care provider, the care recipient, or their input representative can refer to various screens related to the care-related information stored in the album storage means 32. In the present application, the display means 47 is understood in a broad sense, and will be described as having not only a screen display function (such as a liquid crystal display function) but also a function for receiving data for screen display via a network.

[0209] Specifically, as shown in Figure 2, the display means 47 may include, for example, a personal computer (PC) 52 (having a screen display function using an LCD display or the like and a receiving function from a network) installed in a facility or at home, a portable device 53 such as a smartphone or tablet (having a screen display function using an LCD display or the like and a receiving function from a network) operated by the care provider, and a portable device 54 such as a smartphone or tablet (having a screen display function using an LCD display or the like and a receiving function from a network) operated by the care recipient or their input representative.

[0210] (Devices of those involved)

[0211] The coordinator terminal 60 is a terminal device operated by the care coordinator, and is configured by a computer, and is provided with a display means such as a liquid crystal display, and an input means such as a keyboard, mouse, or touch panel.

[0212] The evaluator terminal 70 is a terminal device operated by the care evaluator, and is configured by a computer, and is provided with a display means such as a liquid crystal display, and an input means such as a keyboard, mouse, or touch panel.

[0213] The adjuster terminal 60 and the evaluator terminal 70 may be, for example, a mobile device such as a smartphone or a tablet.

[0214] (System linked to data collection system 10)

[0215] The electronic medical record creation system 80 is a system composed of a computer, which receives data for creating electronic medical records transmitted from the electronic medical record creation data transmission means 24B of the output means 24 via a network (a network within the facility or an external network such as the Internet) and creates electronic medical records.

[0216] The care need management system 81 is a computer-based system that receives care need calculation data transmitted from the care need calculation data transmission means 24C of the output means 24 via a network (a network within the facility or an external network such as the Internet) and calculates and manages care needs (nursing need, care need, etc.). Specifically, it is, for example, a nursing need management system or a care need management system, and uses care provider behavior information (start care item information, care scene start time, care scene end time) of the care provider and care recipient needs information (call time when the button on the call device is pressed and call intention) of the care recipient to calculate the care need for one person or one animal (or a group of animals, in the case of living creatures), whether the quantity and quality (content) of care provided by the care provider is excessive or insufficient, and whether there are enough or insufficient care providers in the facility, and uses this information to adjust care.

[0217] For example, in the case of nursing, the quantity and quality (content) of care provided by nurses varies depending on the condition and severity of the patient's illness, so the hospital needs to allocate the appropriate number of staff (nurses) to each ward to provide the appropriate care required by the patient. For this reason, the nursing need management system, which is the care need management system 81, performs calculations to realize the appropriate allocation of nurses according to the patient's condition and severity.

[0218] In addition, in the case of nursing care, the nursing care need management system 81 does not perform the processing to determine the level of nursing care or support required (processing carried out by local governments), but calculates the quantity and quality (content) of care for care recipients whose level of nursing care or support required has already been determined, and for care recipients who have not applied for nursing care or support, as well as the surplus or shortage of personnel such as helpers.

[0219] The care provider call system 82 is a computer-based system that allows the care recipient or their representative to call a care provider. Specifically, it is, for example, a nurse call system installed in a hospital so that hospitalized patients can call a nurse.

[0220] <Configuration of Data Collection Server 20 / Care Event Information Collection Means 21>

[0221] The care event information collection means 21 accepts input of care event information occurring at the care location, such as provider-sent information (see Figure 4) sent by the care provider in a care situation where the care provider provides care to the care recipient, and recipient-sent information (see Figure 5) sent by the care recipient or their input agent in a non-care situation, which is a time period other than the care situation, and performs processing to store the accepted care event information in the care-related information storage means 30.

[0222] The provider-sent information and recipient-sent information that make up the care event information include, as already detailed in (Data and adjustment information collected by data collection system 10: Figures 3 to 5) in the above-mentioned <Overview and terminology of data collection system 10>, video data, still image data, or audio data showing the state of the care recipient that has been recorded (photographed or audio recorded) by the care provider, or the care recipient or their input agent, as well as provider annotation data (including the care provider's intention to request an evaluation) or recipient annotation data (including the care recipient's intention to request an evaluation or their input agent) that is added to these data.

[0223] 2, when video data or audio data showing the state of the care recipient is photographed by the photographing means 41 or recorded by the recording means 42 and streamed over the network together with a care location ID (a care location ID may not be necessary if the data collection system 10 is for each facility) and device identification information for each photographing device and recording device, the care event information collection means 21 receives the streaming data and stores it in the temporary storage means 31 in association with the care location ID and device identification information. On the other hand, when video data, still image data, or audio data showing the state of the care recipient is photographed by the photographing means 41 or recorded by the recording means 42 and stored in files and transmitted over the network (see FIGS. 35 to 37), the care event information collection means 21 receives the files and stores them in the file storage means 34.

[0224] In addition, as shown in Figure 2, when provider annotation data or recipient annotation data is input by input means 46 and transmitted via the network together with the care provider's ID and the care recipient's ID, or together with the ID of the care recipient or their input agent, the care event information collection means 21 receives the annotation data and stores it in the album database 33.

[0225] At this time, the care event information collecting means 21 also receives timing information for designating a time point of interest that is assigned by the care provider as provider annotation data or assigned by the care recipient or their input agent as subject annotation data. When the video data included in the provider-sent information or the subject-sent information is video data that has been continuously recorded, including a time point of interest indicating a time period when the care provider, the care recipient, or their input agent has determined that attention is required for the condition of the care recipient, as well as time periods other than this time point of interest (in the case of continuous recording as in FIGS. 20 and 21 or in the case of recording the entire care scene as in FIGS. 30 and 31), the adjusting means 23 performs a process of extracting video data for the time period of the time point of interest from the video data to create a record fragment, and the timing information for designating a time point of interest is information for determining the extraction start position (time) and the extraction end position (time).

[0226] In addition, an input agent for the care recipient (for example, a mother, friend, or acquaintance visiting the care recipient) may determine that attention is required regarding the care recipient's condition during times outside of care when the care provider is not present, and inputs timing information for specifying the time period of attention to specify the time period of attention. Furthermore, if the care recipient wishes to check the condition of a body part when the care provider is not present, and is able to operate the input means 46 by himself (if he is old enough and physically able to operate it), he or she may input timing information for specifying the time period of attention to specify the time period of attention. This is because the care recipient can recognize the condition of a body part such as an arm or leg with his or her own eyes, and can recognize the condition of a body part such as the face by looking in a mirror.

[0227] Specifically, the timing information for specifying a time point of interest is the elapsed time from the time point of interest to the time point at which the care provider, the care recipient, or their input representative specifies the timing information, and the duration of the time point of interest (see FIG. 22). The specified elapsed time is the time that the care provider, the care recipient, or their input representative intuitively remembers as to how long ago it was (for example, about one minute ago), or the time that the care provider, the care recipient, or their input representative remembers by checking the clock (such as what time it was today). On the other hand, the duration of the specified time point of interest is the duration for which the condition of the care recipient that is determined to require attention (the condition of the care recipient with whom information is to be shared) continued.

[0228] Furthermore, as shown in Figure 2, when sensor data output from the sensor 43 is sequentially transmitted via the network together with a care location ID (if the data collection system 10 is for each facility, the care location ID may not be necessary) and device identification information for each sensor device, the care event information collection means 21 receives the sensor data and stores it in the temporary storage means 31 in association with the care location ID and device identification information.

[0229] 2 and 4, when provider action information (start care item information, care scene start time, care scene end time) is transmitted via the network as provider-sent information constituting the care event information together with the care provider's ID and the ID of the care recipient or their input agent, the care event information collection means 21 receives the provider action information and stores it in the album database 33. Here, the start care item information is information indicating which care item, among the care items preset by the care setting means 22, the care provider will now start implementing. Note that while it is necessary to record the care scene start time, it is not essential to record the care scene end time.

[0230] 2 and 5, when recipient needs information (call time, call intention) is transmitted via the network together with the ID of the care recipient or their input agent as recipient transmission information constituting the care event information, the care event information collection means 21 receives the recipient needs information and stores it in the call history table 33K (see FIG. 6) of the album database 33. Here, the call time is the time when the care recipient or their input agent presses the button on the call device 56 for calling the care provider or on the input operation device 55 having the function of the call device 56, and the call intention is the reason for calling the care provider (what the care recipient wants done, what the care recipient wants to ask, etc.).

[0231] In this case, when a button on the call device 56 or an input operation device 55 having the function of this call device 56 is pressed, the care event information collection means 21 may receive the above-mentioned subject needs information (call time, call intention), and may also accept input of timing information for specifying a point of interest that is assigned by the care recipient or their input representative pressing a button on the call device 56 or input operation device 55.

[0232] Therefore, in this case, when the care recipient or their input representative presses a button on the call device 56 or input operation device 55, a care provider (nurse, helper, etc.) is called and information for cutting out video data showing the state of the care recipient (timing information for specifying a point of interest) is sent to the network simultaneously. The button pressing operation may be performed by the care recipient themselves (patient, care recipient, etc.) if possible, or by an input representative such as the mother visiting the care recipient. Because this is done simultaneously, the evaluation request intent assigned to the captured video data, etc. (for example, a mother, etc. saying, "You look like you're in pain. What's wrong?") and the call intent (for example, a mother, etc. saying, "You look like you're in pain, please come" etc.) will be approximately the same.

[0233] The care event information collection means 21 then receives, along with the provider-sent information and recipient-sent information, a case ID and / or the ID of the care recipient as information indicating which care recipient the data relates to. Furthermore, as information indicating who sent the data, it receives the ID of the care provider, or the ID of the care recipient or the ID of the person inputting the data on their behalf. These IDs (identification information) are either input when the input means 46 is operated, or are stored in the input means 46 (saved during previous use or upon installation), or are converted and acquired from device identification information (information identifying an individual device, not the type of device) for the device that implements the input means 46. When the input means 46 accepts data input on a screen displayed by an application program installed on the device or on a web screen, for example, on the care provider input screen (compatible with continuous image capture) 90 of Fig. 22, the care provider input screen (compatible with limited-time-of-interest image capture) 200 of Fig. 34, or the file registration screen 300 of Fig. 36, the ID (identification information) is already entered on a screen prior to these screens, or the ID (identification information) stored in the device is read, thereby specifying the case ID, the care recipient ID, and the care provider ID. Furthermore, in the case of an input operation device 55 or a call device 56 having a physical push button, if the care recipient to be used is predetermined, such as a device attached to a bed, the case ID and / or the care recipient ID may be stored in the device, or the case ID and / or the care recipient ID may be converted and obtained from the device identification information for each device (input operation device 55 or call device 56). The correspondence between the device identification information for each device and the case ID and / or care recipient ID can be stored in the album database 33 as a device / care recipient correspondence table (not shown), which is a device / care recipient correspondence storage means.

[0234] 11, when an application acquisition request is received from a device (input means 46) operated by the care provider, the care recipient, or their input agent, the care event information collecting means 21 also executes a process of delivering an application program stored in the application storage means 35 to the device (input means 46) operated by the care provider, the care recipient, or their input agent via the network, in accordance with the setting information of the device used and the application used stored in the device used / application used setting table 33E, which is a device used / application used setting storage means. As shown in FIG. 11, when a web screen display request is received from the device (input means 46) operated by the care provider, the care recipient, or their input agent, the care event information collecting means 21 transmits web screen display data including JavaScript (Java is a registered trademark) and the like via the network to the device (input means 46) operated by the care provider, the care recipient, or their input agent, instead of delivering the application program.

[0235] The application programs and web screens to be used vary depending on the type of device (input means 46) operated by the care provider, care recipient, or their input agent, the type of available image capture means 41 and sound recording means 42, and the usage situation. Therefore, setting information for the device and app in use (see FIG. 11 ) taking into account anticipated usage situations is pre-registered in the device and app settings table 33E. This narrows down the options presented by the care event information collection means 21 when the care provider, care recipient, or their input agent requests a download or web screen display, thereby reducing the effort required for these operations. The device (input means 46) operated by the care provider, care recipient, or their input agent, which has installed the downloaded and acquired application program, subsequently retrieves data required for input processing (such as data in the care session setting table 33D and the related party allocation table 33B) from the album database 33 via the care event information collection means 21 when the program is launched. In the case of a web screen, the data required for input processing is retrieved from the album database 33 when the screen is displayed.

[0236] Furthermore, when the care event information collection means 21 receives care event information (provider-sent information, subject-sent information) via the network from the condition data acquisition means 40 (photography means 41, recording means 42, sensor 43) or the input means 46, it creates a new record in the care history table 33F (see Figure 12) or the condition data management table 33G (see Figure 13).

[0237] More specifically, when the care event information collection means 21 receives from the input means 46, as care provider-sent information constituting the care event information, the start care item information and the care scene start time from the care provider behavior information, the care event information collection means 21 creates a new record in the care history table 33F (see FIG. 12 ). Accordingly, a care history table 33F is created for each care scene. Among the columns of this care history table 33F, the case ID, care session ID, start care item information (care item ID for the care item that actually started: see the selection section 91 in FIG. 22 and the selection section 201 in FIG. 34 ), related party placement ID, placement attribute ID, care recipient ID, care provider ID, and care location ID are received from the input means 46, and the care scene ID is automatically assigned at the start of the care scene. The care scene start time is received from the input means 46 at the start of the care scene, and the care scene end time is received from the input means 46 at the end of the care scene (see the “Start / End” button 92 in FIG. 22 and the “Start / End” button 202 in FIG. 34 ). The number of images and sounds is incremented by one when video data, audio data, and still image data are adjusted by the adjustment means 23, and the number of sensor data is incremented by one when sensor data is adjusted. Of the data received from the input means 46, only the start care item information and care scene start time are input at the start of a care scene (see Figures 22 and 34), and the other data is data input previously (including data read from the album database 33 of the care-related information storage means 30 via the care event information collection means 21). The care coordinator's ID is read from the related party placement table 33B (see Figures 8 and 9) using the case ID, related party placement ID, and placement attribute ID at the time of adjustment by the adjustment means 23 or before that.

[0238] Furthermore, when the care event information collection means 21 receives from the input means 46 provider annotation data or patient annotation data to be added to video data, still image data, and audio data, it creates a new record in the condition data management table 33G (see FIG. 13). Accordingly, a condition data management table 33G is created for each piece of condition data (video data, audio data, still image data, sensor data) indicating the state of the care recipient. In one care scene, a noteworthy state of the care recipient may not appear even once, or may appear two or more times. Therefore, the number of records in the condition data management table 33G (see FIG. 13) for one record in the care history table 33F (see FIG. 12) is not limited to one, but may be zero, two, or more.

[0239] Of the columns in the status data management table 33G (see FIG. 13), the case ID, care session ID, care scene ID, care item ID, related party placement ID, and placement attribute ID are the same as those in the care history table 33F (see FIG. 12) and are copied from the care history table 33F. The status data type is the care coordinator's selection information received by the adjustment means 23. The status data ID is automatically assigned at the time of adjustment by the adjustment means 23. The urgency is received from the input means 46. The care coordinator's ID is entered by the care coordinator who actually performed the adjustment work at the time of adjustment by the adjustment means 23. The transmission date and time of the timing information for specifying the time point of interest (which may be the reception date and time by the care event information collection means 21), the specified relative time (the elapsed time from the representative time point of the time point of interest), and the specified time length are received from the input means 46. The automatic extraction start position and automatic extraction end position store times automatically calculated by the adjustment means 23 using timing information for specifying a time point of interest, and the adjusted extraction start position and adjusted extraction end position store times manually adjusted by the care coordinator and accepted by the adjustment means 23 (see FIG. 16 ). The evaluation request intention is received from the input means 46. The coordinator annotation data for the evaluation request, the coordinator annotation data for the response to the care provider, and selection information for the care evaluator to whom the evaluation request is to be sent are stored as input data by the adjustment means 23. The file storage destination path is automatically set by the adjustment means 23.

[0240] In the case of continuous recording with the equipment arrangement (A) of Figure 20 or recording of the entire care scene with the equipment arrangement (B) of Figure 30, the adjustment means 23 performs automatic cutting out of video data and audio data using timing information for specifying a time point of interest, whereas in the case of recording limited to a time point of interest with the equipment arrangement (C) of Figure 32, the adjustment means 23 does not perform automatic cutting out (see Figure 34), so the transmission date and time (which may be the reception date and time by the care event information collection means 21) of the timing information for specifying a time point of interest in the status data management table 33G (see Figure 13), the specified relative time (the elapsed time from the representative time point of the time point of interest), and the specified time length are left blank. However, even in the case of focused time-point limited recording with equipment arrangement (C) in Fig. 32, automatic cutting processing is not performed, but there are start and end times for recording (photography, audio recording) (for example, when "Record" button 204 in Fig. 34 is clicked), so the start and end times for recording (photography, audio recording) are stored in the columns for automatic cutting start position and automatic cutting end position in condition data management table 33G. Also, in the case of focused time-point limited recording with equipment arrangement (C) in Fig. 32, automatic cutting processing is not performed, but adjustment means 23 may accept manual adjustment by the care coordinator, so the times after this manual adjustment are stored in the columns for adjusted cutting start position and adjusted cutting end position in condition data management table 33G.

[0241] In the case of file transmission in the device arrangement (D) of FIG. 35, the care event information collection means 21 receives a recorded (photographed or audio) video file, still image file, or audio file regardless of whether it is a care scene, and therefore there is often no care history table 33F (see FIG. 12) corresponding to the status data management table 33G (see FIG. 13). The case ID in the status data management table 33G is received from the input means 46, or a case ID corresponding to the ID of the care recipient received from the input means 46 (see the display unit 301 of FIG. 36) is stored. The care session ID, care scene ID, care item ID, related party arrangement ID, and arrangement attribute ID may be left blank. The status data type is the selection information of the care coordinator received by the adjustment means 23. The status data ID is automatically assigned during adjustment by the adjustment means 23. In the case of file transmission, in this embodiment, it is not a target for sending an emergency signal, but it may be possible to send an emergency signal. In that case, the urgency level is received from the input means 46. The care coordinator's ID is entered by the care coordinator who actually performed the adjustment work when the adjustment means 23 made the adjustment. In the case of file transmission using the device arrangement (D) in FIG. 35, the adjustment means 23 does not perform the automatic extraction process, so the transmission date and time (which may be the reception date and time by the care event information collection means 21) of the timing information for specifying the time point of interest in the status data management table 33G, the specified relative time (the elapsed time from the representative time point of the time point of interest), and the specified time length are left blank. However, even in the case of file transmission using the device arrangement (D) in FIG. 35, although the automatic extraction process is not performed, the file creation date and time are recorded, so the file creation date and time are stored in the automatic extraction end position column in the status data management table 33G. Furthermore, in the case of file transmission using the device arrangement (D) in FIG. 35, the automatic extraction process is not performed, but the adjustment means 23 may accept manual adjustment of the data in the file by the care coordinator, so the file update date and time are stored in the post-adjustment extraction end position column in the status data management table 33G.

[0242] Furthermore, not only in the case of file transmission with the equipment arrangement (D) of Figure 35, but also in the case of continuous recording with the equipment arrangement (A) of Figure 20 or recording limited to a particular time point with the equipment arrangement (C) of Figure 32, video data etc. recorded (filmed, audio recorded) during a time period other than the care scene may become video data etc. showing the condition of the care recipient that requires attention. In such cases, there is no care history table 33F (see Figure 12) corresponding to the condition data management table 33G (see Figure 13), and each column in the condition data management table 33G may have blank spaces.

[0243] Furthermore, when the care event information collecting means 21 receives subject needs information (call time, call intention) from the input means 46 as subject transmission information constituting the care event information, it creates a new record in the call history table 33K (see FIG. 6). The case ID and care recipient ID in this call history table 33K may be received from the input means 46, or the case ID and care recipient ID may be converted and acquired from the device identification information received from the input means 46. The call time and call intention are received from the input means 46, but the call time may be the time of reception by the care event information collecting means 21.

[0244] Records in the evaluation / addition table 33H (see FIG. 14) are created not by the care event information collection means 21 but by the evaluation result collection means 26 and the album registration means 25. Details of this will be explained later in <Configuration of the data collection server 20 / album registration means 25> and <Configuration of the data collection server 20 / evaluation result collection means 26>.

[0245] <Configuration of Data Collection Server 20 / Care Setting Means 22>

[0246] The care setting means 22 executes various setting processes, including a setting and registration process for the placement of related parties (see FIG. 7), a care session setting process (see FIG. 10), a setting process for devices and apps used (see FIG. 11), a setting process for new cases such as when a care recipient is first examined, hospitalized, or admitted to a facility (see FIG. 6), and a setting input process for the correspondence between the intentions and emotional expressions of the care recipient and status change detection items for detecting physical status changes in the care recipient that can be recognized visually and auditorily (see FIG. 19). The various care setting screens displayed on the adjuster terminal 60 by this care setting means 22 may be web screens (which may include JavaScript (Java is a registered trademark), etc.) or screens displayed by a dedicated application program.

[0247] (Registering the placement of stakeholders: Figure 7)

[0248] As shown in Figure 7, the care setting means 22 accepts setting inputs for the placement of people or living beings or their substitute inanimate objects to assign roles to each of the care participants, including the care provider who provides care, the care recipient who receives care from the care provider, the care coordinator who coordinates care event information (provider-sent information in Figure 4 and recipient-sent information in Figure 5) that occurs at the care location (facility, home, etc.), and the care evaluator who receives evaluation requests from the care coordinator, and performs a process of storing the inputs in the participant placement table 33B (see Figures 8 and 9), which is a participant placement storage means.

[0249] This care setting means 22 can perform a process of storing, in association with each other, the arrangement of the care provider, care coordinator, care evaluator, and other related parties when providing primary care for the same care recipient, and the arrangement of the care provider, care coordinator, care evaluator, and other related parties when providing secondary care, in the related parties arrangement table 33B (see Figures 8 and 9).

[0250] 7, a related party arrangement screen 400, which is one of the care setting screens displayed on the coordinator terminal 60 by the care setting means 22, includes a care theme display section 401, a selection section 410 for selecting whether the arrangement is basic or modified (information identified by a related party arrangement ID), a selection section 411 for an arrangement attribute indicating the type of care arrangement (information indicating whether the arrangement is primary care or secondary care, identified by an arrangement attribute ID), an input section 420 for the care recipient, an input section 421 for multiple care providers, an input section 422 for multiple care coordinators, an input section 423 for multiple care evaluators, and an input section 424 for multiple other related parties. In the example of FIG. 7, only two input sections 421, 422, 423, and 424 are shown due to space limitations, but more input sections 421-424 may be provided for each. Each of the input sections 421-424 may also be provided in a single section. Furthermore, even if a plurality of each of the input units 421 to 423 are provided, it is not necessary to input data into all of them, and it is sufficient to input data into at least one of the input units 421 to 423, and the remaining input units may be left blank. On the other hand, even if a plurality of input units 424 are provided, it is not necessary to input data into any of the input units 424. In other words, input by other related parties is not essential.

[0251] If multiple care coordinators are registered in the related party allocation table 33B, the emergency signal is sent to all of them. It is not possible for all of them to perform the coordination work at the same time, or a warning is issued. This is to prevent multiple coordination information from being created for one piece of video data, etc., or to prevent the same video data, etc. from being sent multiple times to the same care evaluator. Furthermore, when multiple care evaluators are registered, the care coordinator can select a care evaluator (see the "Select Care Evaluator" button 770 in Figure 16). It is also possible to select all care evaluators and request evaluations from all of them. In the event of an emergency, an emergency signal is sent to all selected care evaluators.

[0252] The input assistance window 430 shown in FIG. 7 includes a customer list display section 431 and a service provider list display section 432. The care coordinator selects personnel, living creatures, or alternative inanimate objects to be assigned to each of the input sections 420-424 from the displays 431 and 432. The display of each display section 431 and 432 is performed using the customer table 33L and the service provider table 33M (see FIG. 15). For living creatures such as pets and alternative inanimate objects such as dolls, names (e.g., "Yamato Taro's Pochi"), addresses (the same as Yamato Taro's), and pre-assigned customer IDs are also registered. The address of a living creature may be the address of the owner or the address of the place where the creature is kept, while the address of a doll or robot may be the address of the management / storage location or the address of the manufacturer.

[0253] The lower part of the participant arrangement screen 400 in Fig. 7 allows the participant arrangement for secondary care to be set on the same screen as the primary care. However, the participant arrangement for secondary care can be set on the upper part of the participant arrangement screen 400, and the participant arrangement for primary care can be set on the lower part, which is a feature of the screen, allowing multiple (two in this embodiment) participant arrangements to be set and registered simultaneously while being compared on the same screen. Note that the primary care and one or more secondary cares can also be set by inputting data into only one of the upper or lower parts of the participant arrangement screen 400 and repeating this process multiple times.

[0254] 7 has, at the bottom, another selection section 440 for selecting basic layout / modified layout, a selection section 441 for layout attributes indicating the type of care layout, an input section 450 for the care recipient, an input section 451 for multiple care providers, an input section 452 for multiple care coordinators, an input section 453 for multiple care evaluators, and an input section 454 for multiple other related parties. These are the same as the selection sections 410, 411 and input sections 420 to 424 provided at the top.

[0255] 7 has a "Send" button 460 at the bottom for transmitting the entered data to the care setting means 22 of the data collection server 20 via the network. When the care coordinator presses this "Send" button 460, the entered data is transmitted from the coordinator terminal 60 to the data collection server 20, and is stored in the participant placement table 33B (see FIGS. 8 and 9) by the care setting means 22. As shown in FIGS. 8 and 9, the participant placement for the primary care and the participant placement for the secondary care are stored in separate records and are associated with the same case ID.

[0256] Specifically, an example of two-stage related party allocation is when the secondary caregiver is model care. In this case, a person other than the primary caregiver serves as the secondary caregiver and provides model care to the care recipient, so that the primary caregiver can use it as a reference when providing the primary care. In this case, the person who will become the primary caregiver is stored in the related party allocation table 33B (see FIG. 8) as the secondary caregiver's care evaluator or care coordinator, or other related party with reference authority. This allows the primary caregiver to view the model reference video data or still image data registered in the secondary care and stored in the album storage means 32 (see FIG. 1) in the capacity of the secondary caregiver or care coordinator, or other related party with reference authority.

[0257] The gist of the above example is that if a primary care provider is registered as a secondary care evaluator or care coordinator, or as one of the other parties with reference authority, that person can refer to the model reference video data, etc. registered in the secondary care and use the information obtained there when implementing the primary care. Therefore, this example has looser conditions and a broader meaning than the example in Figure 8. The example of the two-stage stakeholder arrangement shown in Figure 8 corresponds to Case J4 (see Figure 41) in which the secondary care provider is a model care provider, but the primary care provider and the secondary care evaluator are the same (in Case J4 in Figures 8 and 41, both are nurses). However, primary care providers may be able to refer to model reference video data, etc., even if they are not necessarily registered as a secondary care evaluator. Also, in Case J4 in Figures 8 and 41, the care assessor for the primary care and the care provider for the secondary care are the same (both are the mother), but these do not necessarily have to be the same.

[0258] Another example of two-stage stakeholder arrangement is when the secondary caregiver is a consent confirmation auxiliary caregiver. In this consent confirmation auxiliary caregiver, consent to the implementation of the primary care is obtained in advance from the care recipient in preparation for the subsequent primary care. In this case, the person who will be the primary caregiver is stored as the care evaluator for the secondary care in the stakeholder arrangement table 33B (see FIG. 9). The primary caregiver can then refer to video data (e.g., video data showing the secondary caregiver persuading the care recipient and the care recipient's acceptance) or still image data (e.g., still image data showing the care recipient signing their consent) registered in the secondary care and stored in the album storage unit 32 (see FIG. 1) showing the process of obtaining consent from the care recipient, confirm whether consent has been obtained, and store the confirmation results as the secondary care evaluator annotation data. This specific example corresponds to Case J5 (see FIG. 42), a specific example of two-stage stakeholder arrangement (when the secondary caregiver is a consent confirmation auxiliary caregiver) shown in FIG. 9.

[0259] Furthermore, although not a two-stage participant placement setting, the care setting means 22 can accept and register participant placement settings for educational care using dolls and store them in the participant placement table 33B. Specifically, the participant placement table 33B can store, as care providers, students who receive care training or practice; as care recipients, dolls (including dolls of dogs and cats) that resemble children, elderly people, or other people in need of care (including living creatures such as pets) who are the targets of care training or practice; as care coordinators, teachers or instructors who provide care education; and as care evaluators, guardians or caregivers of people in need of care. This specific example corresponds to Case J10 (see FIG. 47).

[0260] Regarding the case ID for this educational care, if the educational care is to prepare for the care of a specific person or specific creature (such as a pet), the case ID may be the same as the care theme for that specific person or specific creature, or a dedicated case ID may be assigned to the educational care. In the former case, where the case ID is the same as the care theme for a specific person or specific creature, the educational care can be treated as secondary care. In this case, since the care recipient of the secondary care is a doll, the doll's ID is pseudo-registered as the care recipient's ID in the secondary care, but the primary care and secondary care recipient can be considered to be the same. Note that when treating it as secondary care, the doll's ID may be replaced with the ID of a specific person or specific creature (such as a pet).

[0261] Furthermore, if the educational care is intended to prepare for the care of unspecified people or unspecified creatures (such as pets), it is preferable to assign a dedicated case ID to the educational care, at least initially. Later, the dedicated case ID for that educational care can be changed to the same case ID as the care theme for a specific person or creature, making it a secondary care. Alternatively, if that educational care is utilized for the care of multiple care recipients (e.g., care for Patent Taro, care for Patent Jiro, care for Patent Goro, etc.), the care record (including video data, etc.) for that educational care can be copied and considered as educational care for a specific person or creature (e.g., care for Patent Taro only), and assigned the same case ID as the care theme for the specific person or creature, making it a secondary care. The same copying can be done when caring for other people or other creatures (e.g., Patent Jiro, Patent Goro), and the corresponding secondary care can be made.

[0262] (Care session setting: Figure 10)

[0263] The care setting means 22 also executes a process of accepting setting input of care items that the care provider may perform on the care recipient in each care situation or care items that are scheduled to be performed in each care situation. In the present embodiment, as an example, the care setting means 22 sets and registers each care item that is scheduled to be performed in each care situation and the order of performing it as a care session, and stores it in association with a case ID and a care session ID in a care session setting table 33D (see the bottom of FIG. 10 ), which is a care session setting storage means.

[0264] When the care provider operates input means 46 at the care delivery site to input the care item to be started (starting care item information), the care items are typically implemented in order according to the care session settings stored in care session setting table 33D. However, the care items do not necessarily have to be implemented in the order set in advance for the care session; the implementation order may be changed as needed. Furthermore, care for unscheduled care items (in this embodiment, care items not set in the care session) may also be implemented. In such cases, if there is not enough time to search for and input the starting care item information or if it is not possible to find it immediately, the care provider may select "Other Care Items" for the time being and then make adjustments. In subsequent adjustments, the care provider may later become the care coordinator and input the starting care item information, or the care provider may request the care coordinator to input the starting care item information.

[0265] As shown in the upper part of Figure 10, the care item master table 33C stores care type IDs and care type names (e.g., nursing, caregiving, pet sitting, etc.) in association with care item IDs and care item names (e.g., toileting, bathing, suction, diaper changing, treats, dog walking, cat feeding, etc.).

[0266] 10 , a care session setting screen 500, which is one of the care setting screens displayed on the adjuster terminal 60 by the care setting means 22, is provided with a care theme display section 501, a care session name input section 510, an input section 511 for care items to be performed in the first, second, third, fourth, ... care situations, a care type (nursing, caregiving, babysitting, pet sitting, etc.) selection section 520, and a selection section 521 for care items belonging to the care type selected in this selection section 520. Care items are input into the input section 511 by selecting the care items in the selection section 521.

[0267] The care coordinator can also simultaneously set a care session for secondary care on the care session setting screen 500 of Figure 10. Note that the care session for secondary care can also be set on this setting screen 500 at a later date.

[0268] 10 includes an input field 530 for inputting the name of the care session for the secondary care and an input field 531 for inputting the care items to be performed in the first, second, third, fourth, ... care situations. Also, at the bottom of the care session setting screen 500, a "Send" button 540 is provided for transmitting the input data to the care setting means 22 via the network. When the care coordinator presses the "Send" button 540, the input data is registered in the care session setting table 33D. At this time, the care session for the primary care and the care session for the secondary care are stored in separate records, and are assigned separate care session IDs with the same case ID.

[0269] When a care session for secondary care is set, for example, if the secondary care is self-care by the care recipient when returning home temporarily, alternative care provided by the care recipient's mother or other such person when returning home temporarily (care to replace a nurse or other such person at home), self-care after discharge from the hospital, or alternative aftercare provided by the care recipient's mother or other such person after discharge from the hospital, the care recipient and their mother or other such person, although not professionals, can perform self-care or alternative care at home in accordance with the set care session for secondary care. Furthermore, the implementation of such secondary care is recorded in the same way as for primary care.

[0270] (Device and app settings: Figure 11)

[0271] The care setting means 22 accepts setting inputs for devices and application programs used by the care provider, or the care recipient or their input agent, and also performs a process of storing the input setting information for the devices and applications used in association with the case ID in a device and application used setting table 33E, which is a device and application used setting storage means.

[0272] 11, a device / application use setting screen 600, which is one of the care setting screens displayed on the adjuster terminal 60 by the care setting means 22, is provided with a care theme display section 601, a selection section 610 for the image capturing means 41 and the sound recording means 42, and a use situation selection section 611. In the selection section 610, the type of device to be used for image capturing and sound recording is selected, and in the selection section 611, the situation (place or situation) in which the device selected in the selection section 610 will be used is selected.

[0273] Specifically, as in the first example from the top in Figure 11, if "Fixed camera (camera permanently installed at a fixed location within the facility)" is selected in selection unit 610 and "During care within the facility" is selected in selection unit 611, a fixed camera 50 permanently installed in the facility shown in Figure 2 will be used to perform continuous recording (24 hours a day, or with on / off) with equipment arrangement (A) in Figure 20, or to record the entire care scene with equipment arrangement (B) in Figure 30, or to record only at a specific time point in time with equipment arrangement (C) in Figure 32.

[0274] 11 , when the selection unit 610 selects “the camera of the smartphone of the caregiver working at the facility” or “the camera of the tablet provided by the facility” and the selection unit 611 selects “providing care outside the facility,” the camera (photographing means 41) of the mobile device 53, such as the smartphone or tablet of the caregiver shown in FIG. 2, is used to perform time-of-interest limited recording according to the device arrangement (C) of FIG. 32, or to record according to the device arrangement (D) of FIG. 35 and transmit the file. Note that home care (visiting nursing, home care, etc.) falls under “providing care outside the facility,” and care within a facility but in a location where there is no fixed camera 50 is also considered “providing care outside the facility.” Therefore, for example, when a caregiver working at a facility provides care inside a facility building (e.g., in a hospital ward), video data recorded using a fixed camera 50 permanently installed in the facility is used. When the caregiver leaves the facility building (e.g., in a hospital ward) to go for a walk in the garden, video data is captured using the camera of the mobile device 53, such as a smartphone or tablet, that the caregiver carries with him or her. During such "out-of-facility care" such as care for walks, a dedicated app can be launched on a mobile device 53 such as a smartphone or tablet that the user carries with them and video data can be recorded at a specific point in time using the device arrangement (C) in Figure 32, or video data can be recorded using the general-purpose recording function built into the mobile device 53 without using a dedicated app using the device arrangement (D) in Figure 35, and then the file can be sent.

[0275] Furthermore, as in the fourth example from the top in FIG. 11 , when “a camera on a smartphone of a person other than a facility worker (when visiting the facility, observing at home, etc.)” is selected in the selection unit 610 and “outside care” is selected in the selection unit 611, for example, a mother visiting a care recipient or a mother observing a care recipient who has returned home temporarily will, since the care provider is not present (because it is an outside-care scene), use the camera on a mobile device 54 such as a smartphone or tablet that she carries with her as an input agent for the care recipient to capture video data. Therefore, even if the care recipient is inside the facility and is constantly being photographed by a fixed camera 50 when visiting the facility, the mother, who is acting as an input agent for the care recipient, will not use the constantly photographed video data, but will instead use the camera on her own mobile device 54. In this case, a time-point-limited recording is performed using the device arrangement (C) in FIG. 32 or recording is performed using the device arrangement (D) in FIG. 35, and then the file is transmitted.

[0276] 11 , when “Smartphone camera of a person other than a facility employee (in the case of a caregiver providing secondary care)” is selected in selection unit 610 and “During secondary care” is selected in selection unit 611, for example, when the mother of the care recipient (child) is providing consent confirmation assistance care to obtain consent from the care recipient, or when the mother of the care recipient (child) is serving as a secondary caregiver at home during a temporary home visit or immediately after discharge from the hospital, video data is captured using the camera of mobile device 54, such as a smartphone or tablet, carried by the mother herself (mother, etc.). Note that although the mother, etc., serves as a secondary caregiver, in primary care, she is a caregiver for the care recipient (child), and therefore mobile device 54 is used here, not mobile device 53. In this case, time-of-interest limited recording is performed using device arrangement (C) in FIG. 32, or recording is performed using device arrangement (D) in FIG. 35, and then the file is transmitted.

[0277] Furthermore, as in the sixth example from the top in Fig. 11, when "camera of smartphone of person other than facility worker (in the case of a care recipient outside the facility)" is selected in selection unit 610 and "during self-care" is selected in selection unit 611, the care recipient performing self-care at home captures video data using the camera of a mobile device 54 such as a smartphone or tablet that they carry with them. In this case, recording is performed with a time-point-limited recording in device arrangement (C) in Fig. 32, or recording is performed with device arrangement (D) in Fig. 35, and then the file is transmitted.

[0278] 11 also includes a selection section 620 for selecting the input means 46 and application program used by the caregiver working at the facility. For example, if the user selects "Tablet (provided by facility) & provider app for smartphone / tablet (supports continuous photography)" in the selection section 620, the caregiver working at the facility uses the facility-provided tablet, which is the caregiver's mobile device 53, as the input means 46. If the facility-provided tablet does not have the "provider app for smartphone / tablet (supports continuous photography)" (see FIG. 22 ), the caregiver working at the facility requests the care event information collection means 21 to download the app from the input means 46 (mobile device 53), downloads the application program "provider app for smartphone / tablet (supports continuous photography)" stored in the app storage means 35, and installs it on the input means 46 (mobile device 53). Because the facility-provided tablet is a device loaned to the caregiver working at the facility by the facility, the app is usually already installed on the tablet when it is loaned. It should be noted that instead of the "smartphone / tablet implementer app (supports continuous photography)", a web screen (see FIG. 22) transmitted from the care event information collection means 21 may be used.

[0279] Furthermore, the device / app usage setting screen 600 in Fig. 11 is provided with a selection section 630 for the input means 46 (including the call device 56 in Fig. 28) and application program that the care recipient will use within the facility. If, for example, "input operation device used as both a call device (concurrent)" is selected in this selection section 630, the care recipient in the facility will use the input operation device 55E in Fig. 27 as the input means 46 within the facility. Unlike general-purpose devices such as smartphones and tablets, the input operation device 55E is a dedicated device for care, and therefore does not require the downloading of an app.

[0280] 11 also includes an input means 46 and an application program selection section 640 for the caregiver working at the facility to use outside the facility. If, for example, "Practitioner app for smartphones and smartphone / tablets (supports limited time point photography)" is selected in this selection section 640, the caregiver working at the facility will use a smartphone, which is a mobile device 53 that the caregiver carries with them, as the input means 46 outside the facility. As mentioned above, "outside the facility" refers to, for example, home care or care provided when the caregiver leaves the facility building (such as a hospital ward) and goes for a walk in the garden. Furthermore, if the care provider working at the facility does not have the "Provider App for Smartphones and Tablets (Supports Limited Time-of-Minute Photography)" (see FIG. 34) installed on the smartphone that is the mobile device 53 that the care provider carries with them, the care provider requests the care event information collection means 21 to download the app from the input means 46 (smartphone that is the mobile device 53), downloads the application program "Provider App for Smartphones and Tablets (Supports Limited Time-of-Minute Photography)" stored in the app storage means 35, and installs it on the input means 46 (smartphone that is the mobile device 53). Note that instead of the "Provider App for Smartphones and Tablets (Supports Limited Time-of-Minute Photography)", the care provider may use a web screen (see FIG. 34) sent from the care event information collection means 21.

[0281] Furthermore, if the above selection unit 640 selects, for example, "Provider App for Facility-Provided Tablet & Smartphone / Tablet (Supports Time-of-Target Limited Photography)," the care provider working at the facility will use the facility-provided tablet, which is the portable device 53 that the care provider carries with them, outside the facility as the input means 46. Furthermore, if the facility-provided tablet, which is the portable device 53 that the care provider carries with them, does not have the "Provider App for Smartphone / Tablet (Supports Time-of-Target Limited Photography)" (see FIG. 34), the care provider will make a download request for the app from the input means 46 (the facility-provided tablet, which is the portable device 53) to the care event information collection means 21, download the application program "Provider App for Smartphone / Tablet (Supports Time-of-Target Limited Photography)" stored in the app storage means 35, and install it on the input means 46 (the facility-provided tablet, which is the portable device 53). Note that instead of the "Provider App for Smartphone / Tablet (Supports Time-of-Target Limited Photography)," a web screen (see FIG. 34) transmitted from the care event information collection means 21 may be used.

[0282] Furthermore, the device / app setting screen 600 in FIG. 11 is provided with an input means 46 and an application program selection section 650 used by persons other than facility staff (including care recipients outside the facility). For example, if the "Smartphone & File Registration App" is selected in the selection section 650, persons other than facility staff (including care recipients outside the facility, guardians such as the mother visiting, relatives, acquaintances, friends, etc.) use their own mobile device 54, i.e., a smartphone, as the input means 46. Furthermore, if the "File Registration App" (see FIG. 36) is not installed on the mobile device 54, i.e., the smartphone, other than facility staff, requests the care event information collecting means 21 to download the app from the input means 46 (i.e., the mobile device 54), downloads the application program "File Registration App" stored in the app storage means 35, and installs it on the input means 46 (i.e., the smartphone, i.e., the mobile device 53). Instead of the "File Registration App," a web screen (see FIG. 36) transmitted from the care event information collecting means 21 may be used.

[0283] 11, a "Send" button 660 is provided at the bottom for sending the input data to the care setting means 22 via the network. When the care coordinator presses this "Send" button 660, the input data is stored in the device / application used setting table 33E.

[0284] At this time, the photographing / recording device type ID, which identifies the type of photographing means 41 and recording means 42 selected by the selection unit 610, and the corresponding usage scene ID, which identifies the usage scene selected by the selection unit 611, are stored in association with the case ID in the device / application setting table 33E. Also, the input means / application ID, which identifies the input means 46 and application program selected by the selection units 620 to 650, are stored in association with the case ID in the device / application setting table 33E.

[0285] Although not shown in FIG. 11 , not only is the imaging / audio recording device type ID, which identifies the type of imaging means 41 and audio recording means 42, associated with the case ID, but if it is necessary to associate device identification information (such as device number) for each individual device with the case ID, this setting is also performed and stored in the device / application setting table 33E. For example, when using video data captured by a fixed camera 50 permanently installed in a facility, if there are multiple fixed cameras 50 of the same type in the facility and multiple care recipients, it is necessary to set the correspondence between them. More specifically, for example, if there are multiple beds in the facility and a fixed camera 50 is installed for each bed, it is necessary to set which fixed camera 50 of which bed is capturing video data of which care recipient (i.e., which case ID). As described above, the care event information collecting means 21 receives the device identification information of the fixed camera 50 (not just the type but also the device number for identifying each device) along with the video data captured by the fixed camera 50. Therefore, by setting the correspondence between the device identification information (device number, etc.) and the case ID in the device / application setting table 33E, it is possible to identify which care recipient's condition the received video data shows (which case ID the video data is for). Note that if there is only one fixed camera 50 of the same type in a facility, setting the type ID of the recording / photography device to be used is equivalent to setting the device identification information for the device to be used (each device), and therefore it is not necessary to set the device identification information for each device in association with the case ID.

[0286] Furthermore, when there are multiple robots that move following a caregiver (e.g., a nurse) within a facility, and video data captured by the mobile cameras 51 mounted on the robots is used, the caregiver may input (or input in advance) which robot's mobile camera 51 to use along with the caregiver annotation data (e.g., the intent of the evaluation request and timing information for specifying the point of interest) using the input means 46, and the device identification information of the mobile camera 51 of the selected robot may be transmitted together with the caregiver annotation data to the care event information collecting means 21. In this case, a specific robot does not always follow a specific caregiver (e.g., a nurse), and when providing a certain care, the use of the mobile camera 51 mounted on the robot that was following at that time is selected, and the care event information collecting means 21 receives the device identification information of the selected mobile camera 51 together with the caregiver annotation data (which includes the caregiver's ID and the ID and / or case ID of the care recipient for which the caregiver is responsible). On the other hand, the care event information collection means 21 receives video data transmitted from the mobile camera 51 mounted on each robot along with the device identification information of the mobile camera 51, and can therefore identify which care recipient's condition the video data captures (which case ID the video data is for).

[0287] Furthermore, when a robot is dedicated to assisting the care of a specific care recipient, the correspondence between the device identification information (device number, etc.) of the mobile camera 51 mounted on the robot and the case ID can be set in the device / application setting table 33E, similar to the case of a fixed camera 50 provided for each bed, for example. Note that when there is only one mobile camera 51 of the same type in a facility, it is not necessary to associate the device identification information of each device with the case ID.

[0288] Furthermore, video data captured by cameras mounted on mobile devices 53, 54 such as smartphones or tablets owned by individuals (including those loaned from a facility) is sent to the care event information collection means 21 with the ID of the individual (care provider, or care recipient, or their input agent) or with the case ID attached. Therefore, it is possible to identify which care recipient's condition the video data received by the care event information collection means 21 shows (which case ID the video data belongs to). In this case, the mobile devices 53, 54 serve as the imaging means 41 and also as the input means 46. In other words, the cameras mounted on mobile devices 53, 54 such as smartphones and tablets are used to capture images of the care recipient when an operation for capturing images is performed on the screen displayed on the mobile device 53, 54 (for example, when the record button 204 on the care provider input screen (supports capture of images limited to a specific time point) 200 in Figure 34 is clicked), or when an image of the care recipient is captured with the camera of the mobile device 53, 54, converted into a file, and the file is registered and sent on the screen displayed on the mobile device 53, 54 (for example, when sending on the file registration screen 300 in Figure 36).Therefore, the ID of the individual (care provider, or care recipient or their input agent) or the case ID is sent to the care event information collection means 21 along with the video data from the application program or web browser displaying these screens.

[0289] There are roughly two usage modes for the web camera 52A of the personal computer (PC) 52. One is when it is used like a fixed camera 50 or a mobile camera 51 installed in a facility, in which case the PC 52 functions as the imaging means 41 that captures video data, but the input means 46 is composed of another device (for example, portable devices 53, 54, an input operation device 55, another PC 52, etc.). The other is when it is used like a camera of portable devices 53, 54 owned by an individual (including when loaned from a facility), in which case the PC 52 functions as the imaging means 41 that captures video data and also functions as the input means 46 that inputs the implementer annotation data, the subject annotation data, etc. Therefore, the process of identifying which care recipient's condition the video data sent from PC 52 and received by care event information collection means 21 shows (which case ID the video data is for) is the same as when the video data is taken with fixed camera 50 or mobile camera 51, or when the video data is taken with a camera on mobile device 53, 54.

[0290] The above explanation has described the process of identifying which care recipient the video data received by the care event information collection means 21 shows (which case ID the video data is for), but the process of identifying which care recipient the received audio data shows (which case ID the audio data is for) is similar.

[0291] 11, a sensor type ID indicating the type of sensor 43 to be used is also associated with the case ID and set in the device / application setting table 33E. In this case, if there are multiple sensors 43 of the same type in the facility and multiple care recipients using those types of sensors 43, in order to determine the correspondence between them, not only the type of sensor 43 but also device identification information identifying each individual sensor 43 is associated with the case ID and set in the device / application setting table 33E. Because the sensor data is transmitted from the sensor 43 together with the device identification information, it is possible to identify which care recipient's biological information the sensor data received by the care event information collecting means 21 indicates (which case ID the sensor data corresponds to).

[0292] (Setting new cases when a care recipient first visits, is admitted to hospital, or is admitted to a facility: Figure 6)

[0293] The care setting means 22 also executes a new case setting process as a preparation process for a new client who is a care recipient at the time of their first consultation, hospitalization, or admission to a facility. Specifically, the care coordinator registers information such as the name, date of birth, and address of the new client who is a care recipient or a related person such as their mother in the client table 33L (see FIG. 15). At this time, the care setting means 22 assigns a client ID or a secondary client ID to the care recipient or a related person such as their mother.

[0294] Next, on a case setting screen (not shown), which is one of the care setting screens, the care coordinator selects a facility, inputs a care theme (case name) for the new case, and selects a care recipient who is a new client from customer table 33L (see FIG. 15), as shown in Fig. 6, and stores the selected and input data (facility ID, case name, client ID and name of the care recipient, etc.) in case table 33A. At this time, a case ID is assigned by care setting means 22.

[0295] (Settings for inputting the correspondence between the intentions and emotional expressions of the care recipient and the status change detection items for detecting the physical status changes of the care recipient that can be recognized visually and auditorily: Figure 19)

[0296] The care setting means 22 accepts setting inputs on a status change detection item setting screen (not shown), which is one of the care setting screens, regarding the correspondence between the intentions and emotional expressions of the care recipient and status change detection items for detecting physical status changes in the care recipient that can be heard and recognized visually, and also performs a process of storing the accepted correspondence in a status change detection item table 33P, which is a status change detection item storage means, in association with the care recipient's identification information (customer ID) as shown in Figure 19, or in association with the care recipient's identification information (customer ID) and care item ID in the status change detection item table 33P.

[0297] Here, "intentions and emotional expressions of the care recipient" refer to intentions and emotional expressions of the care recipient that may be expressed in the care situation or that may be expressed in the care situation when a specific care item is performed. For example, it could be the expression of universal facial expressions that express the six basic emotions (anger, disgust, fear, joy, sadness, and surprise) according to Ekman's theory. Furthermore, it is not limited to the expression of these six emotions; it could also be the expression of only the emotions of comfort and discomfort. Furthermore, it could also be any request intention, such as "I want you to do ~."

[0298] In addition, "status change detection items for detecting visually and auditorily recognizable physical status changes in the care recipient" include, for example, the movement of each of the care recipient's facial muscles, movement in the direction of gaze, movement in the direction of the face, movement in the position of skeletal and joint points, or movement of other body parts of a human or living thing, or changes in the vocalizations or cries of the care recipient, or changes in intangible or tangible objects emanating from the body of another human or living thing.

[0299] Here, "the movement of each of the multiple facial muscles" refers to a basic unit of movement of facial muscles (groups of facial muscles) called an action unit (AU) that captures facial expressions such as raising the inner eyebrow, raising the outer eyebrow, lowering the eyebrow, raising the upper eyelid, etc. This AU was systematized by Paul Ekman.

[0300] The "intangible objects" in the "intangible or tangible objects emanating from the body of a human or living thing" include, for example, the vocalizations of the person being cared for (including not only linguistically meaningful vocalizations, but also strange cries, sighs, groans, and gasps during hyperventilation), or cries (in the case of living things), sneezing, yawning, coughing, sniffling, etc. These are sounds, and therefore are intangible objects whose state changes can be recognized from the audio data contained in the video data.Furthermore, the light emitted by fireflies, luminous mushrooms, anglerfish, etc. is also light, and so is an "intangible object" as referred to here, and is an intangible object whose state changes can be recognized from the video data. On the other hand, "tangible objects" include, for example, vomit, excrement (in the case of living creatures not wearing pants or diapers), nosebleeds, runny nose, foam around the mouth, tears, eye boogers, etc., and these are tangible objects whose state changes can be recognized in video data. Also, sweating falls under the category of "tangible objects" here if its state changes can be recognized in video data, but when a sweat sensor is used, it is not recognized using video data, so it does not fall under the category of "tangible parts" here, but falls under sensor data.

[0301] The status change detection item table 33P (see FIG. 19 ) stores information provided by the care recipient's guardian (e.g., mother). The stored information includes, for example, information such as what emotion the care recipient feels when making a certain facial expression (information associated with the care recipient ID), or information such as what emotion the care recipient feels when making a certain facial expression while providing care for a certain care item (information associated with the care recipient ID and the care item ID). This information is provided to the facility or care coordinator and stored in the status change detection item table 33P. The information may be provided in any format, such as the results of automatic analysis of a report (which may be audio) submitted by the care recipient's guardian (e.g., mother), or the results of reading or listening by the care coordinator or facility staff.

[0302] <Configuration of Data Collection Server 20 / Adjustment Means 23>

[0303] The adjustment means 23 supports or substitutes for the care coordinator's adjustment work on the care event information stored in the care-related information storage means 30, which is stored in the related party allocation table 33B (see Figures 8 and 9), which is the related party allocation storage means, to create adjustment information including video data, still image data, or audio data for requesting an evaluation, and adjuster annotation data for requesting an evaluation to be added to these data, and executes processing to store the created adjustment information as care-related information in the care-related information storage means 30. Note that the created adjustment information may also include sensor data.

[0304] This processing by the adjustment means 23 is executed after the care event information is received by the care event information collection means 21, but at least notification to the care coordinator is executed immediately after the care event information is received. That is, in this embodiment, when the care event information collection means 21 receives care event information, regardless of whether the care event information includes an emergency signal, the adjustment means 23 immediately notifies the care coordinator operating the coordinator terminal 60 by push notification (a function that displays a message on the terminal regardless of whether the app is running) that care event information to be adjusted has been received (including a notification to that effect in the case of an emergency). This push notification method is arbitrary and can be performed using, for example, a mobile phone number, a social networking service (SNS) account, an email address, or the like stored in the service provider table 33M (see FIG. 15 ). Alternatively, a push notification can be sent using a function of a general-purpose tool such as an SNS or email client to notify the coordinator terminal 60 that care event information to be coordinated has been received (including that fact in the case of an emergency), and the care coordinator who sees the notification can then enter information on an input screen or web screen (e.g., coordinator screen 700 in FIG. 16 ) displayed by a dedicated application program according to the present invention. Alternatively, rather than using a function of a general-purpose tool to send a push notification, the push notification can be sent using a function of the dedicated application program according to the present invention installed on the coordinator terminal 60. Whether or not a care coordinator who receives a push notification immediately performs coordination work on the coordinator terminal 60 depends on the judgment and circumstances of the care coordinator, but at least in the case of an emergency, the care coordinator is informed of the emergency.

[0305] 1 and 2, adjustment means 23 automatically extracts data for a time point of interest from video data or audio data (video data or audio data recorded including time periods other than the time point of interest, in the case of continuous recording in FIGS. 20 and 21 and in the case of recording of the entire care scene in FIGS. 30 and 31) accumulated and stored in temporary storage means 31 at an extraction position determined by timing information for specifying the time point of interest received as provider annotation data or subject annotation data to create a record fragment, and also performs a process of assigning adjuster annotation data for an evaluation request to the created record fragment and storing the record fragment and the adjuster annotation data for an evaluation request as adjustment information in care-related information storage means 30. This process of creating record fragments by automatic extraction by adjustment means 23 substitutes for part of the coordination work (the work of creating coordination information) performed by a care coordinator.

[0306] Of the video data or audio data stored in temporary storage means 31, data that has only been recorded (photographed or audio recorded) at a time of interest (in the case of time-of-interest limited recording in Figures 32 and 33) was acquired by performing a recording operation that assumes that only data at the time of interest will be recorded, so the adjustment means 23 does not perform the process of creating record fragments by automatic segmentation described above, and therefore does not receive timing information for specifying a time of interest for the automatic segmentation process (see Figure 34). However, this data is subject to manual adjustment by the care coordinator.

[0307] On the other hand, the video data, audio data, and still image data received in files stored in file storage means 34 (in the case of file transmission in Figure 35) essentially contains only data recorded at the time of interest, so the adjustment means 23 does not perform the process of creating record fragments by automatic segmentation as described above, and therefore does not receive timing information for specifying the time of interest for the automatic segmentation process (see Figure 34). However, they are subject to manual adjustment by the care coordinator.

[0308] In addition, in all of the above cases (the case of continuous recording in Figures 20 and 21, the case of recording the entire care scene in Figures 30 and 31, the case of recording limited to a time point of interest in Figures 32 and 33, and the case of file transmission in Figure 35), i.e., regardless of whether the process of creating record fragments by automatic cutting is performed or not, the adjustment means 23 transmits care event information (data stored in the temporary storage means 31 or the album storage means 32) to the adjuster terminal 60 via the network, as shown in Figure 1, and receives adjustment instructions from the care adjuster and adjuster annotation data transmitted from the adjuster terminal 60 via the network, thereby performing the process of creating adjustment information to be transmitted to the care evaluator.

[0309] (Adjustment work on the adjuster screen 700 in FIG. 16, processing of gestures and voice data)

[0310] 16 to adjuster terminal 60 via the network, displays this adjuster screen 700 on the screen of adjuster terminal 60, accepts adjustment instructions from the care adjuster on this adjuster screen 700, and executes processing to support the creation of adjustment information. Note that adjuster screen 700 may be a web screen (which may include JavaScript (Java is a registered trademark), etc.) or a screen displayed by a dedicated application program installed on adjuster terminal 60.

[0311] 16, the adjuster screen 700 includes a display section 701 for the care recipient, a display section 702 for the care item, a display section 703 for the care provider who sent information about the event that occurred at the care location, a display section 704 for the date and time of the sending, and a display section 705 for displaying an emergency when an emergency signal is received. The emergency signal is sent from the input means 46 via the network and received by the care event information collection means 21, and is stored in the urgency level of the condition data management table 33G (see FIG. 13) of the album database 33.

[0312] The adjuster screen 700 in FIG. 16 also has a display section 710 for displaying the video to be watched, a "Start Playback" button 711 for starting playback of the video, a "Back" button 712 for rewinding the video, a "Stop" button 713 for stopping the video, a cut-out position adjustment section 720 for adjusting the cut-out position of the video, and a display section 725 for displaying the evaluation request intention of the care provider (or the care recipient or their input agent).

[0313] The extraction position adjustment section 720 displays a time axis in the horizontal direction, and on this time axis, the automatic extraction start position and automatic extraction end position that are automatically determined using timing information for specifying a time point of interest are shown. Note that in the case of time point limited recording in Figures 32 and 33 and in the case of file transmission in Figure 35, automatic extraction processing is not performed, so these automatic extraction start position and automatic extraction end position indicate the start position and end position of the collected video data and audio data.

[0314] The cut-out position adjustment unit 720 also has a cut-out start position adjustment operation unit and a cut-out end position adjustment operation unit, and further displays the current playback position along with the time. The current playback position moves as the care coordinator operates each of the buttons 711 to 713. After confirming and understanding the evaluation request intention of the care provider (or the care recipient or their input agent) displayed on the display unit 725, the care coordinator refers to the video displayed on the display unit 710 and determines whether the automatic cut-out start position and the automatic cut-out end position are appropriate. If the care coordinator determines that the automatic cut-out start position and the automatic cut-out end position are inappropriate, the care coordinator operates the cut-out start position adjustment operation unit and / or the cut-out end position adjustment operation unit to adjust the cut-out start position and / or the cut-out end position. The adjustable range extends beyond the range from the automatic cut-out start position to the automatic cut-out end position. This is because the video data and audio data accumulated and stored in the temporary storage means 31 are continuously recorded, including time periods other than the time of interest.

[0315] Therefore, when creating adjustment information to be sent to the care evaluator, the care coordinator can use data from a time period shorter than the time period of the time point of interest specified by the care provider, care recipient, or their input agent as the adjustment information, or data from a time period longer than the time period of the time point of interest, or data from a time period shifted to the left or right (which may also vary in length). However, in the case of the time point-limited recording of Figures 32 and 33 and the file transmission of Figure 35, since no data outside the time point of interest exists, the care coordinator can only use all of the data as adjustment information or data from a shorter time period as adjustment information. Note that in this application, the data from the automatic extraction start position to the automatic extraction end position determined using timing information for specifying the time point of interest is sometimes referred to as the first record fragment, and the data after manual adjustment by the care coordinator as described above is sometimes referred to as the second record fragment.

[0316] The evaluation request intention of the care provider, or the care recipient, or their input agent, displayed on the display unit 725 is stored in the status data management table 33G shown in Fig. 13. When this evaluation request intention is transmitted as text data from the care location, the text data is stored in the status data management table 33G. When the evaluation request intention is transmitted as a selection signal for the evaluation request intention or a single fixed signal indicating the evaluation request intention (the meaning of the single fixed signal has already been described in detail in the explanation of Fig. 4), the status data management table 33G stores evaluation request intention identification information (such as an identification number or identification code) corresponding to the received signal, or stores text data of the semantic content corresponding to this evaluation request intention identification information, or stores both the evaluation request intention identification information and the corresponding text data.

[0317] Furthermore, when the evaluation request intention is transmitted as voice data from the care site (however, not as voice data in video data recording the state of the care recipient), the adjustment means 23 executes voice recognition processing, and the text data that is the voice recognition result is stored in the status data management table 33G. The text data is then displayed on the display unit 725.

[0318] Furthermore, if the video data or audio data recording the state of the care recipient contains gestures or audio data for conveying an evaluation request intention that corresponds to the care recipient annotation data, the adjustment means 23 automatically converts the gestures or audio data for conveying an evaluation request intention into evaluation request intention identification information or text data indicating the evaluation request intention, and the evaluation request intention identification information and / or text data resulting from the conversion are stored in the status data management table 33G. The stored text data or the text data with meaning corresponding to the stored evaluation request intention identification information is then displayed on the display unit 725.

[0319] This includes cases where a gesture is converted into evaluation request intention identification information, cases where a gesture is converted into text data, cases where voice data is converted into evaluation request intention identification information, and cases where voice data is converted into text data.

[0320] Therefore, for example, if a care provider utters "What does this facial expression mean?" at the care location, the audio data of the utterance may be converted by the adjustment means 23 into evaluation request intention identification information indicating the evaluation request intention of "I don't know the meaning of the care recipient's facial expression," and the text data of the meaning content corresponding to the evaluation request intention identification information, "I don't know the meaning of the care recipient's facial expression," may be stored in the status data management table 33G and displayed on the display unit 725. In other words, the free utterance "What does this facial expression mean?" may be replaced with the formulaic text "I don't know the meaning of the care recipient's facial expression."

[0321] In this case, the recognition process of the gesture for conveying the evaluation request intention by the adjustment means 23 can be performed by existing technology, and the technology used in the section detection process (see FIG. 19) described later can be used. For example, in the case of gestures indicated by facial expressions, action unit (AU) detection technology can be used, and in the case of gestures as hand signals or signals using the whole body, skeleton estimation technology can be used.

[0322] Furthermore, the recognition process of the voice data for transmitting the evaluation request intention by the adjustment means 23 can be realized by a general-purpose voice recognition process. When converting the voice data into the evaluation request intention identification information, for example, the text data obtained as a result of the voice recognition process using the voice data is vectorized, and the text data of the meaning corresponding to the evaluation request intention identification information (prepared standard text data) is also vectorized, and the similarity (for example, cosine similarity) between the vector data of the voice recognition result and the vector data corresponding to the evaluation request intention identification information is calculated, and it is possible to determine which evaluation request intention identification information the voice recognition result is closer to based on the magnitude of the calculated similarity value.

[0323] In addition, if the video data recording the state of the care recipient includes both gestures and voice data for conveying the intention to request evaluation, which correspond to the caregiver annotation data, the adjustment means 23 may convert each of the gestures and voice data for conveying the intention to request evaluation into evaluation request intention identification information or text data indicating the intention to request evaluation, and may perform a process of integrating the gesture conversion results and the voice data conversion results to obtain the final evaluation request intention identification information or text data indicating the intention to request evaluation.

[0324] In this configuration where the gesture conversion result and the voice data conversion result are integrated, even if one conversion result results in an unclear intention, if some intention is obtained from the other conversion result, the adjustment means 23 will adopt the other intention. Note that if the two conversion results are completely different, the implementer annotation data may be generated as an unclear evaluation request intention, or the implementer annotation data may be generated as an implementation data in which both evaluation request intentions are written in parallel.

[0325] When integrating both conversion results, it is necessary to determine whether the two conversion results match or differ, as described above. Therefore, both conversion results are temporarily converted into evaluation request intention identification information by the adjustment means 23, and a match / mismatch determination is performed in that state. Then, the final evaluation request intention identification information indicating the evaluation request intention obtained by the integration and / or the final text data indicating the meaning corresponding to the final evaluation request intention identification information are stored in the state data management table 33G. Furthermore, the stored final text data or the final text data of the meaning corresponding to the stored final evaluation request intention identification information is displayed on the display unit 725.

[0326] Furthermore, gestures or voice data for conveying a call intention may be included in video data or audio data showing the care recipient's appearance (data that is continuously recorded, including times other than when care is being provided, i.e., including times when a caregiver is not present). This call intention, together with the call time, constitutes the care recipient's needs information (see FIG. 5). As already explained using FIG. 5, the care recipient's needs information includes a call intention (e.g., a care item requested by the care recipient or their input agent) when a button on the call device 56 or an input operation device 55 having an equivalent call function is pressed. This call intention may be voice data uttered simultaneously or approximately simultaneously with the button press, or voice data uttered immediately after the button press. Such voice data may be voice data input using a voice input function provided in the call device 56 or the input operation device 55, or voice data input to be included in the video data or audio data showing the care recipient's appearance. Here, the latter type of voice data is used. Furthermore, gestures for conveying a call intention are included in the video data showing the care recipient's appearance.

[0327] In this case, the adjustment means 23 extracts data from the video data or audio data that is continuously recorded, including time periods other than care scenes, for the time period before and after the call time when a button on the call device 56 or an input operation device 55 with equivalent call function is pressed, and performs a process of converting the gesture or audio data for communicating the call intention contained in the extracted data into call intention identification information (such as an identification number or identification code) or text data indicating the call intention.

[0328] This includes cases where a gesture is converted into call intention identification information, cases where a gesture is converted into text data, cases where voice data is converted into call intention identification information, and cases where voice data is converted into text data.

[0329] In this case, the recognition process of the gesture for transmitting a call intention by the adjustment means 23 can be realized by the same technique as the recognition process of the gesture for transmitting an evaluation request intention described above. Also, the recognition process of the voice data for transmitting a call intention by the adjustment means 23 can be realized by the same technique as the recognition process of the voice data for transmitting an evaluation request intention described above.

[0330] The adjuster screen 700 in FIG. 16 also has an input section 730 for adjuster annotation data (included in the adjustment information) for requesting an evaluation from the care evaluator, and an "Audio input" button 731 for inputting this data by voice, an input section 740 for adjuster annotation data (non-adjusted information) for responding to the care provider, and an "Audio input" button 741 for inputting this data by voice, and an input section 750 for adjuster annotation data (non-adjusted information) for responding to the care recipient, and an "Audio input" button 751 for inputting this data by voice.

[0331] Here, in Figure 16, the input section 730 for the adjuster annotation data for evaluation requests is shown in a state where it can be entered freely, but it may also be configured to allow for selective input from a plurality of pre-prepared standard text data, or the care coordinator may select adjuster annotation identification information that aligns with their own adjustment intentions from among the adjuster annotation identification information (identification numbers, identification codes, etc.) that indicates a plurality of pre-prepared adjuster annotation data for evaluation requests, and use the selected adjuster annotation identification information as the adjustment information (adjuster annotation data for evaluation requests) to be sent to the care evaluator.

[0332] Note that input into input units 740 and 750 is not necessary if not required. Conversely, it is possible to input and send only into input units 740 and 750 without inputting into input unit 730. In this case, the care coordinator may again open coordinator screen 700 in FIG. 16 and input into input unit 730 to create coordination information to be sent to the care evaluator.

[0333] The adjuster screen 700 in FIG. 16 also has a display section 760 for displaying the duration of the current video (the duration of the automatically cut-out video data, or the duration of the video data after subsequent adjustment by the cut-out position adjustment section 720), a “Display and confirm one representative frame” button 761 for displaying and confirming one representative frame from the current video data whose duration is displayed on this display section 760, a selection section 762 for selecting to send this one representative frame (still image) to the care evaluator as adjustment information, a “Display and confirm five representative frames” button 763 for displaying and confirming five representative frames (however, this is not limited to five frames, as long as multiple representative frames are sufficient) from the current video data whose duration is displayed on the display section 760, and a selection section 764 for selecting to send these five representative frames (still images) to the care evaluator as adjustment information.

[0334] Therefore, if the care coordinator believes that evaluation is possible using one or more still images instead of a video, the still image data (one representative frame or multiple representative frames) can be used as adjustment information. The one representative frame is a frame automatically selected by the adjustment means 23, and is the center position of the current video data whose duration is displayed on the display unit 760, the center position of the time period of interest at the time point when the first automatic clipping was performed, or a frame at a position where a characteristic movement is present in the video data. The characteristic movement in the video data is a frame detected by a detection process using action units (AUs) or the like, which will be described later. The multiple representative frames may be frames extracted at appropriate time intervals (predetermined time intervals) centered on the one representative frame, or multiple frames in which several characteristic movements are detected.

[0335] 16 further includes a "Perform segmentation processing by segment detection" button 765. When the care coordinator presses this button 765, segment detection processing (see FIG. 19) is executed by adjustment means 23, and details of this processing will be described later.

[0336] The adjuster screen 700 in Figure 16 is provided with a "Select care evaluator" button 770, a "View album" button 771, and a "Send" button 772 for sending the input data to the adjustment means 23 via the network.

[0337] When the care coordinator presses the "Select Care Evaluator" button 770, the care coordinator can select a care evaluator to whom the coordination information will be sent. Care evaluators are pre-registered in the related party allocation table 33B (see FIGS. 7 to 9). When multiple care evaluators are registered, the default state is that care evaluator 1 always receives the evaluation request. However, all care evaluators may be selected and an evaluation request may be sent to them, or an evaluation request may be sent to some care evaluators, including care evaluator 1. Note that if it is known that care evaluator 1 cannot receive the evaluation request, or is in a situation where he or she cannot evaluate even if he or she receives it, or if sending an evaluation request now would cause inconvenience, care evaluator 1 may be removed from the destination list and an evaluation request may be sent to another care evaluator.

[0338] When the care coordinator presses the "View Album" button 771, they can refer to the care-related information stored in the album storage means 32 about the care recipient currently being processed, and can then input the coordinator annotation data after viewing the progress to date.

[0339] (Section detection process using state change detection items: Figure 19)

[0340] As shown in Figure 19, the adjustment means 23 uses video data showing the state of the care recipient to detect status changes in status change detection items stored in the status change detection item table 33P, which is a status change detection item storage means, in association with the identification information (customer ID) of the care recipient who was the subject of the video data, or performs pattern recognition processing to detect status changes in status change detection items stored in the status change detection item table 33P in association with the identification information (customer ID) of the care recipient and the identification information (care item ID) of the care item performed while the video data was being filmed, creates a record fragment including a detection item change section in which a status change in the status change detection item was detected, and assigns adjuster annotation data for an evaluation request to the created record fragment, and performs processing to store the record fragment and the adjuster annotation data for an evaluation request in the care-related information storage means 30 as adjustment information.

[0341] In this embodiment, as shown in Fig. 19, the section detection process using the above-mentioned status change detection items is executed after the automatic extraction process using timing information for specifying a time point of interest. That is, the adjustment means 23 extracts data for a time point of interest from video data recorded including time periods other than the time point of interest (in the case of continuous recording in Figs. 20 and 21 and in the case of recording the entire care scene in Figs. 30 and 31) at an extraction position determined by the timing information for specifying a time point of interest to create a first record fragment, and then uses this first record fragment to create a second record fragment including a section in which a status change of the above-mentioned status change detection item was detected, and also assigns adjuster annotation data for an evaluation request to the created second record fragment, and executes a process of storing the second record fragment and the adjuster annotation data for an evaluation request as adjustment information in the care-related information storage means 30.

[0342] As already described in detail in the explanation of the care setting means 22, the status change detection item table 33P in FIG. 19 stores, in association with the ID of the care recipient currently being processed (in the example of FIG. 19, the customer ID of Tokkyo Daijiro), correspondences between the intentions and emotions of the care recipient and status change detection items for detecting visually and auditory recognizable physical status changes of the care recipient. It also stores information on which care items each of these correspondences appears in. In the example of FIG. 19, if care for the care item "suction" is performed, the settings are registered to associate the care item ID of this "suction" with the correspondences between the intention and emotion expression "anger" and status change detection items b and c, and the correspondences between the intention and emotion expression "disgust" and status change detection items b and c. However, no other correspondences are set for the care item "suction." Therefore, if care for the care item "suction" is performed, it is sufficient to detect status change detection items b and c from the many status change detection items a, b, c, d, e, etc. Alternatively, all the status change detection items may be detected, and then status change detection items b and c may be focused on.

[0343] As already described in detail in the explanation of the care setting means 22, the status change detection items a, b, c, d, e, ... shown in Figure 19 are status change detection items for detecting physical status changes that can be recognized visually and auditory, including the movements of each action unit (AU) such as raising the inner eyebrow, raising the outer eyebrow, lowering the eyebrow, and raising the upper eyelid, movements in the direction of gaze, changes in vocalizations and cries, etc.

[0344] 19 shows a first record fragment created by automatically extracting video data taken during suction care for the currently processed care recipient (Daijiro Tokken) using timing information for specifying a time point of interest. In this embodiment, the section detection process by the adjustment means 23 is performed by focusing on this time period.

[0345] The string of numerical data, such as 0.1, 0.1, 0.0, etc., written in chronological order for each status change detection item a, b, c, d, e, etc., is the recognition result (likelihood indicating the likelihood) obtained by performing pattern recognition processing for each status change detection item using the first record fragment (video data). A threshold is set for this likelihood, and sections that are equal to or exceed the threshold are defined as detection item change sections. In the example of FIG. 19, detection item change sections are detected for status change detection items a, b, and c. However, since status change detection item a is not registered for the care item "suction" in the status change detection item table 33P, only the detection item change sections for status change detection items b and c are of interest. Therefore, when creating a second record fragment solely through section detection processing using status change detection items, the range from the start position TbS of the detection item change section for status change detection item b to the end position TcE of the detection item change section for status change detection item c becomes the second record fragment.

[0346] Here, the pattern recognition processing for each of the state change detection items a, b, c, d, e, ... can be realized by existing technology. For example, the pattern recognition processing for the movements of various facial muscles called action units (AUs) can use the technology described in the aforementioned Non-Patent Document 1, and the implementation technology can employ the technology described in the aforementioned Non-Patent Document 5.

[0347] In addition, the pattern recognition processing for gaze direction movement can use the technology described in the aforementioned Non-Patent Document 2, and the implementation technology can be the technology described in the aforementioned Non-Patent Document 6.

[0348] Furthermore, the pattern recognition processing for facial movement can use the technology described in the aforementioned Non-Patent Document 3, and the implementation technology can be the technology described in the aforementioned Non-Patent Document 7.

[0349] The pattern recognition process for the movement of the skeleton and joint positions can be performed using the technology described in the aforementioned Non-Patent Document 4, and the implementation technology can be the technology described in the aforementioned Non-Patent Documents 8 and 9.

[0350] Furthermore, as an implementation technique for pattern recognition processing of changes in vocalizations and cries, for example, the technique described in the aforementioned Non-Patent Document 10 can be adopted.

[0351] Although not shown in the figure, the data collection server 20 is provided with a plurality of model storage means for storing each model for performing each of the above pattern recognition processes, and the adjustment means 23 executes the pattern recognition process (including the recognition process of gestures for conveying the intention of an evaluation request) using the models (trained parameters) stored in the model storage means prepared for each of these pattern recognition processes. Therefore, the data collection server 20 is provided with model storage means for, for example, action units (AUs), movements in the gaze direction, movements in the face direction, movements in the positions of skeletons and joint points, and changes in vocalizations and cries.

[0352] (Section detection process using sensor data: Figure 19)

[0353] The adjustment means 23 can also combine the section detection process using the sensor data output from the sensor 43 with the section detection process using the above-mentioned status change detection items. Whether or not to combine the section detection process using the sensor data can be selected by the care coordinator when pressing the "Execute section detection based cut-out process" button 765.

[0354] That is, as shown in FIG. 19, the adjustment means 23 uses the sensor data from the time period in which the first record fragment was extracted from the video data showing the condition of the care recipient to determine whether the data is larger or smaller than a predetermined threshold, or to determine whether the data is larger or smaller than a threshold set based on the past average value for the care recipient who was the subject of the video data, thereby detecting a sensor data change section in which the sensor data value changes significantly, and creates a second record fragment from the first record fragment that includes both the sensor data change section and the detection item change section (the section detected by the section detection process using the above-mentioned state change detection item), and also performs a process of assigning adjuster annotation data for an evaluation request to the created second record fragment and storing the second record fragment and the adjuster annotation data for an evaluation request in the care-related information storage means 30 as adjustment information.

[0355] In the example of FIG. 19, the "o" and "x" arranged in chronological order next to the output data of each sensor 43 indicate that the change is large and the change is small, respectively.

[0356] 19, in the condition change detection item table 33P, the pulse sensor (sensor type ID=SN01) is set and registered for the care item "suction," so the sensor data output from the pulse sensor is used to determine whether the sensor data is greater than or equal to a threshold, thereby detecting a sensor data change interval where the sensor data value changes significantly. Note that the example of FIG. 19 also lists sensor data change intervals detected using sensor data output from the incontinence sensor (sensor type ID=SN23), but in the condition change detection item table 33P, the incontinence sensor (sensor type ID=SN23) is only set and registered in association with the care item "injection," and a correspondence between the intention / emotion expression "fear" and the condition change detection items a and c. Therefore, here, attention will be focused only on the sensor data change interval for the pulse sensor (sensor type ID=SN01). Therefore, when combined (merged) with the interval (TbS~TcE) detected by the interval detection process using the above-mentioned state change detection items, the second record fragment ranges from the start position Tsn01S of the sensor data change interval for the sensor data output from the pulse sensor (sensor type ID=SN01) to the end position TcE of the detection item change interval for state change detection item c.

[0357] <Configuration of Data Collection Server 20 / Output Means 24>

[0358] The output means 24 executes output processing to display or transmit the care-related information stored in the care-related information storage means 30 on a terminal operated by a care provider, care coordinator, care evaluator, care recipient, or other relevant party with access authority, print it, output it as audio, transmit it to other linked systems, or write it to a recording medium.

[0359] The album viewing means 24A executes a process for accepting viewing by care persons of the care-related information stored in the album storage means 32. Specifically, in response to a viewing request from a device operated by a care provider, or a care recipient or their input agent (personal computer (PC) 52, mobile devices 53, 54 in FIG. 2), an adjuster terminal 60, an evaluator terminal 70, or any other terminal of a care person with access authority, the album viewing means 24A transmits display data for a care record display screen 900 shown in FIG. 18 via the network to the terminal that made the viewing request, and displays the care record display screen 900 on the terminal.

[0360] The care record display screen 900 in Fig. 18 is a screen displayed on the display means 47 of the device (PC 52, mobile devices 53, 54 in Fig. 2) operated by the care provider, or the care recipient or their input agent, the display means of the adjuster terminal 60 or evaluator terminal 70, or the display means of a terminal operated by another person involved in care, and the information displayed on this screen 900 is stored in the album database 33 of the album storage means 32. Note that, unlike the adjuster screen 700 in Fig. 16 displayed on the adjuster terminal 60 by the adjustment means 23 and the evaluator screen 800 in Fig. 17 displayed on the evaluator terminal 70 by the evaluation result collection means 26, this care record display screen 900 is not a screen that may be displayed in response to an emergency signal.

[0361] The care record display screen 900 in Figure 18 displays the name, date of birth, gender, care theme, etc. of the care recipient specified at the time of the viewing request, and displays the date on which the care was provided, as well as the details of the care provided to the care recipient (care session ID, care scene ID, name of the care item, start time of the care scene, end time of the care scene, name of the care provider, etc.).

[0362] 18 also displays, for video data etc. recorded during the implementation of each care, the image ID, file storage path, file length (length of video data or audio data), file start time and file end time (start and end times of shooting or recording video data or audio data stored in the file), urgency, etc. Clicking on the file storage path displays a display window for video data etc. stored in file storage means 34 of album storage means 32, allowing playback of the video etc.

[0363] Furthermore, the care record display screen 900 of Figure 18 chronologically displays the annotation data assigned to the video data recorded during the implementation of each care, as well as the names of the people who assigned the annotation data. In the example of Figure 18, the name of the care provider and the care provider's intent to request an evaluation (provider annotation data), the name of the care coordinator and coordinator annotation data for the care coordinator's request for an evaluation, coordinator annotation data for the response to the care provider, the name of the care evaluator and the care evaluator's evaluation results (evaluator annotation data) and additional annotation data by the care evaluator, the name of the care provider and additional annotation data by the care provider, and the name of the care coordinator and additional annotation data by the care coordinator are displayed in this order.

[0364] Furthermore, the call time and the call intention are displayed as subject needs information on the care record display screen 900 of Fig. 18. The order in which this subject needs information is displayed is also in chronological order in the order in which the calls were made, and is also in chronological order in relation to the care provided.

[0365] The electronic medical chart creation data transmission means 24B uses the care-related information stored in the album storage means 32 of the care-related information storage means 30 to output electronic medical chart creation data that can be used to create an electronic medical chart and transmit it to the electronic medical chart creation system 80 via a network. This electronic medical chart creation data includes at least the start care item information and care scene start time to be placed in the electronic medical chart, information on the storage location (file path stored in the file storage means 34) of video data, still image data, or audio data showing the state of the care recipient to be used as link information to be attached to the electronic medical chart, and each annotation data, including adjuster annotation data and evaluator annotation data, assigned to the video data, still image data, or audio data placed near the link information. The electronic medical chart creation data is indicated by a black star in Figures 3 to 5.

[0366] The care need calculation data transmission means 24C uses the care-related information stored in the album storage means 32 of the care-related information storage means 30 to output care need calculation data required to calculate nursing need, care need, or other care need levels, and transmits the data via a network to a care need management system 81, such as a nursing need management system. This care need calculation data includes at least the start care item information and care scene start time as provider-sent information, and the call time as subject-sent information, or the call intention in addition to the call time. The care need calculation data is indicated by black circles in Figures 4 and 5.

[0367] <Configuration of Data Collection Server 20 / Album Registration Means 25>

[0368] The album registration means 25 executes a process for accepting registration of care-related information by a care provider into the album storage means 32 of the care-related information storage means 30. Specifically, in response to a registration request from a device operated by a care provider, a care recipient or their input agent (personal computer (PC) 52, mobile devices 53, 54 in FIG. 2), an adjuster terminal 60, an evaluator terminal 70, or another terminal of a care provider who has registration authority, the album registration means 25 transmits display data for a registration screen (not shown) via the network to the terminal that made the registration request, causing the registration screen to be displayed on the terminal, and receives event-related information input on the registration screen via the network and stores it in the album storage means 32.

[0369] The event-related information accepted for registration by the album registration means 25 is not information that is requested to be input by an emergency signal, unlike the adjustment information (adjuster annotation data for evaluation request) entered on the adjuster screen 700 of Figure 16 displayed on the adjuster terminal 60 by the adjustment means 23, or the evaluation results (evaluator annotation data) entered on the evaluator screen 800 of Figure 17 displayed on the evaluator terminal 70 by the evaluation result collection means 26.

[0370] The event-related information accepted for registration by this album registration means 25 includes, for example, additional annotation data to be added to video data, etc. (additional annotation data by the care provider, care recipient or their input agent, care coordinator, and care evaluator), as well as comments on the care as a whole (comments on the care as a whole by the care provider, care recipient or their input agent, care coordinator, and care evaluator), rather than annotation data to be added to individual video data, etc., as shown in Figure 3.

[0371] Then, when the album registration means 25 receives additional annotation data, it creates a new record in the evaluation and addition table 33H (see FIG. 14). Of the columns in this evaluation and addition table 33H, the case ID, care session ID, care scene ID, care item ID, related party placement ID, placement attribute ID, status data type, and status data ID (ID of the adjusted status data) are the same as those in the status data management table 33G (see FIG. 13) and are copied from the status data management table 33G. The inputter attribute ID stores selection information of the person who input the additional annotation data. The inputter ID stores the ID of the person who input the additional annotation data. The column of "evaluator annotation data (initial response including the evaluation result to the evaluation request), or other additional annotation data" stores input data by the latter additional inputter.

[0372] Furthermore, when the album registration means 25 receives a comment, it creates a new record in a comment table (not shown) of the album database 33. In this comment table, comment data is stored in association with the case ID.

[0373] <Configuration of Data Collection Server 20 / Evaluation Result Collection Means 26>

[0374] The evaluation result collection means 26 transmits the adjustment information stored in the care-related information storage means 30 (adjuster annotation data for evaluation requests stored in the album database 33 of the album storage means 32, and adjusted files stored in the file storage means 34) via a network to the evaluator terminal 70 operated by the care evaluator stored in the related party allocation table 33B, which is the related party allocation storage means, and receives evaluator annotation data including the evaluation results by the care evaluator who referred to the adjustment information from the evaluator terminal 70 via the network, and performs a process of associating the received evaluator annotation data with the adjustment information as care-related information and storing it in the care-related information storage means 30 (album database 33 of the album storage means 32).

[0375] This processing by the evaluation result collection means 26 is executed after the adjustment information is created and registered by the adjustment means 23, but at least the notification to the care evaluator is executed immediately after the adjustment information is registered in the care-related information storage means 30. That is, in this embodiment, regardless of whether an emergency signal is received by the care event information collection means 21, once the adjustment information is registered in the care-related information storage means 30 by the adjustment means 23, the evaluation result collection means 26 immediately notifies the care evaluator operating the evaluator terminal 70 by push notification that the adjustment information to be evaluated has been registered (including, in the case of an emergency, that fact). This push notification method is arbitrary and can be performed using, for example, a mobile phone number, a social networking service (SNS) account, an email address, or the like stored in the customer table 33L or the service provider table 33M (see FIG. 15 ). Alternatively, a push notification can be sent using a function of a general-purpose tool such as an SNS or email client to notify the evaluator terminal 70 that adjustment information to be evaluated has been registered (including this fact in the case of an emergency), and the care evaluator who sees the notification can then enter information on an input screen or web screen (e.g., evaluator screen 800 in FIG. 17 ) displayed by a dedicated application program according to the present invention. Alternatively, rather than using a function of a general-purpose tool to send a push notification, the push notification can be sent using a function of the dedicated application program according to the present invention installed on the evaluator terminal 70. Whether or not a care evaluator who receives a push notification immediately performs evaluation work on the evaluator terminal 70 depends on the judgment and situation of the care evaluator, but at least in the case of an emergency, the care evaluator is notified of the urgency.

[0376] Specifically, the evaluation result collection means 26 transmits display data for the evaluator screen 800 shown in Fig. 17 to the evaluator terminal 70 via the network, displays this evaluator screen 800 on the screen of the evaluator terminal 70, receives the evaluation results (evaluator annotation data) input by the care evaluator on this evaluator screen 800 via the network, and executes processing to store them in the album database 33. The evaluator screen 800 may be a web screen (which may include JavaScript (Java is a registered trademark), etc.) or a screen displayed by a dedicated application program installed on the evaluator terminal 70.

[0377] In Figure 17, the evaluator screen 800 has a display section 801 for the care recipient, a display section 802 for the care items, a display section 803 for the care provider who sent information about an event that occurred at the care location, a display section 804 for the care coordinator who sent the evaluation request, a display section 805 for the date and time of the transmission, a display section 806 that indicates an emergency when an emergency signal is received, a display section 810 for the video (adjustment information) to be evaluated, a "Start playback" button 811 for starting playback of the video, a "Back" button 812 for rewinding the video, a "Stop" button 813 for stopping the video, and a display section 815 for coordinator annotation data (adjustment information) for the evaluation request from the care coordinator.

[0378] 17 also includes an input section 820 for inputting the evaluation results (evaluator annotation data) of the care evaluator who has referred to the video (adjustment information) on the display section 810 and confirmed the adjuster annotation data (adjustment information) for the evaluation request displayed on the display section 815, an "audio input" button 821 for inputting the results by voice, a "view album" button 830, and a "send" button 831 for transmitting the input data to the evaluation result collection means 26 via the network. When the care evaluator presses the "view album" button 830, the care evaluator can refer to the care-related information about the care recipient currently being processed that is stored in the album storage means 32, and can input the evaluation results (evaluator annotation data) after viewing the progress up to that point.

[0379] Then, when the evaluation result collection means 26 receives evaluator annotation data including the evaluation results by the care evaluator, it creates a new record in the evaluation and addition table 33H (see FIG. 14). Of the columns in this evaluation and addition table 33H, the case ID, care session ID, care scene ID, care item ID, related party placement ID, placement attribute ID, status data type, and status data ID (ID of the adjusted status data) are the same as those in the status data management table 33G (see FIG. 13) and are copied from the status data management table 33G. The inputter attribute ID stores the selection information of the care evaluator. The inputter ID stores the ID of the care evaluator who actually entered the data. The column for "evaluator annotation data (initial response including the evaluation results in response to the evaluation request) or other added annotation data" stores the input data by the former care evaluator.

[0380] <Configuration of Data Collection Server 20 / Care-Related Information Storage Means 30>

[0381] The care-related information storage means 30 stores care-related information including the care event information collected by the care event information collection means 21 and additional information obtained thereafter.

[0382] As shown in FIGS. 1 and 2 , the temporary storage unit 31 stores and accumulates the following care event information collected by the care event information collection unit 21: video data continuously captured by the image capture unit 41 and transmitted as streaming data; audio data continuously captured by the audio recording unit 42 and transmitted as streaming data; and sensor data continuously measured by the sensor 43 and transmitted sequentially; in association with a care location ID (a care location ID may not be necessary if the data collection system 10 is for each facility) and device identification information for each device. However, the video data, audio data, and sensor data stored in the temporary storage unit 31 are not stored for a long period of time but are stored temporarily. After being stored for a certain period of time, such as three months, the oldest data is deleted. This is similar to the facility recorder 44 installed in the facility. Therefore, the video data, audio data, and sensor data stored in the temporary storage unit 31 are cut into record fragments by the adjustment unit 23, as shown in FIG. 2 , and then stored in the file storage unit 34 of the album storage unit 32 after being filed.

[0383] The album storage means 32 stores data for information sharing for a long period of time, which is what makes it different from the temporary storage means 31. The data stored in this album storage means 32 is made available for viewing by all care workers who have viewing authority via the adjustment means 23, the album viewing means 24A of the output means 24, and the evaluation result collection means 26, as shown in FIG.

[0384] 6 and 2, the album database 33 is made up of a plurality of tables, and stores data (text data) other than the moving image data, still image data, audio data, and sensor data that are filed and stored in the file storage means 34, out of the data stored in the album storage means 32. The data stored in this album database 33 includes storage destination paths of files stored in the file storage means 34, as shown in FIG.

[0385] As shown in FIG. 6, the case table 33A stores a facility ID (care location ID), a case ID, a case name (care theme), a customer ID of a care recipient, a name of a care recipient, and the like in association with each other.

[0386] 8 and 9, the related party allocation table 33B is related party allocation storage means, and stores, in association with each other, the case ID (ID for each care theme to respond to the cause of hospitalization, admission, etc.), related party allocation ID (ID of the allocation of personnel, etc. when the positions of related parties change depending on the content of care, and is distinguished between basic allocation and various modified allocations), allocation attribute ID (distinguishing between primary care and various secondary care), ID of the care recipient, ID of care providers 1, 2, ..., ID of care coordinators 1, 2, ..., ID of care evaluators 1, 2, ..., ID of other related parties 1, 2, ..., etc. Note that when a table is shared by multiple facilities (care delivery locations), the facility ID (care delivery location ID) is also stored in association with each other.

[0387] As shown in Figure 10, the care item master table 33C stores care type IDs and care type names (e.g., nursing, caregiving, pet sitting, etc.) in association with care item IDs and care item names (e.g., toileting, bathing, suction, diaper changes, treats, dog walking, cat feeding, etc.).

[0388] 10, the care session setting table 33D stores, in association with each other, a case ID, a care session ID, a care session name (for example, "basic morning session"), and care item IDs for the first, second, third, fourth, etc. care situations. Although not shown in the figure, the table also stores, in association with each other, a related party placement ID and a placement attribute ID. If the table is to be shared by multiple facilities (care locations), the table also stores, in association with each other, a facility ID (care location ID).

[0389] The device / application settings table 33E is a device / application settings storage means for storing data selected and input on the device / application settings screen 600 shown in Fig. 11. Specifically, the imaging / audio recording device type ID (and, if necessary, device identification information for each device) that identifies the type of imaging means 41 and audio recording means 42 selected in the selection section 610 in Fig. 11, and the corresponding usage scenario ID that identifies the usage scenario selected in the selection section 611 are stored in association with the case ID. In addition, the input means / application ID that identifies the input means 46 and application program selected in the selection sections 620 to 650 in Fig. 11 are stored in association with the case ID. Note that, if a table is to be shared by multiple facilities (care locations), the facility ID (care location ID) is also stored in association with the case ID.

[0390] As shown in FIG. 12, the care history table 33F stores and associates the following information: case ID, care session ID, care scene ID, care item ID, related party placement ID, placement attribute ID, care recipient ID, care provider ID, care coordinator ID, care scene start time, care scene end time, care location ID (if a table is shared among multiple facilities (care locations)), number of images and audio, number of sensor data, etc. The care item ID is the ID of the care item accepted as the start care item information included in the provider action information of the provider-sent information, and the care scene start time and care scene end time are also included in the provider action information. Therefore, each record constituting this care history table 33F corresponds to a care scene.

[0391] As shown in FIG. 13, the status data management table 33G stores the following information: a case ID, a care session ID, a care scene ID, a care item ID, a related party arrangement ID, an arrangement attribute ID, a status data type (video, audio, still image, sensor data), a status data ID (ID of the adjusted status data), a degree of urgency (urgent or non-urgent), a care coordinator ID (ID of the care coordinator who actually made the adjustment), a transmission date and time of the timing information for specifying the time point of interest, a time elapsed from the time point of interest to the time point of transmission indicated by the relative time of the timing information for specifying the time point of interest (specified), and a time point of interest. The table stores, in association with each other, the duration (designation) of the timing information for point designation, the automatic extraction start position (the time automatically calculated by the adjustment means 23), the adjusted extraction start position (the time adjusted by the adjustment means 23), the automatic extraction end position (the time automatically calculated by the adjustment means 23), the adjusted extraction end position (the time adjusted by the adjustment means 23), the evaluation request intent, the adjuster annotation data for the evaluation request, the adjuster annotation data for the response to the care provider, the care evaluator to whom the evaluation request is to be sent, the file storage path, and so on. Each record constituting this status data management table 33G corresponds to each status data (video, audio, still image, and sensor data). In other words, even if the video data is from the same care scene, if the video data to be evaluated is different, it will be a different record. Note that if a table is shared by multiple facilities (care locations), the facility ID (care location ID) is also stored in association with each other.

[0392] As shown in FIG. 14, the evaluation and addition table 33H stores, in association with each other, the case ID, care session ID, care scene ID, care item ID, related party placement ID, placement attribute ID, status data type, status data ID (ID of adjusted status data), inputter attribute ID (1: first response as care evaluator (first response including evaluation results in response to an evaluation request), 2: second or subsequent response as care evaluator, 3: response as care provider, 4: response as care recipient, 5: response as input agent for care recipient, 6: second or subsequent response as care coordinator, 7: response of other related parties), inputter ID, evaluator annotation data (first response including evaluation results in response to an evaluation request), or other additional annotation data. Note that if this table is to be shared by multiple facilities (care delivery locations), the facility ID (care delivery location ID) is also stored in association with each other.

[0393] Here, "3: Response as a caregiver" in the inputter attribute ID refers to, for example, when answering a counter-question from another caregiver, or when the caregiver asks a question in this system and then happens to learn the answer through a phone call from a care evaluator (such as the care recipient's mother), or when the caregiver ultimately derives the answer from past information and answers the question themselves. "4: Response as a care recipient" refers to, for example, when answering a counter-question from another caregiver, or when a care recipient who was sleeping wakes up and answers the question themselves. "5: Response as an input agent for the care recipient" refers to, for example, when answering a counter-question from another caregiver.

[0394] 6, the call history table 33K stores the case ID, care recipient ID, call time (year, month, day, hour, minute, second), and call intention (text data of the voice recognition result or identification information such as the call intention number) in association with each other. Note that if the table is to be shared by multiple facilities (care locations), the facility ID (care location ID) is also stored in association with each other.

[0395] As shown in Figure 15, customer table 33L stores, in association with each other, the name, date of birth, customer ID or customer assistant ID, customer attribute ID (whether the customer is the customer or their relationship to the customer), address, telephone number (preferably a mobile phone number that allows push notifications), email address, My Number or microchip identification information (in the case of living creatures), file storage path for facial photo data, service provider ID (if available), customer ID if the customer assistant also becomes a customer, and customer assistant ID if the customer also becomes a customer assistant, for a customer who has become a care recipient or a care-related person (customer assistant) who can act as an entry agent for the care recipient. Although not shown in the figure, SNS (social networking service) accounts that allow push notifications may also be stored.

[0396] 15, the service provider table 33M stores, in association with each other, the name, date of birth, service provider ID, service provider attribute ID (nurse, doctor, care worker, care manager, helper, etc.), address, telephone number (preferably a mobile phone number capable of sending push notifications), email address, My Number, file storage path for facial photo data, customer ID or customer auxiliary ID (if possessed), etc., of facility staff or care personnel at the facility. Although not shown in the figure, the service provider table 33M also stores accounts for tools capable of sending push notifications, such as SNS (social networking service) accounts, accounts for dedicated communication tools used within the facility, and accounts for dedicated communication tools used by facility staff both inside and outside the facility.

[0397] As shown in FIG. 6, the authority table 33N stores a login ID, a login password, authority details, and the like in association with each other.

[0398] 19, the status change detection item table 33P is a status change detection item storage means, and stores, in association with each other, the identification information (client ID) of the care recipient, an intention / emotion expression ID that identifies the intention / emotion expression, a care item ID, a status change detection item ID, and a sensor type ID that identifies the type of sensor 43. When the table is shared by multiple facilities (care locations), the facility ID (care location ID) is also stored in association with each other.

[0399] The file storage means 34 stores files that contain video data, still image data, audio data, and sensor data from among the data stored in the album storage means 32. As shown in Fig. 2, the files stored in this file storage means 34 include files of video data, audio data, and still image data collected (received) in file form by the care event information collection means 21, files of video data, audio data, and adjusted sensor data that are adjustment information created by the automatic extraction process and section detection process (see Fig. 19) by the adjustment means 23, and files of video data, audio data, still image data, and adjusted sensor data that are adjustment information created by the adjustment means 23 in response to an adjustment job by a care coordinator.

[0400] <Configuration of Data Collection Server 20 / Application Storage Means 35>

[0401] The application storage means 35 stores various application programs (see FIGS. 22 and 34) to be installed in a device (input means 46) operated by a care provider, a care recipient, or their input agent.

[0402] <Data flow in device layout (A) for continuous recording: Figures 20 to 29>

[0403] In the equipment arrangement (A) for continuous recording in Figures 20 to 29, as described in the explanation of (Equipment arrangement (A) for continuous recording (24 hours, or with on / off): Figures 20 to 29) in the above-mentioned <Outline and terminology of data collection system 10>, recording (photography and audio recording) is always performed in real time transmission regardless of whether a care scene is taking place or whether a point of interest has occurred.

[0404] (Explanation of Figure 20)

[0405] In Figure 20, in equipment layout (A), the status data acquisition means 40 uses, as the photographing means 41 and the recording means 42, a fixed camera 50 (50A) installed at a fixed position within the facility, a camera mounted on a robot that circulates within the facility, a mobile camera 51 (51B) such as a camera that circulates along a guide rail within the facility, a web camera 52A and a microphone 52B connected to a personal computer (PC) 52 installed within the facility, and a sensor 43.

[0406] The video data and audio data recorded (photographed and recorded) by these photographing means 41 and recording means 42 are streamed over the network and received by the care event information collection means 21, while the sensor data output from the sensor 43 is also sequentially transmitted over the network and received by the care event information collection means 21, and each is stored in the temporary storage means 31.

[0407] Thereafter, the adjustment means 23 executes an automatic cutting process for creating a first record fragment, a section detection process for creating a second record fragment (see Figure 19), etc., and the created second record fragment is stored as adjustment information in the file storage means 34 of the album storage means 32 (see Figure 2).

[0408] In addition, the input means 46 operated by the care provider may be a mobile device 53 (see Figure 2) such as a smartphone or tablet (including a tablet loaned to the facility) operated by the care provider, a personal computer (PC) 52 (see Figure 2), or an input operation device 55 (55A, 55B, 55D, etc.).

[0409] The care provider behavior information (start care item information, care scene start time, care scene end time) and care provider annotation data (evaluation request intention, timing information for specifying the point of interest, and emergency signal if necessary) input by these input means 46 are transmitted via the network, received by the care event information collection means 21, and each stored in the album database 33 of the album storage means 32.

[0410] Furthermore, the input means 46 operated by the care recipient or their input representative may be a mobile device 54 (see Figure 2) such as a smartphone or tablet operated by the care recipient or their input representative, a personal computer (PC) 52 (see Figure 2), a dedicated input operation device 55 (55A), an input operation device 55 (55C, 55D) that can also be used as a call device (selectable, mode switching type), an input operation device 55 (55E) that can also be used as a call device (simultaneous type), or a call device 56.

[0411] The subject annotation data (evaluation request intention, timing information for specifying the point of interest, and emergency signal if necessary) and subject needs information (call time, call intention) input by these input means 46 are transmitted via the network, received by the care event information collection means 21, and each stored in the album database 33 of the album storage means 32.

[0412] (Explanation of Figure 21)

[0413] In Figure 21, in the equipment arrangement (A) for continuous recording, the timing information for specifying the time point of interest (executor annotation data or subject annotation data) input by the input means 46 is stored in the status data management table 33G (see Figure 13), and this timing information for specifying the time point of interest is used to perform an automatic extraction process to create the first record fragment.

[0414] The care provider, care recipient, or their input representative specifies a retroactive time indicating how far back the time of interest TB (more precisely, the representative time TB of the time period of the time period) was from the time TA when the timing information for designating the time period of interest was input (the time period when the timing information for designating the time period of interest was transmitted), i.e., the elapsed time ΔT1 from the time of interest TB to the time TA when the timing information for designating the time period of interest was transmitted, and this is used as the care provider annotation data or the care recipient annotation data. In addition, the care provider, care recipient, or their input representative specifies the time length of the time period of the time period of interest (the extraction length of the automatic extraction process) ΔT2, and this is used as the care provider annotation data or the care recipient annotation data.

[0415] The representative time TB of the time slot of the time of interest is usually the center position of the time slot of the time of interest (ΔT3 = ΔT4 = ΔT2 / 2), but the ratio of the lengths of ΔT3 and ΔT4 can be set arbitrarily as long as ΔT3 + ΔT4 = ΔT2. Also, either ΔT3 or ΔT4 may be set to zero. Therefore, when the elapsed time ΔT1 and the time length (cut-out length of the automatic cut-out process) ΔT2 are specified, the automatic cut-out start position TC and the automatic cut-out end position TD are calculated.

[0416] Furthermore, in the example of FIG. 21, record fragment P and record fragment Q are automatically extracted within a care scene for care item A (e.g., bathing). In this way, automatic extraction processing of multiple record fragments (first record fragments) may be executed within one care scene. In this case, the two record fragments P and Q may depict the same state of a single care recipient, or may depict completely different states. Even within one care scene, the content of care changes, so different states may appear. Also, even if the state is similar, another image may be taken just to be sure.

[0417] Furthermore, in the example of FIG. 21, a care scene for care item A (e.g., bathing) and a care scene for care item B (e.g., toileting) are captured by one imaging means 41 (e.g., mobile camera 51A that moves around the facility). In this case, the care recipient in the care scene for care item A and the care recipient in the care scene for care item B may be the same or different. If the care recipients are different, multiple care scenes for different care recipients will be captured by one imaging means 41 (e.g., mobile camera 51A that moves around the facility). Also, even when capt...

Claims

1. A computer-based data collection system for collecting data relating to the provision of medical, nursing, caregiving, assistance, or health care or similar care, comprising: a care setting means for receiving a setting input of the placement of people or living beings or alternative inanimate objects for assigning roles to the parties involved in the care, including a care provider who provides the care, a care recipient who receives the care provided by the care provider, a care coordinator who coordinates care event information related to the care, and a care evaluator who receives an evaluation request from the care coordinator, and storing the input in a party placement storage means; a care event information collecting means for receiving input of the care event information, which includes video data, still image data, or audio data showing the state of the care recipient recorded in a care scene in which the care provider provides the care to the care recipient, and provider-sent information including the care provider's evaluation request intention as provider annotation data to be added to these data; a care-related information storage means for storing care-related information including the care event information collected by the care event information collection means and additional information obtained thereafter; an adjustment means for supporting or substituting the adjustment work of the care coordinator stored in the related party allocation storage means for the care event information stored in the care-related information storage means, to create adjustment information including video data, still image data, or audio data for an evaluation request and adjuster annotation data for an evaluation request to be assigned to these data, and to store the created adjustment information in the care-related information storage means as the care-related information; an evaluation result collection means for transmitting the adjustment information stored in the care-related information storage means to an evaluator terminal operated by the care evaluator and stored in the related party allocation storage means via a network, receiving evaluator annotation data including the evaluation results by the care evaluator who referenced the adjustment information from the evaluator terminal via the network, and storing the received evaluator annotation data in the care-related information storage means in association with the adjustment information as the care-related information; an output means for executing an output process of displaying or transmitting the care-related information stored in the care-related information storage means on a screen of a terminal operated by the care provider, the care coordinator, the care evaluator, the care recipient, or other related parties who have access rights, printing it out, outputting it as audio, transmitting it to another linked system, or writing it into a recording medium; A data collection system comprising:

2. The participant location storage means The system is configured to be able to store the arrangement of the care provider, the care coordinator, and the care evaluator when providing primary care and the arrangement of the care provider, the care coordinator, and the care evaluator when providing secondary care for the same care recipient in association with each other in a state where they are different arrangements of related parties.

2. The data collection system of claim 1.

3. The secondary care is model care that a person other than the primary care provider performs as the secondary care provider to the care recipient, so that the primary care provider can use it as a reference when performing the primary care; The participant location storage means includes: The person who will be the care provider of the primary care is stored as the care assessor or care coordinator of the secondary care, or other party with reference authority, The care provider of the primary care can refer to the video data or still image data for model reference registered in the secondary care and stored in the care-related information storage means in the position of the care evaluator or care coordinator of the secondary care, or other related person with reference authority.

3. The data collection system of claim 2.

4. The secondary care is consent confirmation auxiliary care in preparation for the primary care to be performed later, and obtains consent for the implementation of the primary care from the care recipient in advance, The participant location storage means includes: The person who will be the care provider of the primary care is stored as the care evaluator of the secondary care, The care provider of the primary care can refer to the video data or still image data showing how consent was obtained from the care recipient, which is registered in the secondary care and stored in the care-related information storage means, in the position of the care evaluator of the secondary care, confirm whether consent has been obtained, and use the confirmation result as the evaluator annotation data of the secondary care.

3. The data collection system of claim 2.

5. The participant location storage means The system is configured to be able to store, as the care provider, a student who will be training or practicing the care, as the care recipient, a doll that resembles a child, elderly person, or other person in need of care who will be the subject of the care training or practice, as the care coordinator, a teacher or instructor who will be teaching the care, and as the care evaluator, a guardian or caretaker of the person in need of care.

2. The data collection system of claim 1.

6. The care setting means includes: The system is also configured to execute a process of accepting a setting input of a care item indicating the type of care that the care provider may provide to the care recipient in each care situation or the type of care that the care provider plans to provide in each care situation, The care event information collection means The care event information includes information about a start time of a care event, which is a care item that the care provider will start implementing from among the care items preset by the care setting means, and a care event start time.

2. The data collection system of claim 1.

7. The care event information collection means In addition to receiving input of the implementer transmission information constituting the care event information, as subject transmission information constituting the care event information, The system is configured to accept input of video data, still image data, or audio data showing the state of the care recipient recorded in a non-care scene other than the care scene, and the input of an evaluation request intention of the care recipient or their input representative as subject annotation data to be added to these data.

7. The data collection system of claim 6.

8. The care event information collection means In addition to receiving input of the implementer transmission information constituting the care event information, as the subject transmission information constituting the care event information, The system is configured to accept input of subject needs information including a call time when the care recipient or their input representative presses a button on a call device or an input operation device having the function of this call device to call the care provider, or including the call intention in addition to this call time.

8. The data collection system of claim 7.

9. The care setting means includes: The intentions and emotional expressions of the care recipient that may be expressed in the care situation or that may be expressed in the care situation in which a specific care item among the care items indicating the type of care is performed; Accepting a setting input regarding a correspondence relationship with any one or a combination of two or more of the condition change detection items for detecting a visually and auditorily recognizable physical condition change of the care recipient, including the movement of each of the plurality of facial muscles of the care recipient, the movement of the gaze direction, the movement of the face direction, the movement of the position of the skeletal and joint points, or the movement of other body parts of a human or living thing, or a change in the vocalization or cry of the care recipient, or a change in an intangible or tangible object emanating from the body of another human or living thing; The received correspondence relationship is stored in a condition change detection item storage means in association with the identification information of the care recipient, or in association with the identification information of the care recipient and the identification information of the care item in the condition change detection item storage means, The adjusting means is Using the video data showing the state of the care recipient, a pattern recognition process is performed to detect the state change of the state change detection items stored in the state change detection item storage means in association with the identification information of the care recipient who was the subject of the video data recording, or to detect the state change of the state change detection items stored in the state change detection item storage means in association with the identification information of the care recipient and the identification information of the care items performed while the video data was being recorded, The system is configured to create a record fragment including a detection item change section in which the state change of the state change detection item is detected, and to add adjuster annotation data for the evaluation request to the created record fragment, and to execute a process of storing the record fragment and the adjuster annotation data for the evaluation request in the care-related information storage means as the adjustment information.

2. The data collection system of claim 1.

10. The video data included in the implementer transmitted information is The video data is continuously recorded including a time point of interest indicating a time period when the care provider judges that attention is required for the state of the care recipient, and a time period other than the time point of interest, The provider-sent information includes timing information for specifying a time point of interest that is assigned by the care provider as the provider annotation data to the video data recorded including a time period other than the time point of interest, The adjusting means is The system is configured to: extract data for the time point of interest from the video data recorded including the time period other than the time point of interest at an extraction position determined by timing information for specifying the time point of interest to create a first record fragment; then, use this first record fragment to create a second record fragment including a section in which the status change of the status change detection item was detected; add adjuster annotation data for the evaluation request to the created second record fragment; and store the second record fragment and the adjuster annotation data for the evaluation request in the care-related information storage means as the adjustment information.

10. The data collection system of claim 9.

11. The care event information collection means The care event information is configured to receive input of sensor data output from a sensor that measures biological information of the care recipient, The adjusting means is The sensor data for the time period in which the first record fragment was extracted from the video data is used to determine whether the sensor data is large or small relative to a predetermined threshold, or whether the sensor data is large or small relative to a threshold set based on a past average value for the care recipient who was the subject of the video data. This is then used to detect a sensor data change section in which the value of the sensor data changes significantly, and the second record fragment is created from the first record fragment, including both the sensor data change section and the detection item change section. Adjuster annotation data for the evaluation request is assigned to the created second record fragment, and the second record fragment and the adjuster annotation data for the evaluation request are stored in the care-related information storage means as the adjustment information.

11. The data collection system of claim 10.

12. The video data included in the implementer transmitted information is The video data is continuously recorded including a time point of interest indicating a time period when the care provider judges that attention is required for the state of the care recipient, and a time period other than the time point of interest, The provider-sent information includes timing information for specifying a time point of interest that is assigned by the care provider as the provider annotation data to the video data recorded including a time period other than the time point of interest, The timing information for designating a time point of interest is the elapsed time from the time point of interest to the time point at which the care provider designates the timing information, and the duration of the time point of interest. The adjusting means is The system is configured to execute a process of extracting data for the time period of interest from the video data recorded including the time period other than the time period of interest at an extraction position determined by timing information for specifying the time period of interest to create a record fragment, and to add adjuster annotation data for the evaluation request to the created record fragment, and to store the record fragment and the adjuster annotation data for the evaluation request in the care-related information storage means as the adjustment information.

2. The data collection system of claim 1.

13. The video data included in the target person transmitted information is The video data is continuously recorded including a time point of interest indicating a time period when the care recipient or their input agent determines that attention is required for the state of the care recipient, and a time period other than the time point of interest, The subject transmission information includes timing information for specifying a time point of interest that is assigned by the care recipient or their input representative as the subject annotation data to the video data recorded including a time period other than the time point of interest, The adjusting means is the video data recorded including the time period other than the time period of interest is cut out at a cutout position determined by the timing information for specifying the time period of interest to create a record fragment, and the adjuster annotation data for the evaluation request is added to the created record fragment, and the record fragment and the adjuster annotation data for the evaluation request are stored in the care-related information storage means as the adjustment information, The care event information collection means In addition to receiving input of the implementer transmission information constituting the care event information, as the subject transmission information constituting the care event information, The input of the care recipient's needs information includes a call time when the care recipient or their input representative presses a button on a call device or an input operation device having the function of the call device to call the care provider, or includes a call intention in addition to the call time; The system is configured to accept input of timing information for designating the attention point, which is given by the care recipient or their input representative pressing a button on the call device or the input operation device.

8. The data collection system of claim 7.

14. the motion picture data or the audio data included in the implementer transmitted information includes gesture or audio data for conveying an evaluation request intention corresponding to the implementer annotation data, The adjusting means is The gesture or voice data for transmitting the evaluation request intention is converted into evaluation request intention identification information or text data indicating the evaluation request intention.

2. The data collection system of claim 1.

15. The video data included in the implementer transmitted information includes both gesture and voice data for conveying an evaluation request intention, which correspond to the implementer annotation data, and The adjusting means is The system is configured to convert each of the gesture and voice data for transmitting the evaluation request intention into evaluation request intention identification information or text data indicating the evaluation request intention, integrate the conversion result of the gesture and the conversion result of the voice data, and execute a process of obtaining final evaluation request intention identification information or text data indicating the evaluation request intention.

2. The data collection system of claim 1.

16. the target person needs information includes the call time and the call intention; The call intention is indicated by gestures or voice data for conveying the call intention included in the video data or the voice data that continuously records the state of the care recipient not only in the care scene but also in a time period other than the care scene, The adjusting means is The system is configured to extract data from the video data or audio data that is continuously recorded, including time periods other than the care scene, for the time period before and after the call time when a button on the call device or the input operation device having the function of this call device is pressed, and to convert the gesture or audio data for communicating the call intention contained in the extracted data into call intention identification information or text data that indicates the call intention.

9. The data collection system of claim 8.

17. The output means The care-related information stored in the care-related information storage means is used to execute a process of outputting data for creating an electronic medical record that can be used to create an electronic medical record, The electronic medical record creation data includes at least The start care item information and the start time of the care scene to be placed in the electronic medical record; Information on the storage location of the video data, the still image data, or the audio data showing the state of the care recipient, which is used as link information to be attached to the electronic medical record; The annotation data includes the adjuster annotation data and the evaluator annotation data that are assigned to the video data, the still image data, or the audio data that are arranged in the vicinity of the link information.

7. The data collection system of claim 6.

18. The output means The care-related information storage means is configured to execute a process of outputting care need calculation data required for calculating nursing need, care need, or other care need using the care-related information including the start care item information and the care scene start time as the provider-sent information stored in the care-related information storage means, and the call time as the subject-sent information, or the call intention in addition to the call time.

9. The data collection system of claim 8.

19. The care setting means includes: The device is configured to receive setting inputs of devices and applications used by the care provider, the care recipient, or an input representative thereof, and to execute a process of storing the input setting information of the devices and applications used in a device and application setting storage means; The care event information collection means The application is distributed via a network to a device used by the care provider, the care recipient, or the care recipient's input agent, according to the setting information of the device and application stored in the device and application setting storage means.

2. The data collection system of claim 1.

20. The care event information collection means The implementer annotation data is configured to be capable of receiving an input of an emergency signal for requesting an urgent evaluation, The input means operated by the care provider is Whether or not the emergency signal is to be sent can be controlled by the number of times a button is pressed. If a predetermined time has passed after a first pressing operation without a second pressing operation being performed, the emergency signal is sent to the care event information collecting means. If a second pressing operation is performed within the predetermined time after the first pressing operation, the sending of the emergency signal is avoided.

2. The data collection system of claim 1.

21. The adjusting means is When creating the coordination information, the care event information stored in the care-related information storage means is transmitted to a coordinator terminal operated by the care coordinator via a network; The system is configured to receive, in addition to the adjustment instructions from the care coordinator for creating the adjustment information and the coordinator annotation data for requesting an evaluation from the care evaluator, advice from the care coordinator to the care provider or other response coordinator annotation data transmitted from the coordinator terminal via a network, and to execute a process of storing this response coordinator annotation data in the care-related information storage means in association with the adjustment information as the care-related information.

2. The data collection system of claim 1.

22. A program for causing a computer to function as the data collection system according to any one of claims 1 to 21.

Images