Information input system and control method thereof, and record management apparatus and control method thereof

Abstract

To provide an information input system and method for easily retrieving a record including information generated on the basis of an image.SOLUTION: The information input system includes a client device, a record management server, and a database server. The record management server transmits an AI command of the health condition of the client based on the client image and a generation command of the keyword based on the health condition descriptive text generated according to the AI command to the generating and analyzing device together with the client image (step 74), and stores the hashtag including the keyword generated by the generating and analyzing device in the database of the database server in association with the health condition descriptive text (step 78). Thus, the burden of data entry is reduced by the generation of the health condition description sentence by the settlement agent device generation and AI device, and data search is facilitated by using the keyword as a hashtag.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information input system and method, and also to a record management device and a control method thereof. [Background technology]

[0002] Patent Document 1 discloses an electronic medical record creation support device that infers information about symptoms (chief complaints, etc.) from audio data and video data and reflects this information in an electronic medical record. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-135178

[0004] The creation of electronic medical records is supported by incorporating information obtained from audio and video data into the electronic medical records. However, Reference 1 does not at all consider making it possible to easily search for a specific electronic medical record among a large number of electronic medical records. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention aims to make it possible to easily search for records containing information generated based on image data.

[0006] The information input system of the present invention comprises a client device, a record management server and a database server, wherein the client device comprises an image data transmission means for transmitting user image data representing a user image to the record management server, the record management server transmits to a generation AI device, together with the user image data received from the client device, an analysis command for a user's health condition based on the user image represented by the user image data and an instruction to generate keywords based on a health condition description generated in accordance with the analysis command, and a generation AI device utilization means for receiving the health condition description and keywords generated by the generation AI device in accordance with the analysis command and generation command, and a transmission means for transmitting the health condition description and keywords generated by the generation AI device utilization means to a database server, and the database server comprises a storage means for storing the health condition description in association with a classification tag including the keyword.

[0007] The present invention also provides a control method for an information input system, comprising a client device, a record management server, and a database server, wherein the client device transmits user image data representing a user image to the record management server via an image data transmission means, the record management server transmits, via a generation AI device utilization means, to the generation AI device an instruction to analyze the user's health condition based on the user image represented by the user image data and an instruction to generate keywords based on a health condition description generated in accordance with the analysis instruction, along with the user image data received from the client device, receives the health condition description and keywords generated by the generation AI device in accordance with the analysis instruction and generation instruction, transmits the generated health condition description and keywords via a transmission means to a database server, and the database server associates a classification tag including the keywords with the user's health condition description and stores it in a storage means.

[0008] A user image represented by user image data sent from a client device is given to a generation AI device along with an instruction to analyze the user's health condition based on the user image, and the health condition description (analysis result) generated by the generation AI device is stored in a database server. Since the generation AI device assists in inputting the health condition description stored in the database server, the burden of data input is reduced.

[0009] According to this invention, in addition to an instruction to analyze a user's health condition based on a user image, an instruction to generate keywords based on a health condition description generated in accordance with the analysis instruction is also given to the generation AI device. Classification tags including the keywords generated by the generation AI device are associated with the health condition description and stored in the storage means of the database server, so that specific health condition descriptions can be easily searched for using the classification tags.

[0010] Preferably, the health condition description is part of a record form created for each user and stored in the storage means of the database server, and the classification tag is associated with the record form. A specific record form can be searched for using the classification tag from among the multiple record forms stored in the storage means.

[0011] In one embodiment, the client device includes an imaging means, and a recent image of the user captured by the imaging means can be used to generate the health status description.

[0012] Preferably, the user image includes a facial image of the user. This can improve the accuracy of analysis of the user's health condition based on the user image using the generation AI device.

[0013] The present invention also provides a record management device that can constitute the above-mentioned information input system. The record management device according to the present invention comprises: a processing request statement creation means for creating a processing request statement including an analysis command for a user's health condition based on a user image represented by user image data representing a given user image, and an instruction to generate keywords based on a health condition description generated in accordance with the analysis command; and a generation AI device utilization means for transmitting the processing request statement created by the processing request statement creation means together with the user image data to a generation AI device, and for receiving the health condition description and keywords generated by the generation AI device in accordance with the analysis command and generation command.

[0014] According to this invention, when user image data is provided (including reading), a processing request statement is created that includes an instruction to analyze the user's health condition based on the user image represented by the user image data and an instruction to generate keywords based on a health condition description generated in accordance with the analysis instruction, and the created processing request statement and the provided user image data are sent to a generation AI device. The generation AI device generates a health condition description and keywords in accordance with the processing request statement and receives them. Because the health condition description is created by the generation AI device, the burden of data entry of the health condition description is reduced. Furthermore, by associating classification tags containing keywords generated by the generation AI device with health condition descriptions also generated by the generation AI device, specific health condition descriptions can be easily searched for using the classification tags.

[0015] The present invention also provides a method for controlling the above-mentioned record management device. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a hardware configuration of a care content input system. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of a client device. [Figure 3] 1 shows a health care record form. [Figure 4] Shows user image. [Figure 5] 1 shows a record editor window with a healthcare record form displayed. [Figure 6] 1 shows the record editor window with a user image displayed. [Figure 7] 10 is a flowchart showing an image analysis process. [Example]

[0017] FIG. 1 is a block diagram showing the hardware configuration of the care content input system.

[0018] The care content input system includes a client device 11, a record management server 12, a generation AI device 13, and a database server 14. The database server 14 is connected to a record database 15 that stores record data about users that is input using the care content input system.

[0019] The client device 11 is used to input various care (support) details provided to users (typically elderly people or people with disabilities) of a care business (care facility) (hereinafter simply referred to as "business") by staff (employees, etc.) and to record (save) the input care details (care records). The care records are used for examining care plans for users of the business, utilizing them for high-quality care services and care management, improving the qualifications of business staff, sharing information, etc. The care detail input system is used in common at multiple business offices and is operated by staff from each of the multiple business offices. Therefore, multiple client devices 11 operated by multiple staff members are used to input care details (care records).

[0020] As mentioned above, there are multiple business establishments, each of which generally has multiple users (elderly people or people with disabilities) affiliated with it. Care (support) for each of the multiple users at the multiple business establishments is provided almost daily, and records are kept almost daily. The recorded care (support) details include numerous items (care items), such as the user's vital data (body temperature, blood pressure, percutaneous arterial oxygen saturation, etc.), dietary details (amount of food eaten), oral care, height, weight, exercise, visitors, and recreational details. The care (support) details provided to or measured for each user are entered into a care content input system and recorded (saved). The care (support) details recorded in the care content input system include not only details related to personal care for the user, but also the user's own personal records (the above-mentioned body temperature, height, weight, etc.), and items to report.

[0021] A record management server 12 is connected to the client device 11 via a network (such as the Internet). The record management server 12 manages the care (support) provided and to be provided to each user of the facility (record management, schedule management).

[0022] In this embodiment, the generation AI device 13 is used to automate record entry and generate classification tags (hashtags) used for data classification, as described below, and can be, for example, ChatGPT, copilot, or other generation AI devices.

[0023] Fig. 2 is a block diagram showing the hardware configuration of the client device 11. Fig. 3 shows a health management record form 31, which is one of the record forms for inputting (recording) care (support) details. A record form such as that shown in Fig. 3 is prepared in advance, and care (support) details and the like are entered into the input fields using the record form.

[0024] 2, client device 11 includes a CPU 20B that comprehensively controls the overall operation of client device 11, a RAM 20C that temporarily stores data and programs, storage (SSD, HDD, etc.) 20D that stores an operating system, various application programs, etc., an input device (keyboard, etc.) 20E for inputting characters, numbers, etc., a communication device 20F that sends and receives data via a network, a microphone 20G that inputs sound (voice), a speaker 20H that outputs sound, a camera 20I that captures an image of a subject, and a display device 20J that displays a recording form, etc., all of which are connected to a data bus 20A. A portable computer device that can be connected to a network can be used as client device 11.

[0025] The record management server 12 and the database server 14 are also computers equipped with a CPU, RAM, storage, communication devices, etc. However, unlike the client device 11, these servers 12 and 14 are not computer devices that are directly operated by employees of the business, and therefore do not necessarily require input devices such as a keyboard, a display device, a microphone, a speaker, or a camera.

[0026] Various data entered and recorded using the care content input system can be tagged (assigned, associated) with hashtags (classification tags). Figure 3 shows a schematic diagram of how hashtags 31A to 31C with the keywords (hashtag names) "bright eyes," "even skin tone," and "calm facial expression" are tagged to a health management record form 31 created on a specific day (date and time) for a specific user.

[0027] The health management record form 31 shown in Figure 3 is created basically daily for each user who uses a care facility. By tagging each (not necessarily all) of the care (support) details that are entered and recorded daily for each of the many users at many facilities with a hashtag, it becomes easier to search for the care (support) details at a later date. A hashtag can be associated with each record form created; for example, if a search is made for a health management record form 31 with the hashtag "#brighteyes" (this can be limited to those for a specific user or within a specific date and time range), the search will return health management record forms 31 tagged with the hashtag "#brighteyes" from among the many health management record forms 31.

[0028] Figure 4 shows a user image 32 that is managed integrally (associated with) the health management record form 31 shown in Figure 3. For example, the health management record form 31 shown in Figure 3 can be treated as the front side of the form, and the user image 32 shown in Figure 4 can be treated as the back side of the form.

[0029] It is not necessary to associate a different user image 32 with each record form (for example, each health management record form 31 created daily for a specific user), but the user image 32 may be managed in association with the user (for example, a user ID that identifies the user). In this case, for example, the latest user image 32 for a specific user is associated with the health management record form 31.

[0030] Referring to the health management record form 31 in Fig. 3, the health management record form 31 includes a condition observation field 31D. In addition to characters (text) input by a staff member or the like using the input device 20E of the client device 11, text automatically generated from the user image 32 shown in Fig. 4 can also be input into this condition observation field 31D. The process of automatically generating text to be input into the condition observation field 31D based on the user image 32 shown in Fig. 4 will be described in detail below.

[0031] 5 and 6 show a record editor window 61 displayed on the display screen of the display device 20J of the client device 11, which is used when entering care details into a record form using the client device 11.

[0032] The record editor window 61 has a form display field 61A on the left side. Various pre-prepared record forms are displayed in the form display field 61A, and data can be entered into the input fields of the displayed record form. Figure 5 shows the record editor window 61 with the health management record form 31 (Figure 3) described above displayed in the form display field 61A.

[0033] In the right area of ​​the record editor window 61, a record form selection field 61B, a hashtag display field 61C, and a hashtag input field 61D are displayed side by side in the vertical direction.

[0034] The names of multiple types of record forms (record form names) that are prepared in advance are displayed in the record form selection field 61B. When a specific record form name is selected (clicked) in the record form selection field 61B, the record form corresponding to that record form name is displayed in the form display field 61A on the left. When "health management" is selected in the record form selection field 61B, the health management record form 31 is displayed in the form display field 61A in an editable format.

[0035] The hashtag display field 61C displays hashtags tagged to the specific health management record form 31 displayed in the form display field 61A. Figure 5 shows how three hashtags, "#bright eyes," "#even skin tone," and "#calm expression," are tagged to the specific health management record form 31 displayed in the form display field 61A.

[0036] The hashtags displayed in the hashtag display field 61C, i.e., the hashtags associated with the health management record form 31, are automatically input (generated) by analyzing the image represented by the image data, as will be described later. The automatic input of hashtags will be described later in detail.

[0037] The hashtags displayed in the hashtag display field 61C can be modified (including deleted) or added by an employee of the business who is operating the record editor window 61 (client device 11). The hashtag input field 61D is used to modify or add the hashtags displayed in the hashtag display field 61C.

[0038] The particular health management record form 31 displayed in the form display field 61A of the record editor window 61 shown in Figure 5 is tagged with the hashtags "#bright eyes," "#even skin tone," and "#calm expression," and will therefore be found in a form search using any one (or multiple) of these hashtags.

[0039] The record form displayed in the form display field 61A of the record editor window 61, such as the health care record form 31 shown in Fig. 5 as an example, includes a plurality of data entry fields into which data can be entered (edited). Staff members of the business use the record form displayed in the record editor window 61 to enter (record) details of care (support) for the user, the user's own personal records, etc.

[0040] The record editor window 61 shown in Fig. 6 differs from the record editor window 61 shown in Fig. 5 in that a user image 32 is displayed in its form display field 61A instead of the health management record form 31 (Figs. 3 and 5). By clicking the switch icon 62 at the bottom left of the record editor window 61, it is possible to switch between the display of the record editor window 61 shown in Fig. 5 and the display of the record editor window 61 shown in Fig. 6. As described above, the user image 32 shown in Fig. 6 is displayed like the back side of the health management record form 31 shown in Fig. 5.

[0041] A rotation icon 32A, an image analysis icon 32B, an image data save icon 32C, a trash can icon 32D, a folder icon 32E, and a camera icon 32F are displayed superimposed on the user image 32.

[0042] The rotation icon 32A is clicked to rotate and display the user image 32 displayed in the form display field 61A. Each time the rotation icon 32A is clicked, the user image displayed in the form display field 61A is rotated 90° to the right and displayed.

[0043] The image analysis icon 32B is clicked to perform an image analysis of the user image 32 displayed in the record editor window 61, automatically input text into the condition observation field 31D of the health management record form 31, and automatically generate the above-mentioned hashtag. Details of the image analysis process will be described later.

[0044] The image data save icon 32C is clicked when image data representing the user image 32 displayed in the form display field 61A is to be saved in the storage 20D of the client device 11. When the record editor window 61 is launched, the target user of the care record, etc. is specified in advance, and the image data used for the user image 32 displayed in the form display field 61A is image data that is downloaded from the database server 14 to the client device 11 via the record management server 12 when the record editor window 61 is launched and that is temporarily stored in the RAM 20C of the client device 11. When the image data save icon 32C is clicked, a copy of the image data that is temporarily stored in the RAM 20C is stored in the storage 20D of the client device 11.

[0045] The trash can icon 32D is clicked when image data representing the user image 32 is to be deleted from both the RAM 20C of the client device 11 and the record database 15 connected to the database server 14. When the trash can icon 32D is clicked, the display of the user image 32 disappears.

[0046] The folder icon 32E is clicked when image data stored in the storage 20D of the client device 11 is to be used as the user image 32. When the folder icon 32E is clicked, a file manager application provided in the operating system of the client device 11 is launched. When the desired image data is selected, the image represented by the selected image data is displayed in the form display field 61A as the user image 32, and is recorded in the record database 15 connected to the database server 14.

[0047] The camera icon 32F is clicked when capturing an image of a subject, in this case, a user. When the camera icon 32F is clicked, the camera 20I included in the client device 11 is activated. An image represented by image data showing the captured image of the subject is displayed in the form display field 61A as the user image 32 and is recorded in the recording database 15 connected to the database server 14.

[0048] FIG. 7 is a flowchart showing the flow of the image analysis process (automatic condition observation input process, automatic hash tag generation process) executed in the care content input system.

[0049] When the image analysis icon 32B, one of the multiple icons superimposed on the user image 32 displayed in the form display field 61A of the record editor window 61, is clicked, the image analysis process shown in Fig. 7 starts. Image data representing the displayed user image 32 is uploaded from the client device 11 to the record management server 12 via the network (step 71).

[0050] To facilitate the analysis of the user's health condition, image data including the user's facial image is preferably uploaded. Of course, it is not necessary to include a facial image; for example, image data of the user's skin condition or injured areas may be uploaded as user image data to be analyzed.

[0051] The image data represents a specific user who is using a care facility, etc., but other information than the image data that can identify that specific user (such as name or ID) is not transmitted over the network together with the image data to the record management server 12. As will be explained below, the image data will be transmitted to the generation AI device 13, but the user's personal information is protected by not transmitting any personal information other than the image data.

[0052] The record management server 12, which has received the user image data from the client device 11, creates the following processing request statement (prompt) and sends it to the generation AI device 13 together with the received user image data (step 74). The processing request statement includes a command to request analysis of the user image and a command to generate (extract) keywords, and has the following content, for example:

[0053] {"message":[{"role":"system","content":"Accurately analyze the health status of users in the image. Please understand and explain health information from how the user appears in the image, such as facial expression, skin tone, and posture.\n #Request details\n 1. Identify health characteristics from the user's face, facial expression, and general appearance in the image. Analyze in detail the condition of the skin, eyes, mouth, etc. From the user's posture and movements, provide information on muscle and joint abnormalities and posture health. Finally, summarize the information and list it as the "Health Analysis Results." 2. From the analysis, please list health-related keywords separated by commas. "}],"max_tokens":800,"temperature":0,"top_p":0,"frequency_penalty":0,"presence_penalty":0}

[0054] The generation AI device 13, which has received the processing request statement and the user image data, analyzes the user image 32 represented by the received user image data based on the analysis request command included in the processing request statement, and generates a statement representing the analysis results (health analysis results).The AI ​​device 13 also uses the generated analysis result statement to generate (extract) keywords contained therein (step 77).

[0055] The analysis results and keywords generated by the generation AI device 13 are, for example, as follows: Two examples, Example 1 and Example 2, are shown below.

[0056] <Example 1> Health analysis results: The client's face has a calm expression and healthy, even skin tone. The eyes are bright and the lips are rosy, reflecting good health. The client's posture is upright and there are no signs of abnormalities in the muscles or joints. General appearance suggests that the client is in good health. keyword: Gentle facial expression, even skin tone, bright eyes, rosy lips, upright posture, normal muscles and joints

[0057] <Example 2> Health analysis results: Signs of dermatitis appear, with redness and swelling of the skin. If you notice puffiness or dark circles under your eyes, this may be the result of lack of sleep or stress. Healthy sleep and stress management are essential. keyword: Swelling under the eyes, lack of sleep, dermatitis

[0058] The health analysis results and keywords generated by generation AI device 13 are transmitted to record management server 12, which requested the image analysis, and record management server 12 transmits the received health analysis results and keywords to client device 11 (step 75). In client device 11, the received health analysis results are entered (displayed) in status observation field 31D of health management record form 31, which is displayed in form display field 61A of record editor window 61 (see FIG. 5). In addition, the received keywords are displayed as hashtags in hashtag display field 61C of record editor window 61 (step 72) (see FIGS. 3 and 5). Because the user's health analysis results generated by generation AI device 13 based on user image 32 are automatically entered into status observation field 31D of health management record form 31, the burden of data entry work on staff and the like is reduced.

[0059] The text of the health analysis results generated by the generation AI device 13 and automatically entered into the condition observation column 31D of the health management record form 31 can be edited by the business staff or the like using the input device 20E of the client device 11, and can be corrected as necessary. Clicking the save button 61E in the record editor window 61 proceeds to the save process (step 73).

[0060] When Save button 61E in record editor window 61 is clicked, the data (form input data) entered in health management record form 31 displayed in form display field 61A of record editor window 61 and the hash tag (keyword) entered in hash tag display field 61C are sent to database server 14 via record management server 12 (step 76). Database server 14 associates the received form input data, hash tag, user identification ID, save date and time, form name, etc. with each other and records them in record database 15 (recording of health management record form data) (step 78).

[0061] Health management record form data tagged with hashtags is recorded in the record database 15, so that, as described above, it is possible to easily search for only those health management record forms 31 tagged with a specific hashtag among a large number of health management record forms 31 using the hashtag. [Explanation of symbols]

[0062] 11 Client Device 12 Records Management Server 13 Generation AI device 14 Database Server 15 Record Database 31 Health Care Record Form 31A, 31B, 31C hashtags 32 User Images

Claims

1. a client device, a record management server, and a database server; The client device: an image data transmission means for transmitting user image data representing a user image to a recording management server; The records management server a generation AI device utilization means for transmitting to the generation AI device, together with the user image data received from the client device, an analysis command for the user's health condition based on the user image represented by the user image data, and an instruction to generate keywords based on the health condition description generated in accordance with the analysis command, and receiving the health condition description and keywords generated by the generation AI device in accordance with the analysis command and the generation command; and a transmission means for transmitting the health condition description and keywords generated by the generation AI device utilization means to a database server, The database server a storage means for storing the description of the user's health condition in association with a classification tag including the keyword; Information input system.

2. the health condition description is a part of a record form created for each user and stored in the storage means of the database server; the classification tag is associated with the record form; The information input system according to claim 1 .

3. the client device is equipped with an imaging means, The information input system according to claim 1 .

4. The user image includes a facial image of the user. The information input system according to claim 1 .

5. A method for controlling an information input system including a client device, a record management server, and a database server, comprising: The client device: transmitting user image data representing the user image to the record management server by the image data transmitting means; The records management server The generation AI device utilization means transmits to the generation AI device, together with the user image data received from the client device, an analysis command for the user's health condition based on the user image represented by the user image data, and an instruction to generate keywords based on the health condition description generated in accordance with the analysis command, and receives the health condition description and keywords generated by the generation AI device in accordance with the analysis command and generation command; a transmitting means for transmitting the generated health condition description and keywords to a database server; The database server Associating the user's health condition description with a classification tag including the keyword and storing the same in a storage means; A method for controlling an information input system.

6. a processing request statement creation means for creating a processing request statement including an analysis command for a user's health condition based on a user image represented by the given user image data and an instruction for generating keywords based on a health condition description generated in accordance with the analysis command; a generation AI device utilization means for transmitting the processing request statement created by the processing request statement creation means to the generation AI device together with the user image data, and receiving the health condition description statement and keywords created by the generation AI device in accordance with the analysis command and the generation command; Record-keeping devices.

7. a processing request statement creation means for creating a processing request statement including an analysis command for the user's health condition based on the user image represented by the given user image data and an instruction for generating keywords based on the health condition description generated in accordance with the analysis command; A generation AI device utilization means transmits the processing request statement created by the processing request statement creation means to the generation AI device together with the user image data, and receives the health condition description statement and keywords created by the generation AI device in accordance with the analysis command and the generation command. A method for controlling a record management device.

Images