Medical support system and medical support method

Abstract

To provide medical treatment support information appropriate for incident and accident prevention.SOLUTION: A disclosed medical treatment support system is adapted to implement processing of: recognizing one or a plurality of objects to be used in a medical treatment from image data; identifying a medical practise corresponding to the recognized one or the plurality of objects; and providing medical treatment information corresponding to the identified medical practise for a user.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a medical support system and a medical support method.

Background Art

[0002] Patent Document 1 discloses an information providing system that provides necessary information such as operation procedures of medical devices to a user who is a medical staff at a medical site. Patent Document 1 discloses that a user can wear glasses and perform work smoothly while checking necessary information.

[0003] The information providing system described in Patent Document 1 acquires image data of a specific tool existing within a visible range of a user, determines whether the state of the tool included in the image data is correct, and transmits the determination result to the user's glasses.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] The information providing system of Patent Document 1 merely provides operation procedures of medical devices as information required at a medical site.

[0006] However, only operation procedures of medical devices may not be sufficient as information provided at a medical site.

[0007] In the medical field, incidents and accidents can occur. For this reason, in order to prevent incidents and accidents, medical staff not only need to understand the operating procedures of equipment, but also need to be constantly aware of the precautions they need to take when performing medical procedures on patients. However, busy medical staff are not always able to be aware of the precautions they need to take when performing medical procedures.

[0008] To prevent incidents and accidents, for example, a supervising physician could accompany the patient and provide instructions on precautions. However, it is difficult for a supervising physician to always accompany the patient in busy medical settings. In addition, even experienced medical personnel may be affected by their physical and mental conditions or fatigue.

[0009] Therefore, it is desirable to provide appropriate medical support information.

[0010] One aspect of the present disclosure is a medical support system configured to execute a process including recognizing one or more objects used in medical treatment from image data, identifying medical procedures corresponding to the recognized one or more objects, and presenting medical support information corresponding to the identified medical procedures to a user.

[0011] Another aspect of the present disclosure is a medical support method. The disclosed medical support method is a medical support method executed by a medical support system, which recognizes one or more objects used in medical care from image data, identifies a medical procedure corresponding to the recognized one or more objects, and presents medical support information corresponding to the identified medical procedure to a user. Be prepared to do so.

[0012] Further details will be described in the following embodiments. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a configuration diagram of a medical support system. [Diagram 2]FIG. 2 is a block diagram showing a medical support process executed by the medical support system. [Diagram 3] FIG. 3 is a diagram showing the structure of the database. [Figure 4] FIG. 4 is a diagram showing an example of the medical material-medical procedure table and the medical support information database. [Diagram 5] FIG. 5 is a flowchart of the object recognition process. [Figure 6] FIG. 6 is a flowchart of the medical support information output process. [Figure 7] FIG. 7 is a timing chart showing an example of the operation of the medical support system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] <1. Overview of the medical support system and medical support method>

[0015] (1) A medical support system according to an embodiment is configured to execute a process including recognizing one or more objects used in medical care from image data, identifying a medical procedure corresponding to the recognized one or more objects, and presenting medical support information corresponding to the identified medical procedure to a user. Since this medical support system is configured to identify the medical procedure corresponding to one or more objects used in medical care, it is possible to present the medical support information corresponding to the medical procedure to a user.

[0016] (2) It is preferable that identifying the medical procedure is identifying the medical procedure corresponding to the one or more objects appearing in the image data from among a plurality of medical procedure candidates including a first medical procedure associated with a first object and a second medical procedure associated with a second object other than the first object and the first object. In this case, the first medical procedure and the second medical procedure share a common object, the first object, but the first medical procedure and the second medical procedure can be distinguished from each other and medical support information corresponding to each medical procedure can be presented to the user.

[0017] (3) When the first object appears but the second object does not appear at a first time point in the image data and the second object appears at a second time point after the first time point, it is preferable to identify a first medical procedure corresponding to the first object before the second time point and identify a second medical procedure corresponding to the first object and the second object after the second time point. In this case, it is possible to identify an appropriate medical procedure corresponding to an object recognized up to each time point.

[0018] (4) The process is preferably configured to repeatedly identify the medical procedure in a plurality of cycles including a first cycle and a second cycle following the first cycle, and to not present the medical support information to the user in the second cycle if the medical procedure identified in the second cycle is the same as the medical procedure identified in the first cycle. In this case, it is possible to prevent the medical support information from being unnecessarily presented to the user repeatedly.

[0019] (5) It is preferable that the medical support information presented to the user is determined according to the attributes of the user. In this case, user support information according to the attributes of the user can be presented to the user. The attributes of the user are, for example, attributes of the user who is a medical professional, and more specifically, a doctor or a nurse.

[0020] (6) The process preferably further comprises estimating a posture of a patient receiving the medical procedure from the image data, and the medical support information presented to the user preferably includes information based on the estimated posture. In this case, the information based on the patient's posture can be presented to the user.

[0021] (7) It is preferable that the processing further includes determining compatibility between at least one of the recognized object and the identified medical procedure and a patient receiving the medical procedure, and that the medical support information presented to the user includes information based on the compatibility determination result.

[0022] (8) It is preferable that the process further includes presenting the identified medical procedure together with the medical support information to a user. In this case, the user can recognize the medical procedure identified by the medical support system and can judge the appropriateness of the presented medical support information.

[0023] (9) In the process, it is preferable that, when an instruction to cancel presentation of the presented medical support information is received from the user, the medical support information is made non-presentable. In this case, the presented medical support information can be made non-presentable in response to a user instruction.

[0024] (10) The medical support information is preferably presented to the user by a wearable device worn by the user, the wearable device preferably including a camera for acquiring the image data. In this case, the image data is acquired by the camera of the wearable device worn by the user, so that a medical scene according to the position where the user is present is captured. Therefore, even if the medical scene is a medical procedure performed at the patient's bedside or a medical procedure performed as a home medical examination, the medical scene can be captured, and the user can easily obtain medical support information according to the medical scene.

[0025] (11) A medical support method according to an embodiment is preferably a medical support method executed by a medical support system, and includes recognizing one or more objects used in medical care from image data, having the medical support system identify a medical procedure corresponding to the recognized object or objects, and presenting medical support information corresponding to the identified medical procedure to a user.

[0026] <2. Examples of medical support systems and medical support methods>

[0027] Hereinafter, an embodiment of the present invention will be described in more detail with reference to the drawings. The following description is a preferred example and is not intended to limit the present invention.

[0028] FIG. 1 shows a medical support system 100 (hereinafter simply referred to as "system 100") according to an embodiment. The system 100 presents medical support information to a user 10. The user is, for example, a medical professional such as a doctor or a nurse. The system 100 may include a server 200 and a user terminal 300.

[0029] The system 100 is configured to execute a medical support process 60 (see FIG. 2). The medical support process 60 may be executed by both the user terminal 300 and the server 200 in a shared manner, or may be executed by the user terminal 300 alone. The server 200 may be responsible for most of the medical support process 60, and the user terminal 300 may be responsible only for user interface functions such as image data acquisition and medical support information output. In the following, the medical support process 60 will be described as being executed by both the user terminal 300 and the server 200 in a shared manner. Of the medical support process 60, the process executed by the server 200 is referred to as a first medical support process 60A, and the process executed by the user terminal 300 is referred to as a second medical support process 60B. The user terminal 300 and the server 200 are connected to each other via a network 450 so as to be able to communicate with each other.

[0030] The server 200 is configured by a computer including a processor 210 and a storage device 220. The storage device 220 is connected to the processor 210. A computer program 230 is stored in the storage device 220. The storage device 220 may simply be called a memory 220. The storage device 220 includes, for example, a primary storage device and a secondary storage device. The primary storage device is, for example, a RAM. The secondary storage device is, for example, a hard disk drive (HDD) or a solid state drive (SSD). The processor 210 reads out the computer program 230 stored in the storage device 220 and executes the first medical support process 60A. The computer program 230 has a program code for the first medical support process 60A.

[0031] The storage device 220 of the server 200 includes a database 400. Part or all of the database 400 may be stored in the storage device 320 of the user terminal 300 or the storage device of another computer connected via the network 450. The database 400 will be described later.

[0032] The user terminal 300 includes a processor 310 and a storage device 320. A computer program 330 is stored in the storage device 320. The storage device 320 may simply be referred to as the memory 320. The storage device 320 is connected to the processor 310. The storage device 320 includes, for example, a primary storage device and a secondary storage device. The processor 310 reads out the computer program 330 stored in the storage device 320 and executes the second medical support process 60B. The computer program 330 has program codes for the second medical support process 60B.

[0033] The user terminal 300 is preferably, for example, a wearable device worn by the user 10. The wearable device preferably includes a display 350 for presenting information to the user. The wearable device is preferably worn on the head of the user 10. The wearable device worn on the head is, for example, smart glasses. The smart glasses include a camera 340 and a retinal projection display 350. The smart glasses can display information superimposed on the actual scenery that is the user's field of view. For this reason, the smart glasses are suitable for users such as medical staff to obtain information while performing medical procedures.

[0034] The smart glasses, which are the user terminal 300, include a main body 301 configured in a glasses shape, and the smart glasses include the aforementioned camera 340 and display 350. The camera 340 is provided on the main body 301 so as to image a range overlapping the field of view of the user 10. The display 350 is a retinal projection type and is provided on the main body 301 so as to be able to present information to the user superimposed on the user's field of view.

[0035] The smart glasses, which are the user terminal 300, further include a speaker 360, a microphone 370, an operation unit 380, and a communication interface 390. The speaker 360 outputs medical support information and the like as sound. The microphone 370 is used for voice input of user instructions and the like. The operation unit 380 is composed of buttons and the like provided on the main body 301, and accepts manual operations from the user 10 to accept user instructions. The communication interface 390 is used for communication with the outside. The communication interface 390 is used, for example, for communication with the server 200 via a network 450. The network 450 includes, for example, a wireless LAN. The wireless LAN is, for example, an in-hospital LAN constructed in a hospital, an in-facility LAN constructed in a facility where the patient is staying, or an in-home LAN constructed in the patient's home. The network 450 may include other networks such as the Internet.

[0036] The user terminal 300 worn by the user 10 captures, by the camera 340, images of a medical scene performed by the user 10. A medical scene is a scene of a medical procedure performed on a patient by a medical professional. The medical procedure is performed, for example, in an examination room, a treatment room, an examination room, a hospital room, a facility where the patient is staying, or the patient's home. The medical procedure is, for example, a procedure or examination performed on a patient.

[0037] In an embodiment, the medical procedure is performed using one or more objects used in medical treatment, such as medical devices. The objects used in medical treatment are, for example, medical devices, medicines administered to patients, drugs used in medical treatment, etc. Hereinafter, these objects used in medical treatment are also collectively referred to as "medical materials."

[0038] Medical procedures include those that use only one medical material and those that use multiple medical materials (for example, the first medical device 31 and the second medical device 32 in FIG. 1). Also, even when the same medical material is used, the medical procedures may be different. For example, a central venous catheter can be used for both insertion and removal of the catheter. Central venous catheter insertion and central venous catheter removal should be regarded as different medical procedures, although both use a central venous catheter. Also, in the case of more general-purpose medical instruments (for example, a curette), they may be used for a variety of medical procedures.

[0039] Also, in a medical scenario, within the range (field of view) visible to the user 10, medical materials may appear in the medical procedures performed on the patient. Also, the patient 20 may appear within the range visible to the user 10. In a medical scenario, the patient 20 may be present, for example, on a table 40 on which the patient 20 should lie in order to receive medical treatment, such as a bed, an examination table, or a treatment table. The camera 340 provided in the smart glasses captures the medical scenario within the field of view visible to the user 10. Therefore, medical materials and the patient may appear in the image data (medical scenario video data) captured by the camera 340. The range captured by the camera 340 may be part or all of the user 10. The image data is preferably a video, but may also be a still image.

[0040] In the embodiment, since the camera 340 is attached to the user 10, the imaging range varies depending on the posture or position of the user 10. Therefore, not all of the medical materials and patients present in the medical scenario may appear in the image data at the same time.

[0041] In a medical scene, medical materials or other objects that are unrelated or have little relevance to the medical procedure performed on the patient may appear in the range (field of vision) viewed by the user 10. For example, a treatment room in which a certain medical procedure is about to be performed may have stocks of medical materials for other medical procedures. In this way, objects that are unrelated or have little relevance to the medical procedure performed on the patient may appear in the image data captured by the camera 340 worn by the user 10.

[0042] In the medical scene described above, the system 100 according to the embodiment identifies a medical procedure from image data capturing the visual field of a user 10 who is about to perform the medical procedure on a patient, and presents medical support information corresponding to the identified medical procedure in real time to the user 10. Therefore, the user 10 can obtain information that is useful for performing the medical procedure.

[0043] The medical support information may include, for example, warning information indicating matters that medical personnel should be aware of in order to prevent incidents or accidents that may occur in the identified medical practice. Therefore, the system 100 may compensate for the lack of experience and knowledge of medical personnel. Furthermore, the system 100 may prevent careless mistakes, etc., made by busy medical personnel.

[0044] The medical support information may also include procedure instruction information that indicates the appropriate procedure for a medical procedure. The medical support information may also include patient data that indicates the patient's medical condition. By including such information in the medical support information, medical personnel can obtain even more useful information.

[0045] The medical support information may be presented in a form that the user can visually recognize, that is, a screen display, or in a form that the user can audibly recognize, that is, a voice output. In this embodiment, the medical support information is presented mainly by a screen display, and voice output is also performed in addition. This point will be described later.

[0046] FIG. 1 shows an example of display data 500 displayed on the display 350. The display data 500 is displayed by the smart glasses superimposed on the scenery of the medical scene viewed by the user 10. The display data 500 may include, for example, a medical procedure 510, a level 520, and medical support information 530. The medical procedure 510 included in the display data 500 is a medical procedure identified by the system 100. The level 520 is, for example, a level indicating the importance of the medical support information 530 to be displayed, and may be called an alert level. The medical support information 530 presented to the user is determined according to the level of the user 10. The level of the user 10 is determined according to the experience of the user 10, etc., and is set in advance in the system 100.

[0047] Fig. 2 shows the configuration of medical support processing 60 executed by the system 100. The medical support processing 60 is configured to output medical support information from image data (medical scene video) captured by the camera 340. The medical support processing 60 shown in Fig. 2 includes an object recognition processing 601, a medical procedure identification processing 602, and a medical support information output processing 610. The medical support processing 60 shown in Fig. 2 also includes a posture estimation processing 603 and an age / gender estimation processing 604.

[0048] Here, as an example, an object recognition process 601, a medical procedure identification process 602, a posture estimation process 603, and an age / gender estimation process 604 constitute a first medical support process 60A, which is executed by the server 200. Also, a medical support information output process 610 constitutes a second medical support process 60B, which is executed by the user terminal 300. For the execution of each process, information is transmitted between the user terminal 300 and the server 200 via a network 450. Also, the user terminal 300 can access the database 400 via the network 450 as necessary.

[0049] The object recognition process 601 acquires image data of a medical scene, and recognizes one or more objects appearing in the image data from the image data. The object recognition process 601 recognizes objects, for example, by using an object recognition model. The object recognition model is configured, for example, by a machine learning model that outputs the name of an object (medical material) appearing in the image when image data is given as input. Here, the name may be an identifier of the object or medical material, and the term "name" may include the identifier.

[0050] As an example, machine learning for object recognition was performed using video (or still images) of the medical material to be recognized captured in an actual usage environment or an environment close to the usage environment as learning data (training data), and the captured medical material name (object name) as the correct label. The model obtained by this machine learning was used as a model for object recognition processing 601.

[0051] Here, there are multiple medical materials that are the recognition targets of the system 100. The medical materials that are the recognition targets are objects used in medical procedures identified by the system 100. There are multiple candidates for medical procedures identified by the system 100, and objects used in each candidate medical procedure are learned as recognition targets.

[0052] If the medical material has a machine-readable code such as a barcode or two-dimensional code, the object recognition process 601 may recognize the medical material by reading the code that appears in the image data. By combining recognition by the object recognition model and code reading, more detailed data on the medical material can be obtained.

[0053] The recognized name or names of the medical materials (medical equipment, etc.) are provided to a medical procedure identification process 602. The recognized name or names of the medical materials (medical equipment, etc.) may be provided to a medical support information output process 610.

[0054] The medical procedure identification process 602 identifies a medical procedure performed by a user from one or more recognized medical materials. Note that a computer program module for the medical procedure identification process 602 is referred to as a medical procedure identification module.

[0055] In the embodiment, the medical procedures are identified using the medical material-medical procedure table 430. In the medical material-medical procedure table 430, a plurality of candidates for medical procedures to be identified are set, and one or a plurality of medical materials are associated with each of the plurality of candidates for medical procedures.

[0056] When the medical procedure identification process 602 acquires one or more medical materials recognized by the object recognition process 601, it refers to the medical material-medical procedure table 430 and identifies the medical procedure corresponding to the recognized one or more medical materials. In the embodiment, it does not stop at simply recognizing medical materials such as medical equipment, but identifies the medical procedure that the user intends to perform from the medical materials appearing in the image data, so that it is possible to present the user with appropriate medical support information corresponding to the identified medical procedure. The identified medical procedure is provided to the medical support information output process 610 for presenting the corresponding medical support information to the user.

[0057] The posture estimation process 603 acquires image data of a medical scene, and estimates the posture of the patient 20 appearing in the image data from the image data. The posture estimation process 603 estimates the posture of the patient 20, for example, using a posture estimation model. The posture estimation model has, for example, a machine learning model that outputs the positions of the joints of a human from image data in which the human appears, and estimates the posture of the human from the positions of the joints. The estimated posture is, for example, any of supine position, prone position, left lateral position, right lateral position, sitting position, and standing position. The estimated patient posture is provided to the medical support information output process 610 for use in presenting medical support information.

[0058] The age / gender estimation process 604 acquires image data of a medical scene, and estimates at least one of the age and posture of the patient 20 appearing in the image data from the image data. Here, it is assumed that both age and gender are estimated. The age / gender estimation process 604 estimates the age and gender of the patient 20, for example, using an age estimation model and a gender estimation model, or a model that can estimate both age and gender. A rough estimate of the age, such as a decade, is sufficient. The estimated age and gender are provided to the medical support information output process 610.

[0059] The medical support information output process 610 determines medical support information corresponding to the medical action identified by the medical action identification process 602, and presents the determined medical support information to the user. Note that the computer program module for the medical support information output process 610 is referred to as an output processing module.

[0060] In the embodiment, when medical support information is determined from a medical practice, a medical support information database 440 is used. In the medical support information database 440, medical support information and the like are associated with each of a plurality of medical practices.

[0061] 3 shows various databases included in a database 400 used in the medical support process 60. In this embodiment, the database 400 includes a medical material-medical procedure table 430 and a medical support information database 440. The database 400 further includes a user database 410 and a patient database 420. These databases 410, 420, 430, and 440 may be distributed across multiple computers.

[0062] The user database 410 includes user data for each of the multiple users 10. Each user data has data items such as a user ID 411, a name 412, an occupation 413, and a level 414. The user ID 411 indicates an identifier given to each medical professional who is the user 10. The name 412 indicates the name of the medical professional who is the user 10. The occupation 413 indicates the occupation of the medical professional who is the user. The occupation is, for example, a doctor or a nurse. The level 414 is a level according to the experience of the medical professional, and is expressed by a number from 1 to 3, for example. As an example, 1 indicates low experience, 2 indicates medium experience, and 3 indicates high experience. The occupation 413 and the level 414 indicate attributes of the user 10. In an embodiment, the user attributes such as the occupation 413 or the level 414 are used to determine the content or presentation mode of medical support information presented to the user 10.

[0063] The patient database 420 includes patient data for each of a plurality of patients 20. Each patient data has data items such as a patient ID 421, a name 422, a sex 423, an age 424, and other data 425. The patient ID 421 indicates an identifier given to each patient 20. The name 422 indicates the name of the patient 20. The sex 423 indicates the sex of the patient 20. The age 424 indicates the age of the patient 20. The other data 425 includes contraindication data, medication information, examination schedule and results, instructions for fever, instructions for pain, allergy information, images, and the like. The contraindication data indicates drugs that should be contraindicated for the patient 20. The medication information indicates medicines currently being taken by the patient 20. The contraindication data and medication information, and the like, are useful as information for determining whether medical materials (especially medicines) used in a medical situation appearing in the image data are suitable for the patient 20.

[0064] The medical material-medical action table 430 is configured by associating each of a plurality of medical actions 431 with one or a plurality of medical materials 432. The medical material-medical action table 430 is used to identify the medical action 431 from the medical material 432. The medical actions 431 set in the medical material-medical action table 430 are candidates for medical actions that can be identified by the system 100. In addition, the medical materials 432 set in the medical material-medical action table 430 are candidates for medical materials that can be recognized by the system 100.

[0065] In the medical material-medical procedure table 430, one medical procedure 431 may be associated with one medical material 432, or may be associated with multiple medical materials 432. A certain medical material may be commonly associated with multiple medical procedures. In this case, multiple medical procedures that use a common medical material can be distinguished from each other by other medical materials that are not in common.

[0066] The medical support information database 440 is configured by associating medical support information 444 and the like with each of a plurality of medical actions 441. The medical support information database 440 is used to determine corresponding medical support information from a medical action. User data 442, patient data 443, and patient posture 445 may be further associated with the medical action 441. The user data 442 may include the user's job title 442A and the user's level 442B. The patient data 443 may include the patient's gender 443A and the patient's age 443B. In the embodiment, the medical support information 444 presented to the user may vary depending on at least one of the user attribute and the patient attribute. For this reason, the user data 442 and the patient data 443 in the medical support information database 440 are set in association with the medical support information 444. This allows appropriate medical support information to be determined according to the user attribute or the patient attribute. That is, in the medical support information database 440, for each medical procedure, medical support information according to the age of the patient may be set, and in addition or instead, medical support information according to the gender of the patient may be set.

[0067] Figure 4 shows an example of a medical material-medical procedure table 430 and a medical support information database 440. In Figure 4, for convenience, the medical material-medical procedure table 430 and the medical support information database 440 are integrally configured. Thus, the medical material-medical procedure table 430 and the medical support information database 440 may be integrally configured or separately configured.

[0068] In Figure 4, seven medical procedures represented by numbers "1" to "7" are illustrated. The medical procedure numbered "1" is "central venous catheter operation". "Central venous catheter operation" indicates an act in an inclusive concept including "central venous catheter insertion" and "central venous catheter removal". Only one medical material, "central venous catheter", is associated with the "central venous catheter operation". That is, here, the medical procedure and the medical material correspond one-to-one. Some medical support information is set associated with the medical procedure: "central venous catheter operation".

[0069] Medical procedure: In the case of "central venous catheter operation", the medical support information presented to the user may vary depending on user attributes such as the user's job title / level. For example, the medical support information in the medical procedure "central venous catheter operation" may be different between the information to be presented to a user who is a doctor and the information to be presented to a user who is a nurse. For example, when there is information to be presented to a doctor, as shown in FIG. 4, in the medical support information database 440, "doctor" is set as the "job title" corresponding to "central venous catheter operation", and the "medical support information" corresponding to "doctor" is set. Also, among doctors, there are differences in levels due to experience, and it is better to distinguish the information to be presented. Therefore, "1" and "3" are set as the "levels" corresponding to "doctor". For level "1", the medical support information "During insertion: Please keep the needle flat so as not to puncture the subclavian artery" is associated. Also, for level "3", the medical support information "During insertion: Are you taking any oral antiplatelet agents or anticoagulants?" and "During removal: When removing the catheter, lie in the supine position. There is a risk of air embolism in the sitting position." are associated.

[0070] User 10 is presented with information at a level higher than the level of User 10 himself / herself. Therefore, when the medical procedure "central venous catheter operation" is identified, for User 10 who is a doctor and at level 1, all of "During insertion: Please keep the needle flat so as not to puncture the subclavian artery", "During insertion: Are you taking any oral antiplatelet agents or anticoagulants?", and "During removal: When removing the catheter, lie in the supine position. There is a risk of air embolism in the sitting position." are presented as medical support information.

[0071] Also, when the medical procedure "central venous catheter operation" is identified, for User 10 who is a doctor and at level 2 or 3, "During insertion: Are you taking any oral antiplatelet agents or anticoagulants?" and "During removal: When removing the catheter, lie in the supine position. There is a risk of air embolism in the sitting position." are presented as medical support information.

[0072] In FIG. 4, medical support information for nurses corresponding to the medical procedure: "Central venous catheter operation" is omitted, but medical support information for the job title: nurse may be set. In addition, in FIG. 4, medical support information for each user attribute is not shown, but since the medical support information to be presented to the user 10 may change depending on the user attributes such as the age / gender of the patient 20, medical support information for each user attribute may be set for each medical procedure. The user attributes such as the age and gender of the patient 20 may be the age / gender acquired from the patient database 420, or may be the estimated age / gender acquired by the age / gender estimation process 604. In addition, in order to acquire patient data from the patient database 420, for example, the patient ID can be read from the patient code (such as a barcode given to the patient) that appears in the image data, and the patient data can be obtained by referring to the patient database 420 based on the patient ID.

[0073] In FIG. 4, the medical procedure numbered "2" is "Central venous catheter insertion." "Central venous catheter insertion" is the act of inserting a central venous catheter into the body of patient 20. "Central venous catheter insertion" is associated with multiple medical materials, namely, a "central venous catheter" and a "guidewire." In other words, here, there is a one-to-many correspondence between medical procedures and medical materials.

[0074] In FIG. 4, the medical procedure numbered "3" is "removal of central venous catheter." "Removal of central venous catheter" is the act of removing the central venous catheter inserted into the body of patient 20. "Removal of central venous catheter" is associated with multiple medical materials, namely, "central venous catheter" and "scissors (Cooper)." In other words, here, there is a one-to-many correspondence between medical procedures and medical materials.

[0075] In FIG. 4, for each of the medical procedures numbered "1", "2", and "3", a common medical material, the "central venous catheter", is associated. In a medical scenario, just recognizing the presence of the "central venous catheter" does not allow for distinguishing whether it is an insertion or a removal. However, if the "guide wire" used for central venous catheter insertion is recognized together with the "central venous catheter", the medical procedure the user is attempting can be identified as not the abstract "central venous catheter manipulation" but specifically "central venous catheter insertion". Also, if the "scissors (cooper)" used for central venous catheter removal is recognized together with the "central venous catheter", the medical procedure the user is attempting can be identified as not the abstract "central venous catheter manipulation" but specifically "central venous catheter removal".

[0076] Thus, even if there is a common medical material (e.g., central venous catheter) for a plurality of medical procedure candidates (e.g., the medical procedures numbered "1", "2", and "3"), the medical procedures can be distinguished and identified based on the presence or absence of other medical materials or what other medical materials (e.g., guide wire or scissors) are.

[0077] Also, in the system 100 of this embodiment, although common medical materials are used, such as in "central venous catheter insertion" and "central venous catheter removal", when there are medical procedures that should be distinguished from the perspective of the medical support information to be presented, even if those medical procedures cannot be distinguished, useful information can be presented to the user 10. That is, in the system 100 of this embodiment, a comprehensive conceptual medical procedure (e.g., central venous catheter manipulation) associated only with a common medical material (e.g., central venous catheter) is set. For this reason, even when the system 100 of this embodiment cannot distinguish the more specific concepts of "central venous catheter insertion" and "central venous catheter removal", it can identify the comprehensive conceptual medical procedure of "central venous catheter manipulation" and present medical support information corresponding to the comprehensive conceptual medical procedure.

[0078] When the comprehensive medical procedure "central venous catheter operation" is identified, the system 100 can present information calling attention to both insertion and removal while distinguishing which medical support information is for which procedure, so that appropriate information is presented whether the specific procedure is insertion or removal. The user 10 can naturally understand whether the medical procedure the user 10 is performing is insertion or removal, and can therefore appropriately refer to the presented medical support information that applies to the procedure that the user 10 is actually performing.

[0079] In this way, by making it possible to present medical support information corresponding to a comprehensive conceptual medical procedure, it is possible to quickly present medical support information before identifying a specific medical procedure. Furthermore, if a specific medical procedure can be identified, it is possible to present more appropriate medical support information corresponding to the specific medical procedure.

[0080] In FIG. 4, for the medical procedure: "insertion of a central venous catheter," "physician" is set as the "job title," and "medical support information" corresponding to the "physician" is set. In addition, "1" and "3" are set as the "level" corresponding to the "physician," and "medical support information" corresponding to each level is set. Level: "1" is associated with the medical support information "Hold the needle at an angle so as not to puncture the subclavian artery." In addition, level: "3" is associated with the medical support information "Are you taking any oral antiplatelet or anticoagulant drugs?"

[0081] In Figure 4, for the medical procedure: "removal of central venous catheter," "job title" is set to "doctor" and "nurse," and "medical support information" corresponding to "doctor" and "nurse" are set. "3" is set as the "level" corresponding to "doctor," and "medical support information" corresponding to level "3" is set as "When removing the catheter, the patient should be in a supine position. There is a risk of air embolism in a sitting position." "1" is set as the level corresponding to "nurse," and "medical support information" corresponding to level "1" is set as "Please place the patient in a supine position before the doctor removes the catheter."

[0082] 3, in the medical support information database 440, the type of appropriate posture 445 of the patient 20 receiving the medical procedure may be set in association with each medical procedure 441. The type of posture 445 to be set may be, for example, any of supine position, prone position, left lateral position, right lateral position, sitting position, and standing position.

[0083] For example, removal of a central venous catheter is appropriately performed with the patient 20 in a supine position. The posture 445 set in association with each medical procedure 441 is used for posture determination as to whether the patient posture estimated by the posture estimation process 603 matches the posture 445 set in association with each medical procedure 441. The system 100 can generate medical support information according to the posture determination result. For example, when "supine" is set as the posture 445 corresponding to the medical procedure: "removal of a central venous catheter," if the estimated posture of the patient 20 is different from "supine" (for example, sitting), the system 100 can present the user with medical support information such as "please place the patient in a supine position."

[0084] In FIG. 4, the medical procedure numbered "4" is "lumbar puncture." One medical material, a "lumbar puncture needle," is associated with the "lumbar puncture." Several pieces of medical support information are associated with the medical procedure: "lumbar puncture."

[0085] For the medical procedure: "lumbar puncture", medical support information for when the "job title" is "physician" and medical support information for when the "job title" is "nurse" are associated separately. In addition, "1" and "3" are set as "levels" corresponding to "physician". Level: "1" for physicians is associated with medical support information such as "Please puncture from the 3rd-4th or 4th-5th lumbar intervertebral space, aiming at the 4th lumbar vertebra where the line connecting the upper edge of the left and right iliac crests (Jacoby line) intersects" and "Bend your neck to look at your navel, hug your knees, and curl your body like a shrimp". Level: "3" for physicians is associated with medical support information such as "Are you taking any platelet agents or anticoagulants?" In addition, "1" is set as "level" corresponding to "nurse". Level: "1" for nurses is associated with medical support information such as "Please support the patient's curled-up position without touching the sterile field".

[0086] In FIG. 4, the medical procedure "number 5" is "urethral balloon insertion." One medical material, a "urethral balloon," is associated with "urethral balloon insertion." Medical procedure: "urethral balloon insertion" is associated with medical support information that is presented in common regardless of whether the user 10 is a doctor or a nurse. In this way, the job titles set in the database may span multiple job titles. Here, for medical procedure: "urethral balloon insertion," "Please check that urine is flowing into the catheter and inflate the balloon" is set as medical support information for level "1" that is common to both doctors and nurses.

[0087] In Figure 4, the medical procedure "number 6" is "pleural puncture." "Pleural puncture" is associated with a number of medical materials, namely, an "echo probe" and a "thoracentesis needle." For the medical procedure: "pleural effusion needle insertion," the user 10's occupation is "doctor," and "Please needle along the upper edge of the ribs" is set as medical support information for level "1."

[0088] In FIG. 4, the medical procedure "number 7" is "bone marrow puncture." One medical material, a "bone marrow puncture needle," is associated with "bone marrow puncture." Medical support information for doctors is set for the medical procedure "bone marrow puncture." Furthermore, levels "1" to "3" are set for the doctors. As medical support information for level "1," "The puncture site is the ilium (posterior superior iliac spine, anterior superior iliac spine)" is set, as medical support information for level "2," "Please adjust the length of the puncture needle with the anesthetic needle" is set, and as medical support information for level "3," "Are you taking any oral antiplatelet or anticoagulant drugs?" is set.

[0089] Fig. 5 shows the procedure of the object recognition process 601 shown in Fig. 2. In an embodiment, the object recognition process 601 recognizes an object in image data for each predetermined period. The length of the predetermined period may be, for example, about several minutes (e.g., 3 minutes) or about several seconds. In the image data that is a moving image, the object recognition process 601 recognizes one or more medical materials that appear in one period in the image data that is a moving image, and provides one or more medical materials recognized in each period to the medical procedure identification process 602 for each period.

[0090] A more specific example of the process is as shown in FIG. 5. First, when the object recognition process 601 starts, a period timer that counts a predetermined period starts (step S51). This period timer counts the time from 0 to one period length, and when one period has elapsed, the timer is reset (step S55). During one period, the processes of steps S52 to S54 are performed. In step S52, medical materials in the image are recognized by the object recognition model. Recognition of the medical materials is repeated during one period (step S53). Then, when one period ends, one or more medical materials recognized during one period are provided to the medical procedure identification process 602.

[0091] The multiple medical materials recognized during one period may appear in the image data at the same time, or may appear in the image data at different times during one period. The field of view of the user 10 performing the medical procedure may vary, and the range of the field of view is limited. Therefore, even if multiple medical materials for a certain medical procedure are present in a medical scene, the multiple medical materials do not necessarily appear simultaneously in the image data. However, in this embodiment, it is sufficient that multiple medical materials appear in the image data at any time within a certain time range of one period, and the system 100 recognizes them, so that multiple medical materials corresponding to one medical procedure can be properly recognized. As a result of being able to properly recognize multiple medical materials, the system 100 can properly identify the medical procedure.

[0092] Fig. 6 shows the procedure of the medical support information output process 610 shown in Fig. 2. In the medical support information output process 610, first, the medical procedure identified by the medical procedure identification process 602 is acquired for each cycle (step S61). Then, it is determined whether the medical procedure acquired in the current cycle is the same as the medical procedure acquired in the previous cycle (step S62). If they are the same, the medical support information corresponding to the medical procedure acquired in the current cycle is not presented, and the process returns to step S61 and waits until the medical procedure in the next cycle is acquired.

[0093] If the medical procedure acquired in the current cycle is different from the medical procedure acquired in the previous cycle, medical support information corresponding to the medical procedure acquired in the current cycle is generated and output (steps S63 to S70).

[0094] For example, to generate medical support information, medical support information corresponding to the medical practice acquired in the current cycle is determined by referring to the medical support information database 440 (step S63). At this time, by acquiring user data indicating the job title and level of the user 10 and patient data indicating the age and sex of the patient 20, appropriate medical support information according to the acquired user data and patient data may be determined.

[0095] Furthermore, an estimated posture of the patient is obtained (step S64), and a posture determination may be performed as to whether the estimated posture of the patient is an appropriate posture corresponding to the acquired medical act (step S65). For example, if the posture corresponding to the medical act is the supine position but the estimated posture of the patient is the sitting position, etc., and the posture is inappropriate, medical support information regarding the posture (information based on the estimated posture), such as "Please place the patient in the supine position," is generated (step S67). Note that if the estimated posture of the patient is an appropriate posture corresponding to the acquired medical act, the medical support information regarding the posture may or may not be generated.

[0096] Also, in the medical support information output process 610, in order to perform a determination of the compatibility between the recognized medical material (for example, medicine) and the patient receiving the medical act (step S68), data acquisition for compatibility determination is performed (step S67). The data for compatibility determination includes, for example, the name of the medical material recognized by the object recognition process 601. Also, the data for compatibility determination includes information to be compared with the recognized medical material (for example, medicine), such as the contraindication data or medication information of the patient 20. By comparing the recognized medical material with the contraindication data or medication information of the patient 20, a determination of compatibility (step S68) is performed.

[0097] For example, if the medicine as the medical material used in the medical act is contraindicated for the patient 20, there is a possibility that the medicine is inappropriate for the medical act, so information indicating incompatibility (support information regarding compatibility) is generated. Also, in the compatibility determination, a determination of compatibility (step S68) may be performed by comparing the identified medical act with the contraindication data or medication information of the patient 20. For example, during a medical act of central venous catheter insertion, it is necessary that the patient 20 is not taking an antiplatelet agent or an anticoagulant. In the compatibility determination, the medicine (antiplatelet agent or anticoagulant) contraindicated in the medical act of central venous catheter insertion may be compared with the medication information for the patient 20, and it may be determined that it is an inappropriate situation to perform the medical act.

[0098] If the result of the suitability determination (step S68) is that the medical support is not suitable, information indicating that fact (medical support information regarding suitability) is generated (step S69). Note that if the medical support is suitable, the medical support information regarding suitability may or may not be generated.

[0099] The medical support information generated in steps S63, S66, and S69 is output for presentation to the user (step S70). The medical support information is displayed on the display 350 of the smart glasses. As shown in FIG. 1, the medical support information 530 is displayed together with the level 520 (alert level) corresponding to the medical support information 530 and the identified medical procedure 510. It is preferable that the display manner of the medical support information 530 is different, such as by changing the display color, according to the level 520. For example, level 1 can be displayed in yellow, level 2 in orange, and level 3 in red. In this case, the user 10 can intuitively recognize the importance of the medical support information. In addition, a sound according to the level may be output from the speaker 360 together with the display of the medical support information on the display 350. In this case, the user can auditorily grasp the level.

[0100] Furthermore, since the medical procedure 510 identified by the system 100 is displayed together with the medical support information 530, the user 10 can determine whether the medical procedure that the user 10 intends to perform matches the medical procedure 510 identified by the system 100. If the medical procedure 510 identified by the system 100 is not appropriate, the user 10 can ignore the presented medical support information 530.

[0101] When the system 100 receives a cancellation instruction for the presentation of medical support information presented to the user 10 from the user, the system 100 stops presenting the medical support information. When the user 10 determines that the presented medical procedure 510 is inappropriate, the user 10 can issue a cancellation instruction for the presentation of the presented medical support information by operating the operation unit 380 or giving a voice instruction to the microphone 370. When the system 100 receives the cancellation instruction, the system 100 erases the display of the display data 500 such as the presented medical support information 530. Further, when the system 100 receives the cancellation instruction, the system 100 may cancel the display of the display data presented in the current cycle and redisplay the display data presented in the previous cycle.

[0102] FIG. 7 shows an example of recognition of medical materials 31 and 32, identification of medical procedures, and presentation of medical support information. Here, it is assumed that a central venous catheter 31 (first object) appears in the first image data 701 captured at time point T1-1 in the first cycle. However, it is assumed that the guide wire 32 does not appear in the first image data 701 at time point T1-1. Also, at time point T2-1 in the second cycle, the central venous catheter 31 (first object) appears. Further, in the second cycle, at time point T2-2 after time point T2-1, it is assumed that a guide wire 32 (second object) appears in the second image data 702. It is assumed that the first object 31 and the second object 32 do not appear in the image data at the same time at each of the time points T1-1, T2-1, and T2-2. Also, at time point T3-1 in the third cycle, it is assumed that the central venous catheter 31 (first object) and the guide wire 32 (second object) appear in the image data at the same time.

[0103] In the first cycle, the system 100 recognizes only one medical material, the central venous catheter (first object) (step S71). Therefore, in the first cycle, the system 100 recognizes "central venous catheter operation" (first medical procedure) as a medical procedure corresponding to one medical material, the central venous catheter (first object), and presents it to the user (step S82). Screen 711 in Fig. 7 shows an example of the display of the medical procedure, alert level, and medical support information recognized in the first cycle.

[0104] In the second cycle following the first cycle, the system 100 recognizes the central venous catheter 31 (first object) at time T2-1 (step S72), and recognizes the guidewire 32 (second object) at time T2-2 (step S73). Therefore, in the second cycle, the system 100 identifies "insertion of a central venous catheter" (second medical procedure) as a medical procedure corresponding to the two medical materials, the central venous catheter 31 (first object) and the guidewire 32 (second object) (step S83). Since the medical procedure identified in the second cycle is different from the medical procedure identified in the first cycle, the medical support information corresponding to the medical procedure identified in the second cycle: "insertion of a central venous catheter" is presented to the user as the latest information (step S84). A screen 712 in FIG. 7 shows an example of a display of the medical procedure, alert level, and medical support information recognized in the first cycle.

[0105] In the third cycle, the system 100 recognizes the central venous catheter 31 (first object) and the guidewire 32 (second object) at time T3-1 (steps S74, S75). Therefore, in the third cycle, the system 100 re-identifies "central venous catheter insertion" as a medical procedure corresponding to the two medical materials, the central venous catheter 31 (first object) and the guidewire 32 (second object) (step S85). Since the medical procedure identified in the third cycle is the same as the medical procedure identified in the second cycle, medical support information corresponding to the medical procedure identified in the third cycle: "central venous catheter insertion" is not presented. This prevents the medical support information from being displayed unnecessarily repeatedly, improving usability.

[0106] The present invention is not limited to the above-described embodiment, and various modifications are possible. [Explanation of symbols]

[0107] 10: User 20 :Patient 31: Central venous catheter (first object; first medical device) 32: Guidewire (second object; second medical device) 40: stand 60: Medical support processing 60A: First medical support processing 60B: Second medical support processing 100: Medical support system 200 :Server 210: Processor 220: Storage device 230: Computer Programs 300: User terminal 301: Main body 310: Processor 320: Storage device 330: Computer Programs 340: Camera 350: Display 360: Speaker 370: Mike 380:Operation unit 390: Communication interface 400 : Database 410: User Database 411 : User ID 412 : Name 413: Job title 414: Level 420: Patient database 421 :Patient ID 422: Name 423 :Gender 424: Age 425: Other data 430: Medical supplies - Medical procedure table 431: Medical Procedures 432: Medical supplies 440: Medical Support Information Database 441: Medical Procedures 442: User data 442A: Job title 442B :Level 443: Patient Data 443A :Gender 443B: Age 444: Medical support information 445 :Patient posture 450: Network 500: Display data 510: Medical Procedures 520: Level 530: Medical support information 601: Object recognition processing 602: Medical procedure identification processing 603: Posture estimation processing 604: Age / gender estimation processing 610: Medical support information output processing 701: First image data 702: Second image data 711: Screen 712: Screen

Claims

1. Recognizing one or more medical objects from the image data; Identifying a medical procedure corresponding to the recognized object or objects; Present to the user medical support information corresponding to the identified medical procedure. The method is configured to perform a process comprising: Identifying the medical procedure includes identifying the medical procedure corresponding to the one or more objects that appear in the image data from among a plurality of medical procedure candidates including a first medical procedure associated with a first object and a second medical procedure associated with a second object other than the first object and the first object. Medical support system.

2. and if the first object appears but the second object does not appear at a first time point in the image data and the second object appears at a second time point after the first time point, the method is configured to identify a first medical procedure corresponding to the first object before the second time point and to identify a second medical procedure corresponding to the first object and the second object after the second time point. The medical support system according to claim 1 .

3. The process comprises: The method is configured to repeatedly identify the medical procedure for a plurality of periods including a first period and a second period subsequent to the first period; When the medical procedure identified in the second period is the same as the medical procedure identified in the first period, the medical support information is not presented to the user in the second period. The medical support system according to claim 1 or 2.

4. The medical support information presented to the user is determined according to the attributes of the user. The medical support system according to any one of claims 1 to 3.

5. The process further comprises estimating a posture of a patient receiving the medical procedure from the image data; The medical support information presented to the user includes information based on the estimated posture. The medical support system according to any one of claims 1 to 4.

6. The process further includes determining compatibility between the recognized object and / or the identified medical procedure and the patient receiving the medical procedure by comparing the recognized object and / or the identified medical procedure with contraindication data or medication information of the patient receiving the medical procedure stored in a patient database, The medical support information presented to the user includes information based on the result of the suitability determination. The medical support system according to any one of claims 1 to 5.

7. The process further comprises presenting the identified medical procedures along with the medical assistance information to a user. The medical support system according to any one of claims 1 to 6.

8. When an instruction to cancel presentation of the presented medical support information is received from the user, the process makes the medical support information non-presentable. The medical support system according to any one of claims 1 to 7.

9. The medical support information is presented to the user by a wearable device worn by the user; The wearable device includes a camera for acquiring the image data. The medical support system according to any one of claims 1 to 8.

10. A medical support method executed by a medical support system, comprising: Recognizing one or more medical objects from the image data; The medical support system identifies a medical procedure corresponding to the recognized one or more objects; Present to the user medical support information corresponding to the identified medical procedure. It is equipped with Identifying the medical procedure includes identifying the medical procedure corresponding to the one or more objects that appear in the image data from among a plurality of medical procedure candidates including a first medical procedure associated with a first object and a second medical procedure associated with a second object other than the first object and the first object. Medical assistance methods.

Images