Surgical planning and executing method and system

Abstract

The embodiment of the invention provides an operation planning and executing method which is implemented on a computing device with at least one processing device and at least one storage device, an intelligent agent capable of self-evolving based on the artificial intelligence technology is configured on the computing device, and the method is executed by the intelligent agent. The method comprises the following steps: acquiring sensing information acquired by sensing equipment in an operating room during an operation of a patient; performing EOI detection on the perceptual information based on an event of interest (EOI) detection rule, the EOI detection rule being obtained by an agent learning from a historical record, the EOI including at least one of an instruction issued by an operation participant, detection of an operation risk, an abnormal physiological state of a patient, a number of operation tools less than a preset value, and completion of an operation; in response to the detection that the EOI executes the preset operation corresponding to the EOI, the preset operation corresponding to the EOI is determined based on the corresponding relationship between the EOI and the preset operation, and the corresponding relationship is obtained by learning from the historical record by the agent.

Description

Cross-references

[0001] This application is a divisional application for the Chinese application with application number 202411743185.5 filed on November 29, 2024, and invention name “Method and system for surgical planning and execution”. The Chinese application claims priority to the international application with application number PCT / CN2024 / 109059 filed on July 31, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the technical field of surgical planning, and in particular to a method and system for surgical planning and execution. Background Art

[0003] Before performing surgery on a patient, the doctor will first develop a surgical plan based on the patient's data (e.g., the patient's examination report, scan images, etc.). Whether the pathological characteristics of the patient's lesion (e.g., tumor) and its surrounding tissues or organs (e.g., the complex geometric characteristics of the lesion, etc.) can be fully and intuitively displayed directly affects the accuracy of the surgical plan, thereby affecting the execution and effectiveness of the surgery.

[0004] Based on this, it is expected to propose a method and system for surgical planning and execution to help with more reasonable and safe surgical planning and execution. Summary of the invention

[0005] One or more embodiments of the present specification provide a preoperative patient education method implemented on a computing device having at least one processing device and at least one storage device, and characterized in that it includes: generating explanation materials for explaining the surgical plan to the patient; simultaneously displaying the explanation materials to the patient, the doctor and the patient's family through the patient's first terminal device, the doctor's second terminal device and the patient's family's third terminal device; determining feedback information about the surgical plan based on the perception information collected by the perception device during the explanation of the surgical plan; confirming or updating the surgical plan based on the feedback information; obtaining a first confirmation instruction about the surgical plan input by the patient through the first terminal device, and a second confirmation instruction about the surgical plan input by the patient's family through the third terminal device; in response to the first confirmation instruction and the second confirmation instruction, controlling the first terminal device, the second terminal device and the third terminal device to respectively present a surgical consent form; and obtaining signature information of the surgical consent form from the first terminal device, the second terminal device and the third terminal device, respectively.

[0006] One or more embodiments of the present specification provide a preoperative guidance method, which is implemented on a computing device having at least one processing device and at least one storage device, wherein the computing device is configured with a nurse agent, including: determining a planned route from a patient's current position to a waiting area in an operating room; controlling a smart chair to transport the patient to the waiting area along the planned route; performing preoperative care on the patient through the patient's terminal device based on the patient's patient data and surgical plan during the patient's transportation to the waiting area; and after the patient is transported to the waiting area, performing patient verification on the patient through the nurse agent based on sensory information related to the patient collected by sensory devices in the waiting area, wherein the sensory information related to the patient includes biological information.

[0007] One or more embodiments of the present specification provide a method for surgical planning and execution, which is implemented on a computing device having at least one processing device and at least one storage device, and includes: formulating a surgical plan for the patient based on the patient's patient data; providing preoperative care for the patient through a first terminal device worn by the patient based on the surgical plan and the patient data; controlling an intelligent robot nurse to prepare surgical tools in the operating room before the operation based on the surgical plan; and controlling the intelligent robot nurse to assist in the operation based on sensory information collected by the sensory equipment in the operating room during the operation.

[0008] One or more embodiments of the present specification provide a method for surgical planning and execution, which is implemented on a computing device having at least one processing device and at least one storage device, wherein the computing device is configured with an agent capable of self-evolving based on artificial intelligence technology, and the method is executed by the agent. The surgical planning and execution method includes: obtaining sensory information collected by a sensory device in an operating room during a patient's surgery; performing EOI detection on the sensory information based on an event of interest (EOI) detection rule, wherein the EOI detection rule is learned by the agent from historical records, wherein the EOI includes at least one of the following: a surgical participant issues an instruction, a surgical risk is detected, the patient's physiological state is abnormal, the number of surgical tools is less than a preset value, and the operation is completed; and in response to detecting the occurrence of an EOI, executing a preset operation corresponding to the EOI, wherein the preset operation corresponding to the EOI is determined based on a corresponding relationship between the EOI and the preset operation, wherein the corresponding relationship is learned by the agent from historical records.

[0009] One or more embodiments of the present specification provide a surgical planning and execution system, including a processing device, wherein the processing device is used to execute any one of the methods described above.

[0010] One or more embodiments of the present specification provide a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes any one of the methods described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0012] Figure 1 is a block diagram of an exemplary medical service system according to some embodiments of the present application;

[0013] Figure 2 is a schematic diagram of an exemplary medical service system according to some embodiments of the present application;

[0014] Figure 3 is a schematic diagram of an exemplary hospital support platform according to some embodiments of the present application;

[0015] Figure 4 is a schematic diagram of modules of an exemplary surgical planning and execution system according to some embodiments of the present specification;

[0016] Figure 5 is a flowchart of an exemplary surgical planning and execution method according to some embodiments of this specification;

[0017] Figure 6 is a flowchart of an exemplary surgical planning according to some embodiments of this specification;

[0018] Figure 7 is a schematic diagram of an exemplary process of generating a surgical plan according to some embodiments of this specification;

[0019] Figure 8 is a schematic diagram of an exemplary process of generating a surgical plan according to other embodiments of this specification;

[0020] Fig. 9 is a flowchart of an exemplary surgical simulation according to some embodiments of the present specification;

[0021] Fig.10 is a flowchart of an exemplary surgical simulation according to some embodiments of the present specification;

[0022] Fig.11 is a schematic diagram of exemplary preoperative patient education according to some embodiments of this specification;

[0023] Fig.12is a schematic diagram of an exemplary preoperative patient education process according to some embodiments of this specification;

[0024] Fig.13 is a schematic diagram of a flow chart of an exemplary agreement signing process according to some embodiments of this specification;

[0025] Fig.14 is a schematic diagram of an exemplary preoperative guidance according to some embodiments of the present specification;

[0026] Fig.15 is a schematic diagram of an exemplary surgical procedure performed according to some embodiments of the present specification;

[0027] Fig.16 is a flowchart of an exemplary surgical execution according to some embodiments of the present specification;

[0028] Fig.17 is a flowchart of exemplary surgical execution according to other embodiments of this specification;

[0029] Fig.18 is a schematic diagram of an exemplary surgical review process according to some embodiments of this specification;

[0030] Fig.19 is a schematic diagram of an exemplary process of generating a surgical plan according to other embodiments of this specification; and

[0031] Fig. 20 It is a flowchart of exemplary preoperative preparation and surgical execution shown in other embodiments of this specification. DETAILED DESCRIPTION

[0032] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0033] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0034] In general, the terms "module", "unit" or "block" are used here to refer to the logic or set of software instructions contained in hardware or firmware, or a set of software instructions. The modules, units or blocks described herein can be implemented as software and / or hardware, and can be stored in any type of non-temporary computer-readable medium or other storage device. In some embodiments, the software modules / units / blocks can be compiled and linked into executable programs. It should be recognized that software modules can be called from other modules / units / blocks or from themselves, and / or can be called in response to detected events or interrupts. Software modules / units / blocks configured to be executed on a computing device can be provided on a computer-readable medium (e.g., a CD-ROM, a digital video floppy disk, a flash drive, a floppy disk, or any other tangible medium), or provided as a digital download (can be initially stored in a compressed or installable format, which needs to be installed, decompressed or decrypted before execution). The software code herein can be stored in part or in whole in the storage device of the computing device performing the operation and applied to the operation of the computing device. The software instructions can be embedded in firmware, such as an EPROM. It should also be understood that hardware modules / units / blocks may be included in connected logical components, such as gates and triggers, and / or may include programmable units, such as programmable gate arrays or processors. The modules / units / blocks or computing device functions described herein may be implemented as software modules / units / blocks, but may be represented in hardware or firmware. Generally, the modules / units / blocks referred to herein refer to logical modules / units / blocks, which may be combined with other modules / units / blocks or divided into sub-modules / sub-units / sub-blocks, even if they are physically organized or memory blocks. This description may apply to a system, an engine, or a portion thereof.

[0035] It should be understood that, unless the context clearly indicates otherwise, when a unit, engine, module or block is said to be "on" another unit, engine, module or block, "connected" or "coupled" to another unit, engine, module or block, it can be directly on, connected or coupled or communicate with the other unit, engine, module or block, or there may be intermediate units, engines, modules or blocks. In the present disclosure, the term "and / or" may include any one or more of the relevant listed items or combinations thereof. In the present disclosure, the term "image" may refer to a 2D image, a 3D image, or a 4D image.

[0036] These and other features, characteristics and methods of function and operation of the relevant structural elements of the present disclosure, as well as the combination of components and manufacturing economies, may be more clearly represented from the following description and the accompanying drawings, which form a part. However, it should be understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of the present disclosure. It should be understood that the accompanying drawings are not to scale.

[0037] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0038] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0039] In the scenario of surgical execution and planning, when encountering complex or difficult operations that cannot be completed by local doctors or doctors in a single department, the traditional practice is to invite experts from other places or other departments (e.g., medical experts) to the local area for joint discussions and / or participation in the operation. This method requires high manpower and time costs. In some embodiments, medical experts from multiple different places or departments participate in discussions or surgical planning through online consultations. However, if the online consultation only relies on traditional forms of display (e.g., text, pictures, videos, etc.) of patient data (e.g., patient examination reports, scanned images, etc.), it may not be possible to fully and intuitively display the pathological characteristics of the lesion (e.g., tumor) and its surrounding tissues or organs (e.g., complex geometric features of the lesion, etc.). As a result, doctors (especially experts who participate in online meetings remotely) may not be able to make accurate judgments, thereby affecting the doctor's surgical planning. On the other hand, patients and their families usually lack professional medical knowledge and cannot have a clearer understanding of the severity of the lesion, the scheduled surgical process, and the risks of the operation.

[0040] In some embodiments of this specification, a method and system for surgical planning and execution are provided. The method and system for surgical planning and execution combine a physical hospital with a digital twin hospital, and realize operations such as surgical planning, surgical simulation, preoperative patient preparation, surgical execution, and postoperative review through the linkage of virtual and real, eliminate the incomplete information of 2D images, assist doctors in accurately planning and executing surgeries, and improve the success rate of surgeries. For more information about the above method, please refer to Figures 5 to 18 and its related description.

[0041] Figure 1 is a block diagram of an exemplary medical service system according to some embodiments of the present application.

[0042] The medical service system 100 can also be called a meta-hospital system. It is built based on a variety of innovative technologies, including metaverse technology, XR technology (for example, augmented reality (AR) technology, virtual reality (VR) technology, mixed reality (MR) technology, etc.), AI technology, digital twin technology, IOT technology, data circulation technology (for example, blockchain technology, data privacy computing technology), spatial computing technology, image rendering technology, etc.

[0043] like Figure 1 As shown, the medical service system 100 may include a physical hospital 110, a virtual hospital 130, a user space application 120, and a hospital support platform 140. In some embodiments, the hospital support platform 140 may map data related to the physical hospital 110 to the virtual hospital 130 corresponding to the physical hospital 110, and provide user services to related users of the physical hospital 110 through the user space application 120.

[0044] Physical hospital 110 refers to a hospital that exists in the physical world and has tangible attributes.Herein, healthcare facilities that provide medical, surgical, and psychiatric care and treatment to people are collectively referred to as hospitals.

[0045] like Figure 1 As shown, the physical hospital 110 may include multiple physical entities. For example, the multiple physical entities may include departments, users, hardware equipment, user services, public areas, medical service processes, etc., or any combination thereof.

[0046] Departments are specialized units or departments that provide specific types of medical care, treatment, and services. Each department may focus on a particular area of ​​medicine and may be staffed by health care professionals with expertise in that area. For example, departments may include outpatient departments, inpatient departments, surgical departments, support departments (e.g., registration departments, pharmacy departments), internal medicine, surgery, specialty medicine, child health departments, etc., or any combination thereof.

[0047] The user may include any user associated with the physical hospital 110 (or referred to as a related user of the physical hospital 110). For example, the user may include a patient (or a part of a patient (e.g., an organ)), a companion, a visitor of a patient, a hospital staff of the physical hospital 110, a supplier of the physical hospital 110, an application developer of the physical hospital 110, etc., or any combination thereof. The hospital staff of the physical hospital 110 may include medical service providers (e.g., doctors, nurses, technicians, etc.), hospital administrators, support staff, or the like, or any combination thereof. Exemplary hospital administrators may include department nursing administrators, clinical administrators, department directors, hospital directors, hospital administrators, functional administrators, or the like, or any combination thereof.

[0048] The hardware devices may include hardware devices located in the physical hospital 110 and / or hardware devices that communicate with hardware devices in the physical hospital 110. Exemplary hardware devices may include terminal devices, medical service devices, sensing devices, basic devices, etc., or any combination thereof.

[0049] The terminal device may include a terminal device that interacts with a user of the medical service system 100. For example, the terminal device may include a first terminal device that interacts with a patient (also referred to as a patient terminal), a second terminal device that interacts with a patient's doctor (also referred to as a doctor terminal), a terminal device that interacts with a nurse (also referred to as a nurse terminal), a terminal device that interacts with a remote visitor (also referred to as a remote terminal device), or a public terminal of a hospital (e.g., a consulting room terminal, a bedside terminal device, a terminal device in a waiting area, an intelligent surgical terminal), etc., or any combination thereof. In the present application, unless clearly obtained from the context or otherwise specified by the context, terminal devices owned by a user and terminal devices provided to a user by the physical hospital 110 are collectively referred to as terminal devices of the user or terminal devices that interact with the user.

[0050] The terminal device may include a mobile terminal, an XR device, a smart wearable device, etc. The mobile terminal may include a smart phone, a personal digital assistant (PDA), a display, a gaming device, a navigation device, a handheld terminal (POS), a tablet computer, etc., or any combination thereof.

[0051] XR devices may include devices that allow users to participate in extended reality experiences. For example, XR devices may include VR components, AR components, MR components, etc., or any combination thereof. In some embodiments, XR devices may include XR helmets, XR glasses, XR patches, stereo headphones, etc., or any combination thereof. For example, XR devices may include Google Glass TM 、Oculus Rift TM 、Gear VR TM、AppleVisionpro TM Etc. Specifically, the XR device may include a display component on which virtual content can be presented and / or displayed. In some embodiments, the XR device may further include an input component. The input component enables user interaction between the user and the virtual content (e.g., a virtual surgical environment) displayed by the display component. For example, the input component may include a touch sensor, a microphone, an image sensor, etc., which is configured to receive user input, which can be provided to the XR device and used to control the virtual world by changing the visual content presented on the display component. The input component may include a handle, a glove, a stylus, a console, etc.

[0052] Smart wearable devices may include smart bracelets, smart shoes and socks, smart glasses, smart helmets, smart watches, smart clothes, smart backpacks, smart accessories, etc., or any combination thereof. In some embodiments, smart wearable devices may obtain physiological data (e.g., heart rate, blood pressure, body temperature, etc.) of the user.

[0053] The medical service equipment can be configured to provide medical services to patients. For example, the medical service equipment can include examination equipment, nursing equipment, treatment equipment, etc., or any combination thereof.

[0054] The inspection device may be configured to provide inspection services to the patient, for example, to collect the patient's inspection data. Exemplary inspection data may include heart rate, respiratory rate, body temperature, blood pressure, medical imaging data, body fluid test reports (e.g., blood test reports), etc., or any combination thereof. Accordingly, the inspection device may include a vital sign monitor (e.g., a blood pressure monitor, a blood glucose meter, a heart rate meter, a thermometer, a digital stethoscope, etc.), a medical imaging device (e.g., a computer tomography (CT) device, a digital subtraction angiography (DSA) device, a magnetic resonance (MR) device, etc.), a laboratory device (e.g., a routine blood test device, etc.), or any combination thereof.

[0055] Nursing equipment can be configured to provide nursing services to patients and / or assist medical service providers in providing nursing services. Exemplary nursing equipment can include hospital beds, patient care robots, smart nursing carts, smart medicine boxes, smart wheelchairs, etc.

[0056] The therapeutic device can be configured to provide therapeutic services to patients and / or assist medical service providers in providing therapeutic services. Exemplary therapeutic devices can include surgical devices, radiation therapy devices, physical therapy devices, etc., or any combination thereof.

[0057] The perception device may be configured to collect perception information related to the environment in which it is located. For example, the perception device may include an image sensor, a sound sensor, etc. The image sensor may be configured to collect image data in the physical hospital 110, and the sound sensor may be configured to collect voice signals in the physical hospital 110. In some embodiments, the perception device may be an independent device or may be integrated into another device. For example, the sound sensor may be part of a medical service device or a terminal device.

[0058] The infrastructure can be configured to support data transmission, storage and processing. For example, the infrastructure can include networks, computer room facilities, computing equipment, computing chips, storage devices, etc.

[0059] In some embodiments, at least a portion of the hardware devices of the physical hospital 110 are IoT devices. IoT devices refer to devices with sensors, processing capabilities, software, and other technologies that connect and exchange data with other devices and systems via the Internet or other communication networks. For example, one or more medical service devices and / or sensing devices of the physical hospital 110 are IoT devices and are configured to transmit collected data to the hospital support platform 140 for storage and / or processing.

[0060] User services may include any services provided to users by the hospital support platform 140. For example, user services include medical services provided to patients and / or accompanying persons, support services provided to staff of the physical hospital 110 and / or suppliers of the physical hospital 110, etc. In some embodiments, user services may be provided to patients, doctors, and hospital administrators through the user space application 120, which will be described in detail in the following description.

[0061] Public areas refer to shared spaces accessible to users (or portions of users) in the physical hospital 110. For example, public areas may include a reception area (eg, front desk), a waiting area, a corridor, etc., or any combination thereof.

[0062] The medical service process refers to the process of providing corresponding medical services to patients. The medical service process usually includes several links and / or steps, and the user needs to go through these links and / or steps to obtain the corresponding medical services. Exemplary medical service processes may include outpatient processes, inpatient processes, surgical processes, or similar processes, or any combination thereof. In some embodiments, the medical service process may include medical service processes corresponding to different departments, different diseases, etc. In some embodiments, a preset data collection protocol can be set and the standard links involved in the medical service process and how to collect data related to the medical service process can be specified.

[0063] The user space application 120 provides users with access to user services provided by the hospital support platform 140. The user space application 120 can be an application, a plug-in, a website, a small program, or any other suitable form. For example, the user space application 120 is an application installed on a user terminal device, and the application includes a user interface for the user to initiate a request and receive the corresponding service.

[0064] In some embodiments, the user space application 120 may include different applications corresponding to different types of users. For example, the user space application 120 includes a patient space application corresponding to the patient, a medical space application corresponding to the doctor, a management space application corresponding to the manager, etc., or any combination thereof. The user services provided by the patient space application, the medical space application, and the management space application are also referred to as patient space services, medical space services, and management space services, respectively. Exemplary patient space services include registration services, path guidance services, pre-consultation services, remote medical services, hospitalization services, discharge services, etc. Exemplary medical space services include scheduling services, surgical planning services, surgical simulation services, patient management services, remote ward rounds services, remote outpatient services, etc. Exemplary manager space services include monitoring services, medical service evaluation services, equipment parameter setting services, service parameter setting services, resource scheduling services, etc.

[0065] In some embodiments, the patient space application, the medical space application, and the management space application may be integrated into one user space application 120, and the user space application 120 may be configured to provide access portals for each type of user (e.g., patients, medical service providers, managers, etc.). As an example only, a specific user may have a corresponding account that can be used to log in to the user space application, view corresponding diagnosis and treatment data, and obtain corresponding user services.

[0066] According to some embodiments of the present application, by providing user space applications for different types of users, each type of user can easily obtain various user services that he / she may need on his / her corresponding user space application. In addition, currently users usually need to install various applications to obtain different user services, which leads to poor user experience and high development costs. Therefore, the user space application of the present application can improve user experience, improve service quality and efficiency, enhance service security, and reduce development or operation costs.

[0067] In some embodiments, the user space application 120 can be configured to provide relevant users of the physical hospital 110 with an access portal for interacting with the virtual hospital 130. For example, through the user space application 120, the user can enter instructions for retrieving the digital content of the virtual hospital 130 (e.g., hardware equipment, patient organs, digital twin models of public areas), view the digital content, and interact with the digital content. As another example, through the user space application 120, the user can communicate with a virtual character representing an intelligent body. In some embodiments, the public terminal of the hospital can install a tube space application, and the administrator account of the department corresponding to the public terminal can be logged in in the tube space application. Users can receive user services through the tube space application installed in the public terminal.

[0068] The virtual hospital 130 is a digital twin (ie, a virtual representation or virtual copy) of the physical hospital 110 and is used to simulate, analyze, predict, and optimize the operating status of the physical hospital 110. For example, the virtual hospital 130 may be a real-time digital copy of the physical hospital 110.

[0069] In some embodiments, the virtual hospital 130 may be presented to the user using digital technology. For example, when the relevant user interacts with the virtual hospital 130, at least a portion of the virtual hospital 130 may be presented to the relevant user using XR technology. As an example only, at least a portion of the virtual hospital 130 may be superimposed on the relevant user's real-world view using MR technology.

[0070] In some embodiments, the virtual hospital 130 may include a digital twin of a physical entity associated with the physical hospital 110. A digital twin refers to a virtual representation of a physical entity (e.g., a virtual copy, a mapping, a digital simulator). A digital twin can reflect and predict the state, behavior, and performance of a physical entity in real time. For example, the virtual hospital 130 may include a digital twin of at least a portion of the medical services, departments, users, hardware equipment, user services, public areas, medical service processes, etc. of the physical hospital 110. The digital twin of a physical entity may be in a variety of forms, including models, images, graphics, text, numerical values, and the like. For example, a digital twin may be a virtual hospital corresponding to a physical hospital, virtual personnel (e.g., virtual doctors, virtual nurses, and virtual patients) corresponding to personnel entities (e.g., doctors, nurses, and patients), virtual equipment (e.g., virtual imaging equipment and virtual scalpels) corresponding to medical service equipment (e.g., imaging equipment and scalpels), and the like.

[0071] In some embodiments, the digital twin may include one or more first digital twins and / or one or more second digital twins. The state of each first digital twin may be updated based on the update of the state of the corresponding physical entity. For example, in the process of mapping data related to the physical hospital 110 to the virtual hospital 130, one or more first digital twins may be updated. One or more second digital twins may be updated by at least one of the user space applications 120, and the update of each second digital twin may result in an update of the state of the corresponding physical entity. In other words, when the corresponding physical entity changes its state, the first digital twin may be updated accordingly; when the second digital twin is updated, the state of the corresponding physical entity also changes accordingly. For example, one or more first digital twins may include digital twins of public areas, medical services, users, hardware devices, etc., and one or more second digital twins may include digital twins of hardware devices, user services, medical service processes, etc. It should be understood that the digital twin may be a first digital twin or a second digital twin.

[0072] According to some embodiments of the present application, by generating a virtual hospital 130 that includes a digital twin of physical entities related to the physical hospital 110, the physical hospital 110 (including hardware equipment, users, user services, medical service processes, etc.) can be simulated and tested in a safe and controllable environment. Through virtual reality linkage (for example, real-time interaction between the physical hospital 110 and the virtual hospital 130), various medical scenarios can be predicted and responded to more accurately, thereby improving the quality and efficiency of medical services. In addition, the use of XR technology and virtual reality integration technology makes the interaction between relevant users more natural and intuitive, providing a more comfortable and efficient medical environment, thereby enhancing the user experience.

[0073] In some embodiments, the virtual hospital 130 may further include an intelligent agent that achieves self-evolution based on data related to the physical hospital 110 and AI technology.

[0074] An agent refers to an agent that acts in an intelligent manner. For example, an agent may include a computing / software entity that can autonomously learn and evolve, and perceive and analyze data to perform specific tasks and / or achieve specific goals (e.g., medical service processes). Through AI technology (e.g., reinforcement learning, deep learning, etc.), the agent can continuously learn and self-optimize in interaction with the environment. In addition, the agent can collect and analyze massive data (e.g., relevant data of the physical hospital 110) through big data technology, and mine patterns and learning rules from the data to optimize the decision-making process, thereby identifying environmental changes in uncertain or dynamic environments, responding quickly, and making reasonable judgments. For example, the agent can autonomously learn and evolve based on AI technology to adapt to changes in the physical hospital 110. As an example only, the agent can be built based on NLP technology (e.g., large language models, etc.), and can automatically learn and autonomously update through a large amount of language text (e.g., hospital business data and patient feedback information) to improve the quality of user services provided by the physical hospital 110.

[0075] In some embodiments, agents may include different types of agents corresponding to different medical service processes, different user services, different departments, different diseases, different hospital positions (e.g., nurses, doctors, technicians, etc.), different links in the medical service process, etc. Agents of a specific type are used to process tasks corresponding to a specific type. In some embodiments, an agent may correspond to different medical service processes (or different medical services, or different departments, or different diseases, or different hospital locations). In some embodiments, agents may operate with reference to basic configuration data (e.g., dictionaries, knowledge graphs, templates, etc.) of the departments and / or diseases corresponding to the agents. In some embodiments, multiple agents may collaborate and share information through network communications to jointly complete complex tasks.

[0076] In some embodiments, the configuration of the agent can be set. For example, basic configuration data used by the agent in operation can be set. The basic configuration data can include dictionaries, knowledge databases, templates, etc. For another example, the use rights of the agent can be set for different users. In some embodiments, the administrator of the physical hospital 110 can set the configuration of the agent through the management space application.

[0077] In some embodiments, the agent can be integrated into or deployed on a hardware device. For example, the agent corresponding to the inpatient service can be integrated into a hospital bed or a presentation device of a hospital bed. In some embodiments, the agent can be integrated into or deployed on an embodied intelligent robot. An embodied intelligent robot refers to a robotic system that combines physical presence (embodiment) with intelligent behavior (cognition). An embodied intelligent robot can be configured to interact with the real world in a way that mimics or supplements human capabilities, using physical form and cognitive functions to perform tasks, make decisions, and adapt to the environment. By leveraging artificial intelligence and sensor technology, embodied intelligent robots can operate autonomously, interact with the environment, and continuously improve performance. For example, an embodied intelligent robot can be configured with an agent corresponding to a surgical service and assist doctors in performing surgery.

[0078] In some embodiments, at least a portion of user services may be provided based on an agent. For example, at least a portion of user services may be provided to relevant users based on a processing result, wherein the processing result is generated by at least one of the agents based on data related to the physical hospital 110. As an example only, the data related to the physical hospital 110 may include data related to a medical service process of the physical hospital 110, the agent may include an agent corresponding to the medical service process, and user services may be provided to relevant users of the medical service process by processing the data using the agent corresponding to the medical service process.

[0079] The hospital support platform 140 may be configured to provide technical support to the medical service system 100. For example, the hospital support platform 140 may include computing hardware and software to support innovative technologies, including XR technology, AI technology, digital twin technology, data circulation technology, etc. In some embodiments, the hospital support platform 140 may include at least a storage device for data storage and a processing device for data calculation.

[0080] In some embodiments, the hospital support platform 140 can support the interaction between the physical hospital 110 and the virtual hospital 130. For example, the processing device of the hospital support platform 140 can obtain data related to the physical hospital 110 from the hardware device and map the data related to the physical hospital 110 to the virtual hospital 130. For example, the processing device of the hospital support platform 140 can update a part of the digital twin in the virtual hospital 130 (e.g., one or more first digital twins) based on the obtained data, so that each part of the digital twin in the virtual hospital 130 can reflect the updated state of the corresponding physical entity in the physical hospital 110. Based on this digital twin that is constantly updated with the corresponding physical entity, the user can understand the state of the physical entity related to the physical hospital 110 in real time, thereby realizing the monitoring and evaluation of the physical entity. As another example, the intelligent agent corresponding to the data related to the physical hospital 110 can be trained and / or updated based on the data related to the physical hospital 110, thereby self-evolving and self-learning.

[0081] In some embodiments, the hospital support platform 140 may support and / or provide user services to relevant users of the physical hospital 110. For example, in response to receiving a user service request from a user, a processing device of the hospital support platform 140 may provide a user service corresponding to the service request. For another example, in response to detecting that a user service needs to be provided to a user, the processing device of the hospital support platform 140 may control a physical entity or a virtual entity corresponding to the user service to provide the user service. For example, in response to detecting that a patient is admitted to a hospital ward, the processing device of the hospital support platform 140 may control a smart nursing cart to guide a nurse to the hospital ward to perform an admission examination on the patient.

[0082] In some embodiments, at least a portion of user services may be provided to the relevant user based on the interaction between the relevant user and the virtual hospital 130. Interaction refers to the interaction or influence (e.g., conversation, behavior, etc.) between the relevant user and the virtual hospital 130. For example, the interaction between the relevant user and the virtual hospital 130 may include the interaction between the relevant user and the digital twin in the virtual hospital 130, the interaction between the relevant user and the intelligent agent, the interaction between the relevant user and the virtual character, etc., or any combination thereof.

[0083] In some embodiments, at least a portion of user services may be provided to the relevant user based on the interaction between the relevant user and at least one of the digital twins. For example, an update instruction for a second digital twin input by a relevant user may be received through the user space application 120, and the corresponding physical entity of the second digital twin may be updated according to the update instruction. As another example, a user may view a first digital twin of a physical entity (e.g., a 3D digital twin model of a patient's organ or a hardware device) through the user space application 120 to understand the state of the physical entity. Optionally, the user may change the display angle, display size, etc. of the digital twin.

[0084] In some embodiments, the processing device of the hospital support platform 140 can present a virtual character corresponding to the intelligent agent through a user space application, interact with the relevant user, and provide at least part of the user service to the relevant user based on the interaction between the relevant user and the virtual character.

[0085] In some embodiments, the hospital support platform 140 may have a five-layer structure, including a hardware device layer, an interface layer, a data processing layer, an application development layer, and a service layer. Figure 3 In some embodiments, the hardware devices of the physical hospital 110 may be part of the hospital support platform 140 .

[0086] According to some embodiments of the present application, a virtual hospital corresponding to the physical hospital can be established by comprehensively integrating various internal and external resources of the physical hospital (e.g., medical service equipment, hospital personnel, medicines and consumables, etc.). The virtual hospital can reflect the real-time status (e.g., changes, updates, etc.) of the physical entities related to the physical hospital, thereby enabling monitoring and evaluation of the physical entities. This integration can provide accurate data support for the operation and intelligent decision-making of medical services. In addition, through the virtual hospital, users related to medical services can jointly establish an open and shared ecosystem, thereby promoting the innovation and improvement of medical services.

[0087] In addition, it is possible to provide full-life cycle healthcare services for patients that are linked inside and outside the hospital. The perspective of medical services has expanded from simple disease treatment to cover the entire life cycle of patients, including prevention, diagnosis, treatment, rehabilitation, health management, etc. By establishing linkages inside and outside the hospital, physical hospitals can better integrate online and offline resources to provide patients with comprehensive and continuous medical and health services. For example, through remote monitoring and online medical consultations, patients' health conditions can be followed up in real time, treatment plans can be adjusted in a timely manner, and treatment effects can be improved.

[0088] Figure 2 is a schematic diagram of an exemplary medical service system 200 according to some embodiments of the present application.

[0089] like Figure 2 As shown, the medical service system 200 may include a processing device 210, a network 220, a storage device 230, one or more medical service devices 240, one or more sensing devices 250, one or more patient terminals 260 of a patient 261, and one or more doctor terminals 270 of a doctor 271 associated with the patient 261. In some embodiments, the components in the medical service system 200 may be interconnected and / or communicated via wireless connections, wired connections, or a combination thereof. The connections between the components of the medical service system 200 may be variable.

[0090] The processing device 210 may process data and / or information obtained from the storage device 230, the medical service device 240, the sensing device 250, the patient terminal 260, and / or the doctor terminal 270. For example, the processing device 210 may map data related to a physical hospital to a virtual hospital corresponding to the physical hospital, and provide user services to the patient 261 and the doctor 271 through the patient terminal 260 and / or the doctor terminal 270, respectively, by processing the data related to the physical hospital. As another example, the processing device 210 may maintain a digital smart object, and provide user services to the patient 261 and the doctor 271 through the patient terminal 260 and / or the doctor terminal 270, respectively, by allowing the digital smart object to participate in processing the data related to the physical hospital.

[0091] In some embodiments, the processing device 210 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processing device 210 may be located locally or remotely from the medical service system 200. In some embodiments, the processing device 210 may be implemented on a cloud platform. For example, a cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-cloud cloud, a multi-cloud, etc., or a combination thereof.

[0092] In some embodiments, the processing device 210 may include one or more processors (e.g., a single-core processor or a multi-core processor). For illustration purposes only, only one processing device 210 is described in the medical service system 200. However, it should be noted that the medical service system 200 in the present application may also include multiple processing devices. Therefore, as in the present application, the operations and / or method steps performed by one processing device 210 may also be performed jointly or individually by multiple processing devices.

[0093] The network 220 may include any suitable network capable of facilitating information and / or data exchange for the healthcare service system 200. The network 220 may be or include a wired network, a wireless network (e.g., an 802.11 network, a Wi-Fi network), a Bluetooth™ network, a near field communication (NFC) network, etc., or any combination thereof.

[0094] The storage device 230 may store data, instructions, and / or any other information. In some embodiments, the storage device 230 may store data obtained from other components of the healthcare delivery system 200. In some embodiments, the storage device 230 may store data and / or instructions that the processing device 210 may execute or use to execute the exemplary methods described in this application.

[0095] In some embodiments, the data stored in the storage device 230 may include multimodal data. Multimodal data may include data in multiple forms (e.g., images, graphics, videos, text, etc.), various types of data, data obtained from different sources, data related to different medical services (e.g., diagnosis, surgery, rehabilitation, etc.), and data related to different users (e.g., patients, medical staff, managers, etc.). For example, the data stored in the storage device 230 may include medical data of the patient 261 reflecting the health status of the patient 261. For example, the medical data may include an electronic health record of the patient 261. An electronic health record refers to an electronic file that records various types of patient data (e.g., basic information, examination data, imaging data). For example, an electronic health record may include a three-dimensional model of multiple organs and / or tissues of the patient 261.

[0096] In some embodiments, the storage device 230 may include a mass storage device, a removable storage device, a volatile read-write memory, a read-only memory (ROM), etc., or any combination thereof. In some embodiments, the storage device 230 may include a data lake and a data warehouse. Figure 3 Describe in detail.

[0097] The medical service device 240 can be used to provide or assist medical services. Figure 2 As shown, the medical service equipment 240 includes a clinic terminal 240-1, a hospital bed 240-2, an intelligent surgery terminal 240-3, an intelligent nursing cart 240-4, an intelligent wheelchair 240-5, etc., or any combination thereof.

[0098] The clinic terminal 240-1 refers to a terminal device configured in the clinic for use by doctors and patients in the medical outpatient process. For example, the clinic terminal 240-1 may include one or more of a screen, a sound output component, an image sensor, or a sound sensor. The screen of the clinic terminal 240-1 may display a consultation interface, and data may be displayed on the consultation interface to facilitate communication between doctors and patients. Exemplary data may include electronic health records (or part thereof), pre-consultation records, medical images, 3D organ models, examination results, consultation recommendations, etc.

[0099] The hospital bed 240-2 refers to a bed in a hospital ward that can support inpatients and provide user services to patients. The hospital bed 240-2 may include a bed, a bedside terminal device, a bedside inspection device, a sensor, etc., or any combination thereof. The bedside terminal device may include an XR device, a display device, a mobile device, etc., or any combination thereof. In some embodiments, the hospital bed 240-2 may be controlled by an intelligent agent corresponding to an inpatient service, wherein the hospital bed may also be referred to as an intelligent hospital bed or a meta-hospital bed.

[0100] The intelligent surgical terminal 240-3 refers to a device configured with an intelligent agent for assisting surgery, and is controlled by an intelligent agent corresponding to the surgical service. The intelligent surgical terminal 240-3 can perceive the interaction (e.g., conversation, behavior, etc.) between the medical service provider, the patient, and the intelligent agent, and obtain the data captured by the sensing device 250, thereby providing surgical assistance. In some embodiments, the intelligent surgical terminal 240-3 can be configured to perform risk warnings for surgical operations based on the intelligent agent configured therein, generate surgical records of the surgical process, etc.

[0101] The smart nursing cart 240-4 refers to a nursing cart with an automatic driving function that can assist in the treatment and care of patients. For example, the smart nursing cart 240-4 can be configured to guide a nurse to a hospital ward for an admission examination of a patient. In some embodiments, the smart nursing cart can be controlled by an intelligent agent (e.g., an intelligent agent corresponding to an inpatient service, a nursing intelligent agent). In some embodiments, the smart nursing cart 240-4 may include a trolley, a presentation device, one or more inspection devices and / or nursing tools, a sensing device (such as an image sensor, a GPS sensor, a sound sensor, etc.), etc. In some embodiments, the smart nursing cart 240-4 may be configured to obtain relevant treatment and nursing information of the patient, and generate physical examination data, nursing data, etc. The physical examination data may include the patient's vital signs data. Nursing data may include detailed records of nursing operations, such as nursing time, nursing operators, nursing measures, patient reactions, etc.

[0102] Smart wheelchair 240-5 refers to a transportation device for intelligently picking up and dropping off patients. In some embodiments, smart wheelchair 240-5 can be configured to perform autonomous navigation through integrated sensors and maps, locate the patient's position using radio frequency identification devices (RFID), Bluetooth or Wi-Fi signals, and identify the patient through biometric technology. In some embodiments, smart wheelchair 240-5 can be controlled by an intelligent agent (e.g., an intelligent agent corresponding to inpatient services, an intelligent agent corresponding to surgical services). In some embodiments, smart wheelchair 240-5 can be configured to generate data (e.g., a record of the content of the interaction between the intelligent agent and the patient) by sensing interaction data through a built-in camera / sensor.

[0103] The sensing device 250 may be configured to collect sensing information related to the environment in which it is located. In some embodiments, the sensing device 250 may include a sensing device in the physical hospital 110. For example, the sensing device 250 may include an image sensor 250-1, a sound sensor 250-2, a temperature sensor, a humidity sensor, and the like.

[0104] The patient terminal 260 may be a terminal device that interacts with the patient 261. In some embodiments, the patient terminal 260 may include a mobile terminal 260-1, an XR device 260-2, a smart wearable device 260-3, etc. The doctor terminal 270 may be a terminal device that interacts with the doctor 271. In some embodiments, the doctor terminal 270 may include a mobile terminal 270-1, an XR device 270-2, etc. In some embodiments, the patient 261 may access user space applications (e.g., patient space applications) through the patient terminal 260, and the doctor 271 may access user space applications (e.g., medical space applications) through the doctor terminal 270. In some embodiments, the patient 261 and the doctor 271 may communicate remotely with each other through the patient terminal 260 and the doctor terminal 270, thereby providing telemedicine services, such as remote outpatient services, remote ward rounds services, remote follow-up services, etc.

[0105] The sensing device 250, the patient terminal 260, and the doctor terminal 270 may be configured as data sources to provide information to the medical service system 200. For example, these devices may transmit the collected data to the processing device 210, and the processing device 210 may provide user services based on the received data.

[0106] It should be noted that the description of the above-mentioned medical service systems 100 and 200 is intended to be illustrative, not to limit the scope of the present application. Many alternatives, modifications and variations will be apparent to those skilled in the art. The features, structures, methods and other features of the exemplary embodiments herein may be combined in various ways to obtain additional and / or optional exemplary embodiments. For example, the medical service system 200 may include one or more additional components, such as terminal devices of other users, public terminal devices of hospitals, etc. As another example, two or more components of the medical service system 200 may be integrated into a single component.

[0107] Figure 3 is a schematic diagram of an exemplary hospital support platform 300 according to some embodiments of the present application.

[0108] like Figure 3As shown, the hospital support platform 300 may include a hardware layer 310 (also referred to as a hardware module), an interface layer 320 (also referred to as an interface module), a data processing layer 330 (also referred to as a data processing module), an application development layer 340 (also referred to as an application development module), and a service layer 350 (also referred to as a service module). It should be understood that the "layers" and "modules" in this application are only used to logically divide the components of the hospital support platform and are not intended to be limiting.

[0109] The hardware layer 310 can be configured to provide a hardware foundation for the interaction between the real world and the digital world, and can include one or more hardware devices related to hospital operations. Exemplary hardware devices can include medical service devices, sensor devices, terminal devices, and basic devices.

[0110] The interface layer 320 may be connected to the hardware layer 310 and the data processing layer 330. The interface layer 320 may be configured to obtain data collected by the hardware devices of the hardware layer 310 and send the data to the data processing layer 330 for storage and / or processing. The interface layer 320 may also be configured to control at least a portion of the hardware devices of the hardware layer 310. In some embodiments, the interface layer 320 may include a hardware interface and a software interface (e.g., a data interface, a control interface).

[0111] The data processing layer 330 may be configured to store and / or process data. The data processing layer 330 may include a processing device, and a plurality of data processing units may be configured on the processing device. The data processing layer 330 may be configured to obtain data from the interface layer 320 and process the data through at least one data processing unit to implement user services related to hospital business.

[0112] The data processing unit may include various preset algorithms for implementing data processing. In some embodiments, the data processing layer 330 may include a processing device (e.g., Figure 2 The data processing unit may be configured on the processing device. In some embodiments, the data processing unit may include an XR unit configured to process data using XR technology to implement XR services, an AI unit (e.g., an intelligent unit) configured to process data using AI technology to implement AI services, a digital twin unit configured to process data using digital twin technology to implement digital twin services, a data circulation unit configured to process data using data circulation technology (e.g., blockchain technology, data privacy computing technology) to implement data circulation services, etc.

[0113] In some embodiments, the data processing layer 330 may also include a data center configured to store data. In some embodiments, the data center may adopt a lake warehouse integrated architecture, which may include a data lake and a data warehouse. The data lake can be used to store large amounts of data in a tamper-proof manner. The data warehouse can be used to store index data corresponding to the data in the data lake. The data stored in the data lake may include native (or original) data collected by hardware devices, derived data generated based on native data, and the like. In some embodiments, the data in the data lake can be processed by a processing device (e.g., processing device 210).

[0114] The application development layer 340 may be configured to support application development, publishing, subscription, etc. The application development layer 340 is also referred to as the ecosystem layer. In some embodiments, the application development layer 340 may be configured to provide an open interface for application developers to access or call at least a portion of the data processing unit and use at least a portion of the data processing unit to develop applications. Figure 3 As shown, the application development layer 340 can provide development toolkits, application markets, multi-tenant operation platforms, cloud official websites, workspaces, and other supporting toolkits to assist developers in their work.

[0115] The service layer 350 may be configured so that users related to hospital business can access user services related to hospital business through user space applications.

[0116] The present application provides a hospital support platform designed for the comprehensive management of various resources within the hospital, including hardware resources, software resources, and data resources. In certain embodiments, the platform further integrates a data processing unit capable of supporting advanced technologies, such as artificial intelligence, XR, digital twins, and blockchain. These advanced technologies are used to improve the service efficiency and quality of the healthcare industry. For example, artificial intelligence technology enables the autonomous evolution and continuous optimization of hospital operations, while XR and digital twin technologies facilitate the creation and maintenance of virtual hospitals. The virtual hospital can interact with users to provide an immersive and novel service experience. In addition, the platform also includes an application development layer for granting access to these advanced technologies to third-party developers in the healthcare industry. This access promotes an open ecosystem, promotes the development and innovation of applications, and thus promotes the advancement of medical services.

[0117] Figure 4 is a module diagram of an exemplary surgical planning and execution system according to some embodiments of the present specification.

[0118] like Figure 4As shown, the surgery planning and execution system 400 may include a surgery planning module 410 , a surgery simulation module 420 , a preoperative preparation module 430 , a surgery execution module 440 and a postoperative review module 450 .

[0119] In some embodiments, the surgical planning module 410 is used to generate a surgical plan. In some embodiments, the surgical planning module 410 generates a surgical plan based on the patient's patient data (e.g., patient personal data, historical diagnosis and treatment data, medical examination data, etc.), doctor feedback information (e.g., second feedback information), and / or sensory information (e.g., fourth sensory information).

[0120] In some embodiments, the surgery planning module 410 generates an initial surgery plan based on the patient data of the patient, and displays the initial surgery plan to the doctor. The surgery planning module 410 further generates a surgery plan based on the initial surgery plan and the second feedback information.

[0121] In some embodiments, the surgery planning module 410 determines the difficulty coefficient of the surgery based on the patient data of the patient, and determines whether an expert meeting needs to be held based on the difficulty coefficient of the surgery. In response to determining that an expert meeting needs to be held, the surgery planning module 410 presents a virtual meeting space through the doctor's second terminal device and the remote expert's fourth terminal device, respectively. During the expert meeting, the surgery planning module 410 obtains the fourth sense information collected by the second terminal device and the fourth terminal device, and generates a surgery plan based on the patient data and the fourth sense information.

[0122] In some embodiments, the surgical planning module 410 is used to process the surgical plan and at least a portion of the patient data to generate a risk assessment result of the surgical plan. The surgical planning module 410 determines risk prevention measures based on the risk assessment results, and presents the risk assessment results and risk prevention measures of the surgical plan to the doctor. For example, the surgical planning module 410 generates a risk assessment result of the surgical plan by processing the surgical plan and at least a portion of the patient data using a risk assessment model.

[0123] In some embodiments, the surgical planning module 410 is used to generate explanation materials for explaining the surgical plan, and to display the explanation materials to the patient and the doctor simultaneously through at least one terminal device. The surgical planning module 410 determines first feedback information about the surgical plan based on the second sensing information collected by one or more second sensing devices during the explanation of the surgical plan; and confirms or updates the surgical plan of the patient based on the surgical plan and the first feedback information.

[0124] In some embodiments, the surgery simulation module 420 is used to perform surgery simulation. In some embodiments, the surgery simulation module 420 generates a virtual surgery scene for surgery simulation based on the surgery plan, and presents the virtual surgery scene to the doctor through the doctor's second terminal device. The surgery simulation module 420 obtains the interaction instructions about the virtual surgery device input by the doctor through the second terminal device or the interaction device corresponding to the virtual surgery device; and based on the interaction instructions, updates the virtual surgery site and virtual surgery device in the virtual surgery scene.

[0125] In some embodiments, the preoperative preparation module 430 is used to perform preoperative preparation for the patient. In some embodiments, the preoperative preparation module 430 visualizes the patient's lesions and surrounding tissues or organs based on the patient's three-dimensional anatomical model through a terminal device (XR glasses or smart display terminal) worn by a participant (e.g., a patient, a doctor, a patient's family member, etc.); and obtains the doctor's explanation materials to the patient and / or his family members through the sensor on the terminal device worn by the doctor, and sends them to the terminal device of the patient or the patient's family member, so that the patient or the patient's family member receives the above explanation materials.

[0126] In some embodiments, the preoperative preparation module 430 is used to generate explanation materials for explaining the surgical plan, and to display the explanation materials to the patient and the doctor simultaneously through at least one terminal device.

[0127] In some embodiments, the surgery execution module 440 is used for surgery execution. In some embodiments, the surgery execution module 440 performs surgery on the patient based on the surgery plan.

[0128] In some embodiments, during the operation, the operation execution module 440 monitors environmental data and personnel behavior through the sensing device. In some embodiments, the operation execution module 440 displays the department's operating room view and the current operating room real-time view on the interactive device based on the operation process. In some embodiments, the operation execution module 440 obtains the first sensing information of the patient during the operation collected by one or more first sensing devices in the operating room, and performs event of interest (EOI) detection on the first sensing information; in response to detecting the occurrence of EOI, one or more preset operations corresponding to EOI are performed.

[0129] In some embodiments, the postoperative review module 450 is used to perform postoperative review. In some embodiments, the postoperative review module 450 performs postoperative care based on surgical records and medical orders. In some embodiments, the postoperative review module 450 monitors the patient's postoperative signs through vital sign monitoring instruments (e.g., electrocardiogram, blood pressure monitor, etc.) in the ward to determine whether the patient's postoperative vital signs are within the normal postoperative vital sign range, whether there are abnormalities, or whether the recovery progress is normal. In some embodiments, the postoperative review module 450 updates the medical orders based on the patient's postoperative signs. In some embodiments, the postoperative review module 450 determines the postoperative care plan based on the updated medical orders. In some embodiments, the postoperative review module 450 generates the doctor's surgical results and operation records based on the surgical records and medical orders so that the doctor can review the surgical process.

[0130] More details about the surgery planning module 410, the surgery simulation module 420, the preoperative preparation module 430, the surgery execution module 440, and the postoperative review module 450 can be found elsewhere in this specification (for example, Figure 5 and its related description), which will not be repeated here.

[0131] It should be understood that Figure 4 The system and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware.

[0132] It should be noted that the above description of the surgical planning and execution system 400 and its modules is only for convenience of description and is not intended to limit the present specification to the embodiments described above. It is understandable that, after understanding the principle of the system, a person skilled in the art may arbitrarily combine the modules or form a subsystem connected to other modules without departing from the principle. Such variations are all within the scope of protection of the present specification.

[0133] Figure 5 is a flowchart of an exemplary surgical planning and execution method according to some embodiments of the present specification.

[0134] In some embodiments, process 500 may be executed by an application scenario of a surgical planning and execution system (e.g., processing device 210 in application scenario 100) or surgical planning and execution system 400. For example, process 500 is stored in a storage device (e.g., storage device 230, storage unit of a surgical planning and execution system) in the form of a program or instruction. Figure 4 When the modules shown execute programs or instructions, the process 500 can be implemented. Figure 5 As shown, in some embodiments, process 500 may include the following steps.

[0135] Step 510 , performing surgical planning. In some embodiments, step 510 is performed by the processing device 210 or the surgical planning module 410 .

[0136] Surgical planning refers to the process of developing a surgical plan for a patient (e.g., an optimal surgical plan or multiple feasible surgical plans, etc.). The optimal surgical plan can be a surgical plan developed for a patient with the lowest surgical risk, the best postoperative recovery effect, or the most suitable for the patient's physical condition, and the surgical plan will be used to perform surgery on the patient. In some embodiments, the surgical plan includes at least one of the surgical site (e.g., abdomen, chest, etc.), surgical time, surgical steps, estimated duration of the operation, anesthesia dosage, surgeon, type of surgery (e.g., minimally invasive surgery or laparoscopic surgery or invasive surgery, etc.), surgical incision location, cutting depth, cutting path, implants (e.g., heart stents, prostheses, etc.), implant paths, types and quantities of surgical tools, risk prevention measures, etc.

[0137] In some embodiments, the processing device 210 or the surgical planning module 410 generates a surgical plan (including an initial surgical plan) based on the patient's patient data (e.g., patient personal data, historical diagnosis and treatment data, medical examination data, etc.), doctor feedback information (e.g., second feedback information), and / or perception information (e.g., fourth perception information).

[0138] In some embodiments, the processing device 210 or the surgical planning module 410 generates an initial surgical plan based on the patient data of the patient and displays the initial surgical plan to the doctor. Further, the processing device 210 or the surgical planning module 410 can generate a surgical plan based on the initial surgical plan and the second feedback information. The second feedback information is the feedback on the initial surgical plan input by the doctor through the second terminal device. The second terminal device refers to an extended reality device used or worn by the doctor. For example, the second terminal device can be an XR glasses worn by the doctor. For more information about this embodiment, see Figure 8 and its related description.

[0139] The terminal device refers to a device worn by a user (e.g., a patient, a doctor, etc.) that can access, view, and interact with a virtual reality space (e.g., a virtual surgical scene). For example, the terminal device may be an extended reality device (e.g., XR glasses, etc.). In some embodiments, the terminal device includes a first terminal device of the patient, a second terminal device of the doctor, a third terminal device of the patient's family, and a fourth terminal device of a remote expert or doctor from another department.

[0140] In some embodiments, when formulating a surgical plan, the processing device 210 or the surgical planning module 410 may generate a surgical plan by introducing an expert meeting (also called a multidisciplinary team meeting).

[0141] In some embodiments, the processing device 210 or the surgical planning module 410 can determine the difficulty coefficient of the operation based on the patient data of the patient, and determine whether an expert meeting is needed based on the difficulty coefficient of the operation. In response to determining that an expert meeting is needed, the processing device 210 or the surgical planning module 410 presents a virtual meeting space through the doctor's second terminal device and the remote expert's fourth terminal device, respectively. The processing device 210 or the surgical planning module 410 obtains the fourth perception information collected by the second terminal device and the fourth terminal device during the expert meeting, and generates a surgical plan based on the patient data and the fourth perception information. The fourth terminal device refers to an extended reality device worn by the remote expert. For example, the fourth terminal device may be XR glasses worn by the remote expert. For more information about this embodiment, see Figure 7 and its related description.

[0142] In some embodiments, the processing device 210 or the surgical planning module 410 generates a risk assessment result of the surgical plan by processing the surgical plan and at least a portion of the patient data using the risk assessment model. The processing device 210 or the surgical planning module 410 determines risk prevention measures based on the risk assessment result, and displays the risk assessment result and risk prevention measures of the surgical plan to the doctor. For more information about this embodiment, see Figure 8 and its related description.

[0143] In some embodiments, the processing device 210 or the surgical planning module 410 generates explanation materials for explaining the surgical plan. The processing device 210 or the surgical planning module 410 can display the explanation materials to the patient and the doctor at the same time through at least one terminal device. On this basis, in some embodiments, the processing device 210 or the surgical planning module 410 can also determine the first feedback information about the surgical plan based on the second perception information collected by one or more second perception devices during the explanation of the surgical plan. The processing device 210 or the surgical planning module 410 can confirm or update the patient's surgical plan based on the surgical plan and the first feedback information. For more information about the above embodiments, please refer to Fig.12 and its related description.

[0144] Step 520 , performing surgery simulation. In some embodiments, step 510 is performed by the processing device 210 or the surgery simulation module 420 .

[0145] Surgical simulation refers to the process in which doctors practice surgery in a safe and controllable environment, improve surgical plans and / or improve surgical skills. For example, for complex or rare surgeries, doctors can use extended reality devices (e.g., second XR devices) to simulate surgery on virtual patients in virtual surgical scenes (e.g., extended reality surgical scenes) based on the formulated surgical plan to identify potential risk points during the operation, so as to formulate corresponding risk prevention measures. Potential risk points refer to risks in the surgical process, abnormal state of organs, or other emergency situations. For example, the potential risks of patients may be manifested as low blood pressure, bradypnea, liver failure, etc. In addition, for multiple surgical plans, doctors can use extended reality devices (e.g., second XR devices) to simulate each surgical plan in a virtual surgical scene to compare the pros and cons of different surgical plans and determine the optimal surgical plan. In addition, doctors can use extended reality devices (e.g., second XR devices) to repeatedly practice the same operation in a virtual surgical scene to deepen their understanding and memory of surgical operations and improve the accuracy and proficiency of surgical operations.

[0146] In some embodiments, the processing device 210 or the surgical planning module 410 generates a virtual surgical scene for surgical simulation based on the surgical plan, and presents the virtual surgical scene to the doctor through the doctor's second terminal device. The processing device 210 or the surgical planning module 410 can obtain the interactive instructions about the virtual surgical device input by the doctor through the second terminal device or the interactive device corresponding to the virtual surgical device. The processing device 210 or the surgical planning module 410 can update the virtual surgical site and virtual surgical device in the virtual surgical scene based on the interactive instructions. For more information about this embodiment, see Fig. 9 , Fig.10 and its related description.

[0147] In some embodiments, surgery simulation is an optional step, that is, process 500 may not include step 520 and directly jump from step 510 to step 530 .

[0148] Step 530 , performing preoperative patient preparation. In some embodiments, step 530 is performed by the processing device 210 or the surgery preparation module 430 .

[0149] Preoperative patient preparation refers to the preparations for the patient before surgery or before entering the operating room. Figure 5 As shown, preoperative patient preparation may include preoperative patient education 532 and preoperative guidance 534.

[0150] Preoperative patient education refers to the process of explaining the patient's condition, surgical plan, and postoperative recovery to the patient and / or the patient's family.

[0151] In some embodiments, preoperative patient education is performed manually and proactively by a physician.

[0152] Exemplarily, the processing device 210 or the preoperative preparation module 430 visualizes the patient's lesions and their surrounding tissues or organs based on the patient's three-dimensional anatomical model through a terminal device (XR glasses or smart display terminal) worn by participants (e.g., patients, doctors, patients' families, etc.). Then, the doctor (e.g., doctor 271) can send explanation materials to the patient and / or his family members, and the processing device 210 or the preoperative preparation module 430 can obtain the doctor's explanation materials to the patient and / or his family members through the sensor on the terminal device worn by the doctor, and send it to the terminal device of the patient or the patient's family members, so that the patient or the patient's family members receive the above explanation materials. The above explanation materials may include explanations of various surgical plans and simulated surgical steps, potential risks, the patient's postoperative recovery process, discharge criteria, etc. The above explanation materials may be in the form of voice information, pattern information, text information, etc.

[0153] In some embodiments, the processing device 210 or the preoperative preparation module 430 generates explanation materials for explaining the surgical plan. The processing device 210 or the preoperative preparation module 430 can simultaneously display the explanation materials to the patient and the doctor through at least one terminal device. For more information about preoperative care and the above embodiments, please refer to Fig.11 , Fig.12 and its related description.

[0154] Preoperative guidance refers to the process of guiding the patient to complete preoperative preparations before surgery. Preoperative guidance may include patient reception, patient verification, preoperative care, preoperative cleaning, establishment of intravenous access, etc. For example, the processing device 210 or the preoperative preparation module 430 uses the nurse agent to guide the nurse to perform preoperative cleaning and / or intravenous access establishment on the patient. The nurse agent can interact with the nurse through the terminal device (XR glasses or smart display terminal) worn by the nurse and guide the nurse. In some embodiments, the processing device 210 or the preoperative preparation module 430 determines the planned route from the patient's current location to the waiting area of ​​the operating room. The processing device 210 or the preoperative preparation module 430 can control the smart chair to transport the patient to the waiting area along the planned route. The processing device 210 or the preoperative preparation module 430 can collect the patient's sensory information (e.g., biological information) through one or more third sensory devices in the waiting area. The processing device 210 or the preoperative preparation module 430 can verify the patient's identity based on the biological information. For more information about preoperative guidance, see Fig.14 and its related description.

[0155] Step 540 , surgery execution. In some embodiments, step 540 is executed by the processing device 210 or the surgery execution module 440 .

[0156] Surgery execution refers to the operation related to surgery performed on the patient after entering the operating room. The surgical execution process can include preoperative preparation, intraoperative matters and postoperative matters.

[0157] In some embodiments, the processing device 210 or the surgery execution module 440 performs surgery on the patient based on the surgical plan. In some embodiments, during the surgical execution, the processing device 210 or the surgery execution module 440 monitors environmental data and personnel behavior through sensing devices. In some embodiments, the processing device 210 or the surgery execution module 440 displays the department's operating room view and the current operating room real-time view on the interactive device based on the surgical progress. In some embodiments, the processing device 210 or the surgery execution module 440 obtains the first sensory information of the patient during the operation collected by one or more first sensory devices in the operating room, and performs event of interest (EOI) detection on the first sensory information. In response to detecting the occurrence of EOI, the processing device 210 or the surgery execution module 440 can execute one or more preset operations corresponding to the EOI. For more information about this embodiment, see Fig.17 and its related description.

[0158] Step 550 , performing postoperative review. In some embodiments, step 550 is performed by the processing device 210 or the postoperative review module 450 .

[0159] Postoperative review can include at least one of updating the doctor's instructions, generating the doctor's surgical results and operation records (so that the doctor can review the surgical process), and generating a postoperative care plan. For more information about postoperative review, please refer to Fig.18 and its related description.

[0160] It should be noted that the above description of process 500 is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to process 500 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0161] Figure 6 FIG. 1 is a flowchart of an exemplary surgical planning according to some embodiments of this specification. Figure 6 As shown, in some embodiments, the process 600 may include the following steps. In some embodiments, the process 600 is executed by the processing device 210 or the surgery planning module 410 .

[0162] Step 610, assessing the patient's condition based on the patient data and generating a condition assessment result.

[0163] Patient data refers to physiological or pathological information related to a patient. Patient data may include patient personal data, historical diagnosis and treatment data, medical examination data, patient digital twins (e.g., a 3D anatomical model of a patient), etc.

[0164] Patient personal data refers to the basic information of the patient. Patient personal data may include the patient's gender, age, height, weight, etc. For example, the patient's personal data is "male, age 39, height 174cm, weight 73kg".

[0165] Historical diagnosis and treatment data refers to the data related to treatment and care before the patient's current surgery or before the current moment. Historical diagnosis and treatment data may include the patient's medical history, historical treatment records (e.g., historical surgery records, historical chemotherapy records, etc.), hospitalization care records (e.g., medical advice records, nursing records, medication records, etc. during hospitalization), historical medication records (e.g., medication records before the current hospitalization, etc.), etc.

[0166] Medical examination data refers to data reflecting the examination results of medical examinations performed on patients. Medical examinations may include blood tests, biochemical tests, urine tests, immunological tests, microbiological tests, allergen tests, imaging tests (e.g., CT scans, MR scans, PET scans, ultrasound scans, etc.), etc. Medical examination data may be examination reports of medical examinations, such as blood test reports, urine test reports, immunological test reports, CT imaging reports, MR imaging reports, PET imaging reports, etc.

[0167] Patient data can be obtained from the Unified Health Records (UHR) or supplemented by the doctor. UHR is an archive for storing patient data. Exemplarily, the patient's personal data (for example, after the first log-in), physical examination report, medical examination report, etc. can be stored in the folder corresponding to the patient in the UHR (for example, a folder named by the patient number, name, etc.). Exemplarily, the doctor initiates a patient data acquisition instruction by operating the interactive terminal of the doctor's workstation (for example, a second terminal device, an intelligent display terminal), and the processing device retrieves the patient data of the current patient from the UHR based on the acquisition instruction and displays it through the interactive terminal.

[0168] The condition assessment result may reflect the patient's health status and / or condition. The condition assessment result may include the stage of the condition, the progression of the condition, the rate of deterioration of the condition, the patient's tolerance to drugs or surgery, etc. For example, if the condition is cancer, the condition assessment result may be "the stage of the condition is advanced, the cancer has spread, the cancer develops and spreads rapidly, the patient has no history of drug allergies, and the patient has a strong tolerance to surgery."

[0169] The evaluation method of the situation evaluation result may include manual evaluation and / or intelligent evaluation.

[0170] Manual assessment refers to the determination of the condition assessment results by the condition assessor (e.g., doctor) based on the patient data. In some embodiments, after the patient data is retrieved and displayed on the display interface of the interactive device (e.g., the second terminal device worn by the doctor, the smart display terminal in the doctor's office), the doctor can comprehensively assess the patient's health status and condition based on the patient data, and input the condition assessment results through the interactive device (e.g., through voice input, interface input, etc.). Exemplarily, the doctor can access the patient data in the UHR through the smart display terminal of the doctor's workstation, accurately locate the lesion through the tools provided by the UHR, and view the exact location, size, and relationship with the surrounding tissues of the lesion, so as to intuitively understand the patient's condition and conduct a preoperative assessment. The sensing device (e.g., microphone, gesture sensor, etc.) on the second terminal device worn by the doctor can capture the doctor's input information (e.g., voice information, gesture information), etc., to generate a condition assessment result.

[0171] Intelligent assessment refers to processing patient data using a condition assessment model to determine a condition assessment result. The condition assessment model may be a machine learning model, the input of the condition assessment model may include patient data, and the output may include a condition assessment result. In some embodiments, the condition assessment model may be integrated in the processing device 210. After the processing device 210 determines the patient's condition assessment result using the condition assessment model, it may output the condition assessment result to the doctor through the interactive terminal of the doctor's workstation (e.g., a second terminal device, a smart display screen, etc.). In some embodiments, the processing device 210 may also update the condition assessment result based on the doctor's feedback information on the displayed condition assessment result.

[0172] In some embodiments, the condition assessment model can be obtained by training. In some embodiments, the condition assessment model can be obtained by training based on multiple first training samples with first labels. For example, the processing device 210 or the surgical planning module 410 can input multiple first training samples with first labels into the initial condition assessment model, determine the value of the loss function through the first label and the result of the initial condition assessment model, and iteratively update the parameters of the initial condition assessment model based on the value of the loss function. When the loss function of the initial condition assessment model meets the first end condition of the end of training, the model training is completed, and a trained condition assessment model is obtained. Among them, the first training sample may include sample patient data, and the sample patient data may be determined based on the patient data of historical patients. The first label may be the condition assessment result of the historical patient corresponding to the first training sample. The first label can be determined by manual labeling or automatic labeling by the system. The first end condition may be the convergence of the loss function (for example, the mean square error of the loss function is less than the first error threshold), the number of iterative training reaches the first number threshold, etc.

[0173] In some embodiments, before determining the condition assessment result, the processing device 210 or the surgical planning module 410 may determine whether the patient data meets the expert consultation conditions. In response to the patient data meeting the expert consultation conditions (for example, the patient's lesion location is dangerous, the patient's age is greater than a preset value, the patient has other lesions in addition to the target lesion, the patient's condition is complex and it is difficult to determine the type of lesion, etc.), the processing device 210 or the surgical planning module 410 may execute step 613, evaluate the patient's condition through an expert meeting, and generate a condition assessment result. In some embodiments, the processing device 210 may determine whether the expert consultation conditions are met based on the patient data.

[0174] In some embodiments, the processing device 210 may execute step 613 in response to the patient data satisfying the expert consultation condition (i.e., an expert meeting is required), evaluate the patient's condition through the expert meeting, and generate a condition evaluation result. For example, the attending physician may proactively make an appointment for expert consultation services (e.g., multidisciplinary consultation) through the consultation application window in the doctor's space displayed on the second terminal device; doctors from other departments may choose to accept or refuse to participate in the discussion and agree on a discussion time (e.g., with the help of the fourth terminal device or the smart display terminal of the doctor's workstation) according to their personal clinical business arrangements.

[0175] In some embodiments, participants in the expert meeting (e.g., remote experts) can remotely participate and view patient data to generate a condition assessment result. For example, doctors from various departments participating in the consultation (including local doctors and remote doctors) can jointly view the patient's three-dimensional model and other data in a virtual environment through the XR glasses they wear, share their respective views in real time, and thus determine the condition assessment result.

[0176] Step 620: Generate a surgical plan based on the condition assessment result.

[0177] Combined with the above, the surgical plan may include the surgical site, surgical time, surgical steps, estimated surgical duration, anesthesia dosage, surgeon, surgical type, surgical incision location and cutting depth, cutting path, implant path, surgical tool type and quantity, etc. For more information about surgical planning, please refer to Fig.12 and its related description.

[0178] In some embodiments, the processing device 210 or the surgical planning module 410 can determine the surgical plan using the plan generation model. For example, the condition assessment results and patient needs are processed and analyzed using the plan generation model to determine the surgical plan. The plan generation model can be a machine learning model. In some embodiments, the input of the plan generation model can include the condition assessment results, and the output can include the surgical plan. In some embodiments, the plan generation model can be integrated in the processing device 210. After the processing device 210 determines the surgical plan using the plan generation model, the surgical plan output can be displayed to the doctor through the interactive device of the doctor's workstation (e.g., a second terminal device, a smart display, etc.). In some embodiments, the processing device 210 can confirm or update the surgical plan based on the doctor's feedback information (e.g., the first feedback information) obtained from the interactive device.

[0179] In some embodiments, the input of the plan generation model may include the condition assessment results and patient needs. Patient needs refer to the patient's special needs for surgery. For example, the patient's needs may be "the length of the surgical incision is less than 5 cm".

[0180] In some embodiments, the input of the plan generation model may also include patient image data. The patient image data refers to scanned image data of the patient's lesion. For example, the patient image data may be a CT scan image containing the patient's lesion.

[0181] In some embodiments, the input to the plan generation model may include patient data and fourth sensory information. Figure 7 (e.g., step 750).

[0182] In some embodiments, the plan generation model can be obtained by training. In some embodiments, the plan generation model can be obtained by training based on multiple second training samples with second labels. For example, the processing device 210 or the surgical planning module 410 can input multiple second training samples with second labels into the initial plan generation model, determine the value of the loss function through the second label and the result of the initial plan generation model, and iteratively update the parameters of the initial plan generation model based on the value of the loss function. When the loss function of the initial plan generation model meets the second end condition of the end of training, the model training is completed, and the trained plan generation model is obtained. Among them, the second training sample may include at least one of the sample condition assessment result, the sample patient demand, the sample patient image, the sample patient data, and the sample fourth perception information. The second label can be the surgical plan of the historical patient corresponding to the second training sample, and the second label can be determined by manual annotation or automatic annotation of the system. The second end condition can be the convergence of the loss function (for example, the mean square error of the loss function is less than the second error threshold), the number of iterative training reaches the second number threshold, etc.

[0183] In some embodiments, the doctor can determine the surgical plan based on the condition assessment results. For example, the doctor can determine the necessity of surgery for the current patient, the most suitable surgical method and technology, the surgical time and the estimated duration of the surgery, etc. based on the condition assessment results, thereby determining the surgical plan.

[0184] In some embodiments, the processing device 210 may generate an initial surgical plan based on the patient data of the patient and display the initial surgical plan to the doctor. The processing device 210 may generate a surgical plan based on the initial surgical plan and the second feedback information about the initial surgical plan input by the doctor through the second terminal device. For more details, see Figure 8 Described in.

[0185] In some embodiments, after the surgical plan is generated, the operator (e.g., the surgeon in charge) can adjust the surgical plan. For example, the operator can shorten or extend the surgical execution time in the surgical plan, shorten or extend the implantation path, and other information in the surgical plan.

[0186] In some embodiments, when the expert consultation conditions are met (for example, the location of the patient's lesion is dangerous, the patient's age is older than a preset value, the patient has other lesions in addition to the target lesion, the patient's condition is complex and it is difficult to determine the type of lesion, the surgical difficulty coefficient is greater than the difficulty coefficient threshold, etc.), the processing device 210 can execute step 613 to determine the surgical plan through an expert meeting. In some embodiments, the processing device 210 can determine whether the expert consultation conditions are met (whether a multidisciplinary team meeting is needed) based on the patient data.

[0187] In some embodiments, the processing device 210 may execute step 613 in response to determining that an expert meeting is needed, and determine the surgical plan based on the patient data and condition assessment results through the expert meeting. For example, the attending physician may invite a specific department doctor or remote expert to participate in the patient's surgical planning discussion through the consultation application window in the doctor's space displayed by the second terminal device (e.g., the second XR device) to actively make an appointment for expert consultation services (e.g., multidisciplinary consultation). The invited department doctors and remote experts can choose to accept or refuse to participate in the discussion and agree on a discussion time (e.g., with the help of a fourth terminal device or a smart display terminal of a workstation) based on their personal clinical business arrangements.

[0188] In some embodiments, participants in the expert meeting (e.g., doctors from other departments, remote experts) can remotely participate and view (e.g., based on a fourth terminal device) patient data and condition assessment results to determine the surgical plan. Exemplarily, doctors from various departments participating in the meeting (including local doctors and remote doctors) can jointly view the patient's three-dimensional anatomical model and other data and condition assessment results in a virtual environment through their respective XR glasses, share their respective views in real time (e.g., share views through voice discussion), and the perception device (e.g., microphone, image sensor) perceives the behavior data of each participant (e.g., voice data, operation data, etc.) in real time to determine the surgical plan.

[0189] In some embodiments, the processing device 210 or the surgical planning module 410 can determine the surgical difficulty coefficient based on the patient's patient data, and determine whether an expert meeting is needed based on the surgical difficulty coefficient. In response to determining that an expert meeting is needed, the processing device 210 or the surgical planning module 410 can present a virtual meeting space through the doctor's second terminal device (e.g., a second XR device) and the remote expert's fourth terminal device, respectively. During the expert meeting, the processing device 210 or the surgical planning module 410 can obtain fourth sensory information collected by the second terminal device and the fourth terminal device. The processing device 210 or the surgical planning module 410 can generate a surgical plan based on the patient data and the fourth sensory information. For more information about this embodiment, see Figure 7 and its related description.

[0190] In some embodiments, the processing device 210 can optimize the surgical plan through an artificial intelligence assistant (e.g., Copilot). For example, a doctor can call Copilot through the intelligent display terminal of the workstation, and the processing device 210 uses Copilot in combination with historical data (e.g., historical diagnosis and treatment data of other patients with the same lesion, etc.) to optimize the surgical plan.

[0191] Step 630: Perform risk assessment on the surgical plan and generate risk assessment results.

[0192] The risk assessment result refers to the prediction result of the risk conditions that may occur during the operation of the patient based on the operation plan. The risk condition refers to the harmful conditions that the patient may encounter during the operation, such as heavy bleeding, cardiac arrest, etc.

[0193] The risk assessment of the surgical plan may include manual assessment and / or intelligent assessment.

[0194] Manual assessment refers to the determination of risk assessment results by a condition assessor (e.g., a doctor). In some embodiments, the doctor can evaluate the surgical plan based on at least a portion of the patient data to generate a risk assessment result. For example, the doctor can view the patient's three-dimensional anatomical model through the smart display terminal in the office, and combine the surgical plan to determine the risks and complications that may occur during the operation. The sensing device (e.g., microphone, gesture sensor, etc.) on the second terminal device worn by the doctor can capture the doctor's input information (e.g., voice information, gesture information, etc.), etc., and then generate a risk assessment result.

[0195] Intelligent assessment refers to the use of a risk assessment model to evaluate the surgical plan and determine the risk assessment results. The risk assessment model may be a trained machine learning model. The input of the risk assessment model may include at least a portion of the patient data and the surgical plan, and the output may include the risk assessment results. In some embodiments, the risk assessment model may be integrated in the processing device 210. After the processing device 210 determines the risk assessment results of the patient using the risk assessment model, the risk assessment results may be output and displayed to the doctor through the interactive device of the doctor's workstation (e.g., a second terminal device, a smart display screen, etc.). In some embodiments, in order to ensure the accuracy of the risk assessment results, the processing device 210 may combine intelligent assessment and manual assessment. For example, the processing device 210 may first perform an intelligent assessment and send the results of the intelligent assessment to a condition assessor (e.g., a doctor), and the condition assessor may confirm or modify the results of the intelligent assessment to determine the final risk assessment results.

[0196] In some embodiments, the risk assessment model can be obtained by training. In some embodiments, the risk assessment model can be obtained by training based on multiple third training samples with third labels. For example, the processing device 210 or the surgical planning module 410 can input multiple third training samples with third labels into the initial risk assessment model, determine the value of the loss function by the third label and the result of the initial risk assessment model, and iteratively update the parameters of the initial risk assessment model based on the value of the loss function. When the loss function of the initial risk assessment model meets the third end condition of the end of training, the model training is completed, and a trained risk assessment model is obtained. Among them, the third training sample can include sample patient data and sample surgical plans, wherein the sample patient data and sample surgical plans can be obtained based on the patient data and surgical plans of historical patients. The third label can be the risk that occurs during the operation of the historical patient corresponding to the third training sample, and the third label can be determined by manual labeling or automatic labeling by the system. The third end condition can be the convergence of the loss function (for example, the mean square error of the loss function is less than the third error threshold), the number of iterative training reaches the third number threshold, etc.

[0197] In some embodiments, the processing device 210 can determine risk prevention measures based on the risk assessment results, and present the risk assessment results and risk prevention measures of the surgical plan to the doctor. In some embodiments, the doctor can formulate corresponding risk prevention measures based on the risk assessment results. For more information about risk prevention measures, please refer to Figure 8 .

[0198] Figure 7 FIG. 1 is a flow chart of an exemplary surgical plan generation process according to some embodiments of this specification. Figure 7 As shown, in some embodiments, the process 700 may include the following steps. In some embodiments, the process 700 may be executed by the processing device 210 or the surgery planning module 410 .

[0199] Step 710, obtaining patient data of the patient.

[0200] Patient data can be obtained, for example, by accessing the patient's UHR (electronic health record) or by supplementing the doctor's record. For more information about UHR and patient data, see Figure 6 and its related description.

[0201] Step 720 , determining a surgical difficulty factor based on the patient data of the patient.

[0202] The surgical difficulty coefficient reflects the degree of difficulty of the surgery. The surgical difficulty coefficient can be characterized based on the surgical difficulty coefficient value. Exemplarily, the surgical difficulty coefficient value can be an integer in the range of [1, 10], and the larger the value, the higher the surgical difficulty coefficient (i.e., the more difficult the surgical operation is).

[0203] The determination method of the surgical difficulty coefficient may include manual determination and / or intelligent determination.

[0204] Manual determination refers to the determination of the operation difficulty coefficient by a difficulty coefficient determination personnel (e.g., a doctor expert) based on patient data. For example, the doctor can determine the operation difficulty coefficient based on operation simulation or historical operation experience through a second terminal device worn or an intelligent display terminal of a workstation.

[0205] Intelligent determination refers to processing patient data using a coefficient determination model to determine the surgical difficulty coefficient. The coefficient determination model may be a machine learning model, the input of the coefficient determination model may include patient data, and the output may include a surgical difficulty coefficient value. In some embodiments, the coefficient determination model may be integrated in the processing device 210. After the processing device 210 determines the surgical difficulty coefficient using the coefficient determination model, the surgical difficulty coefficient output may be displayed to the doctor through an interactive device (e.g., a second terminal device, an intelligent display terminal, etc.) of the doctor's workstation. Further, the doctor may input modification information of the surgical difficulty coefficient through an interactive device (e.g., a microphone, a gesture sensor, a touch screen, etc. on the second terminal device worn by the doctor) to modify the surgical difficulty coefficient determined by the coefficient determination model (e.g., increase the surgical difficulty coefficient value determined by the coefficient determination model by 2 as the final surgical difficulty coefficient). In some embodiments, in order to ensure the accuracy of the surgical difficulty coefficient, the processing device 210 may combine intelligent determination with manual determination. For example, the processing device 210 may first perform intelligent determination and send the result of the intelligent determination to the coefficient determination personnel, who may confirm or modify the result of the intelligent determination to determine the final surgical difficulty coefficient.

[0206] In some embodiments, the coefficient determination model can be obtained by training. In some embodiments, the coefficient determination model can be obtained by training based on a plurality of fourth training samples with a fourth label. For example, the processing device 210 or the surgical planning module 410 can input a plurality of fourth training samples with a fourth label into the initial coefficient determination model, determine the value of the loss function by the fourth label and the result of the initial coefficient determination model, and iteratively update the parameters of the initial coefficient determination model based on the value of the loss function. When the loss function of the initial coefficient determination model meets the fourth end condition of the end of training, the model training is completed, and a trained coefficient determination model is obtained. Among them, the fourth training sample may include sample patient data, and the sample patient data may be obtained based on the patient data of historical patients. The fourth label may be the surgical difficulty coefficient corresponding to the fourth training sample, and the fourth label may be determined by manual annotation or automatic annotation of the system. The fourth end condition may be that the loss function converges (for example, the mean square error of the loss function is less than the fourth error threshold), the number of iterative training reaches the fourth number threshold, etc.

[0207] Step 730, determining whether it is necessary to hold an expert meeting based on the difficulty coefficient of the operation.

[0208] In some embodiments, the processing device 210 can determine whether the surgical difficulty coefficient is greater than the difficulty coefficient threshold to determine whether an expert meeting is needed. In response to determining that the surgical difficulty coefficient is greater than the difficulty coefficient threshold, step 740 is executed to conduct a remote expert meeting; in response to determining that the surgical difficulty coefficient is not greater than the difficulty coefficient threshold, the processing device 210 directly generates a surgical plan (for example, generating a surgical plan using the process described in step 620). The difficulty coefficient threshold can be manually preset or automatically determined by the system.

[0209] Step 740, conduct an expert meeting.

[0210] In some embodiments, the processing device 210 can present a virtual conference space on the second terminal device of the doctor and the fourth terminal device of the remote expert. During the expert conference, the processing device 210 can obtain the fourth perception information collected by the second terminal device and the fourth terminal device, and generate a condition assessment result and / or a surgical plan through the remote expert conference.

[0211] The virtual conference space refers to a virtual conference scene generated by an extended reality device (eg, the aforementioned second terminal device and fourth terminal device).

[0212] The fourth perception information may include voice data of the doctor or remote expert acquired by the acoustic sensor or microphone. The acoustic sensor or microphone may be configured in the second terminal device and the fourth terminal device. In some embodiments, the fourth perception information may include data related to the state assessment result and / or data related to the surgical plan.

[0213] For example, in response to determining that an expert meeting is needed, the processing device 210 may generate a virtual meeting space. Participants of the expert meeting (e.g., Figure 7 The local doctor 741, remote doctor 742, remote expert 743, etc. shown in the figure can enter the virtual conference space through an interactive device (for example, a second terminal device, a fourth terminal device), and then can view the patient data presented in the virtual conference space (for example, a three-dimensional anatomical model of the patient) and share their respective views in real time (for example, based on voice discussions in the virtual conference space with the second terminal device and the fourth terminal device).

[0214] Step 750, generating a surgical plan based on the patient data and the fourth sensory information.

[0215] In combination with the above, the processing device 210 can generate a surgical plan using a plan generation model. In some embodiments, the input of the plan generation model may include patient data and fourth sensory information, and the output may include a surgical plan. Exemplarily, the processing device 210 or the surgical planning module 410 can use the plan generation model to process and analyze the patient data and fourth sensory information to determine the surgical plan. In some embodiments, the input of the plan generation model may include patient data, condition assessment results, patient needs, patient image data, and fourth sensory information. For more information about the plan generation model, see Figure 6 and its related description.

[0216] In some embodiments of the present specification, the above-mentioned method for generating a surgical plan can be used to comprehensively evaluate the overall health status of the patient before surgery, so that a surgical plan suitable for the patient can be formulated, the surgical risks can be minimized, and the adaptability and specificity of the surgical plan to the patient can be improved; by introducing expert meetings, instant feedback and discussion among multidisciplinary experts in different geographical locations during the formulation of the surgical plan can be achieved, thereby solving difficult problems in surgical planning and improving the success rate of subsequent operations.

[0217] Figure 8 FIG. 1 is a flow chart of an exemplary surgical plan generation process according to other embodiments of this specification. Figure 8 As shown, in some embodiments, the process 800 may include the following steps. In some embodiments, the process 800 may be executed by the processing device 210 or the surgery planning module 410 .

[0218] Step 810, obtain patient data of the patient. For more information about patient data, see Figure 6 or Figure 7 and its related description.

[0219] Step 820 , generating an initial surgical plan based on the patient data of the patient.

[0220] In combination with the above, the initial surgical plan may include the surgical site, surgical time, surgical steps, estimated surgical duration, anesthesia dosage, surgeon, surgery type, surgical incision location and cutting depth, cutting path, implant path, surgical tool type and quantity, etc.

[0221] In some embodiments, the processing device 210 may generate an initial surgical plan based on the patient data of the patient using the plan generation model. In this embodiment, the second training sample may include sample patient data, and the second label may be a sample initial surgical plan corresponding to the sample patient data.

[0222] Step 830, present the initial surgical plan to the physician.

[0223] like Figure 8As shown, the processing device 210 or the surgical planning module 410 can send the initial surgical plan to the second terminal device 270 to show it to the doctor 271. The doctor 271 can view the initial surgical plan through the second terminal device 270. The second terminal device 270 refers to a terminal device used or worn by the doctor 271, such as Figure 8 As shown, the second terminal device 270 may include a mobile terminal device 270-1 (e.g., a smartphone or tablet used by the doctor 271), a second XR device 270-2, and a desktop terminal device (e.g., a laptop used by the doctor 271), etc. Exemplarily, the processing device 210 or the surgical planning module 410 may send the initial surgical plan in text form to the second XR device 270-2 worn by the doctor 271. The second XR device 270-2 may present the initial surgical plan in a virtual space for the doctor 271 to review.

[0224] Step 840: Obtain second feedback information about the initial surgical plan input by the doctor through the second terminal device.

[0225] The second feedback information refers to the feedback information made by the doctor regarding the initial surgical plan. For example, the second feedback information may be the adjustment information made by the doctor to the initial surgical plan (for example, reducing the incision depth by 0.5 cm). Figure 8 As shown, the doctor 271 can express the adjustment information of the initial surgical plan by voice, and the perception device (for example, a microphone) on the second XR device 270-2 can collect the doctor's voice information and generate corresponding second feedback information.

[0226] Step 850: Generate a surgical plan based on the initial surgical plan and the second feedback information. The second feedback information refers to feedback on the initial surgical plan input by the doctor through the second terminal device.

[0227] In some embodiments, the processing device 210 or the surgical planning module 410 can adjust the initial surgical plan based on the second feedback information to obtain a surgical plan. For example, if the incision depth in the initial surgical plan is 3 cm and the second feedback information is "reduce the incision depth by 0.5 cm", the incision depth in the generated surgical plan is 2.5 cm.

[0228] At step 860 , a risk assessment result of the surgical plan is determined based on the surgical plan and at least a portion of the patient data.

[0229] In combination with the above, in some embodiments, the processing device 210 can process the surgical plan and at least a portion of the patient data through the risk assessment model to generate a risk assessment result of the surgical plan. The processing device 210 can determine risk prevention measures based on the risk assessment result. The processing device 210 can display the risk assessment result and risk prevention measures of the surgical plan to the doctor.

[0230] For example, Figure 8 As shown in FIG. 8 , in step 860 , the processing device 210 may obtain at least a portion of the patient data of the patient.

[0231] The at least a portion of the patient data may include historical diagnosis and treatment data, medical examination data, or patient twins in the patient data.

[0232] At step 870 , the processing device 210 may obtain a risk assessment model.

[0233] The risk assessment model is a trained machine learning model. For more information on the training of the risk assessment model, see Figure 6 and its related description.

[0234] In step 880, the processing device 210 may process at least a portion of the surgical plan and patient data using the risk assessment model to obtain a risk assessment result. Figure 8 As shown, the processing device 210 or the surgical planning module 410 can send the risk assessment result to the second terminal device 270 and display it to the doctor 271.

[0235] In step 890, the processing device 210 may determine risk prevention measures based on the risk assessment results. Risk prevention measures refer to countermeasures for risk conditions formulated based on the risk assessment results. For example, if the risk condition in the risk assessment results is massive bleeding, the risk prevention measures may include preparing arterial embolism, preparing additional hemostatic gauze and blood bags, etc. For another example, if the risk condition in the risk assessment results is cardiac arrest, the risk prevention measures may include preparing cardiac stimulants, etc. The risk prevention measures may be determined manually and / or intelligently by the processing device 210.

[0236] In some embodiments, Figure 8 As shown, the processing device 210 or the surgical planning module 410 may send the risk prevention measures to the second terminal device 270 and display the risk prevention measures to the doctor 271 .

[0237] The doctor 271 may view the risk assessment results and / or risk prevention measures through the second terminal device 270 (eg, the second XR device 270 - 2 ).

[0238] In some embodiments, the doctor 271 may call Copilot through the intelligent display terminal of the workstation to determine the risk assessment results and / or risk prevention measures of the surgical plan through Copilot.

[0239] In some embodiments, before determining the risk prevention measures, the processing device 210 or the surgical planning module 410 may determine whether the patient data meets the expert consultation conditions (i.e., whether an expert meeting is needed). In response to the patient data meeting the expert consultation conditions (for example, the patient's lesion is located in a dangerous location, the patient is older, the patient has other symptoms in addition to the target lesion, the patient's condition is complex and it is difficult to determine the type of lesion, etc.), the processing device 210 may evaluate the surgical plan, the patient's condition, and the risk assessment results through an expert meeting to determine the risk prevention measures. In some embodiments, the processing device 210 may determine whether the expert consultation conditions are met based on the patient data.

[0240] In some embodiments, the processing device 210 can respond to the patient data satisfying the expert consultation conditions (i.e., the need to hold an expert meeting), evaluate the patient's condition and risk assessment results through the expert meeting, and determine risk prevention measures. For example, the attending physician can proactively make an appointment for expert consultation services (e.g., multidisciplinary consultation) through the consultation application window in the doctor's space displayed on the second terminal device; doctors from other departments can choose to accept or refuse to participate in the discussion and agree on a discussion time (e.g., with the help of the fourth terminal device or the smart display terminal of the doctor's workstation) according to their personal clinical business arrangements.

[0241] In some embodiments, participants in the expert meeting (e.g., remote experts) can remotely participate and view the surgical plan, patient condition, and risk assessment results, and then determine risk prevention measures. For example, doctors from various departments participating in the consultation (including local doctors and remote doctors) can view the patient's three-dimensional model and other data in a virtual environment through their respective XR glasses, share their respective views in real time, and thus determine risk prevention measures.

[0242] Fig. 9 FIG. 1 is a flowchart of an exemplary surgical simulation according to some embodiments of the present specification. Fig. 9 As shown, in some embodiments, process 900 may include the following steps. In some embodiments, process 900 may be performed by processing device 210 or surgery simulation module 420. In some embodiments, process 900 may be performed by a surgery simulation agent. For more information about agents, see Fig.17 and its related description.

[0243] Step 910: Generate a virtual surgical scene.

[0244] The virtual surgical scene can be generated based on the digital twin technology and presented to the doctor 271 through an extended reality device (e.g., the second XR device 270-2). The virtual surgical scene can be a virtual form of an actual operating room.

[0245] In some embodiments, the virtual surgical scene may include one or more virtual surgical devices 913 and a virtual surgical site 915 .

[0246] Virtual surgical equipment refers to the virtual form of real surgical equipment required for surgery. For example, virtual surgical equipment may include virtual scalpels, virtual hemostats, virtual implants, virtual blood transfusion kits, virtual hemostatic gauze, etc.

[0247] The virtual surgical site 915 refers to the virtual form of the part of the patient that needs surgery. For example, the virtual surgical site can be a three-dimensional anatomical model of the patient's chest cavity and the organs and tissues in the chest cavity.

[0248] In some embodiments, the processing device 210 can generate a virtual surgical scene for surgical simulation based on the surgical plan. Exemplarily, the processing device 210 can determine the surgical site based on the surgical plan, and use three-dimensional modeling and other technologies to build a three-dimensional anatomical model corresponding to the surgical site at a preset ratio (for example, the volume ratio of the actual surgical site to the three-dimensional anatomical model is 1:1). The processing device 210 can determine the model and specifications of the target implant based on the surgical plan, select the target implant model from the database, or determine a personalized target implant model (i.e., a virtual target implant). The processing device 210 can determine the target surgical equipment to be used based on the surgical plan, and build a three-dimensional model of the target surgical equipment at a preset ratio (for example, the volume ratio of the surgical equipment to the three-dimensional model is 1:1).

[0249] Step 920, perform simulated surgery.

[0250] In some embodiments, the processing device 210 may generate a simulated surgery plan based on a virtual surgical site and one or more virtual surgical devices.

[0251] The simulated surgery plan may include the surgical plan, surgical operation process, surgical emergencies and the number of repetitions of the simulated surgery.

[0252] The surgical plan for a simulated surgery may include the name of the simulated surgery (for example, simulated tumor resection), the type of simulated surgery (open surgery or implantation surgery), simulated surgical tools and specifications (for example, the specifications of a simulated scalpel), simulated consumable requirements (for example, the quantity required for virtual hemostatic gauze), simulated implant specifications (for example, the model and size of the prosthesis, etc.), operation methods (for example, manual operation by a doctor or using minimally invasive instruments, etc.), etc.

[0253] In some embodiments, the processing device 210 can generate a simulated surgical plan based on patient data and / or historical data in combination with a virtual surgical scenario. For example, the processing device 210 can generate a simulated surgical plan based on the patient's historical diagnosis and treatment data, medical examination data, and historical surgical cases of the same type, surgical results (successful or failed surgery), surgical complications, surgical risk points, standard operating procedures and technical points for various types of surgery, etc.

[0254] In some embodiments, the processing device 210 may determine a simulated surgery plan based on the purpose of the surgery simulation. In combination with the above, the surgery simulation is to improve the surgery plan and / or improve the surgery skills, that is, the surgery simulation purpose may include improving the surgery plan or improving the surgery skills.

[0255] For example, if the purpose of the surgical simulation is to improve surgical skills, the simulated surgical plan may include further subdividing the surgical operation process, such as subdividing it into multiple steps such as virtual surgical site disinfection, scalpel cutting, postoperative suturing, and multiple repeated training.

[0256] As another example, if the purpose of the surgical simulation is to improve the surgical plan, the simulated surgical plan may include rehearsing the surgical process under different surgical plans, complications during the surgical process, risk points, etc., to help doctors improve their coping capabilities and / or improve the surgical plan (e.g., the operating procedure). For example, the processing device 210 may determine the time point when the patient's blood pressure during the simulated surgery is much lower than the average blood pressure of the patient during the historical surgery (e.g., the patient's blood pressure during the simulated surgery is lower than the average blood pressure of the patient during the historical surgery, and the difference between the patient's blood pressure and the average blood pressure is greater than 20 mmHg) as an emergency trigger point.

[0257] Further, the processing device 210 can perform a simulated surgery based on the simulated surgery plan. In some embodiments, the processing device 210 can present a virtual surgery scene to a simulated surgery participant (e.g., doctor 271) through a second terminal device (e.g., a second XR device 270-2). In some embodiments, the simulated surgery may include a teaching phase and a practical phase. The teaching phase refers to the phase in which the simulated surgery process is presented to the simulated surgery participant based on an extended reality device (e.g., a second XR device 270-2). The practical phase refers to the phase in which the simulated surgery participant performs a virtual surgery.

[0258] In some embodiments, Fig. 9As shown, doctor 271 (e.g., surgeon and / or medical student) can enter the virtual surgery scene by wearing the second XR device 270-2, and operate the patient's three-dimensional anatomical model in the virtual surgery scene through the second XR device 270-2 or an interactive device corresponding to one or more virtual surgery devices (e.g., sensory wearable device 271-3), thereby performing simulated surgery in the virtual surgery scene. In some embodiments, the sensory wearable device may include sensory gloves, sensory bracelets, sensory clothing, etc.

[0259] In some embodiments, doctor 271 (e.g., a surgeon and / or a medical student) can perform simulated surgery by operating on a three-dimensional anatomical model of a patient in a virtual surgical scene using real simulated instruments in the physical space (e.g., a scalpel, an operating table, forceps, etc.).

[0260] In some embodiments, the processing device 210 may obtain interactive instructions for one or more virtual surgical devices input by the doctor through the second XR device 270-2 or an interactive device corresponding to one or more virtual surgical devices (e.g., a sensory wearable device). For example, the processing device 210 may obtain interactive instructions for one or more virtual surgical devices input by the doctor 271 during the simulated surgery through the sensory wearable device 271-3. The interactive instructions reflect the doctor's operation data on one or more virtual surgical devices. For example, the operation data on one or more virtual surgical devices may include data such as the type of one or more virtual surgical devices used by the doctor 271, the fixed position of the one or more virtual surgical devices, the moving direction, and the moving amplitude (e.g., the moving distance and angle).

[0261] In some embodiments, the interaction instruction may be a voice instruction, or other operation instructions (e.g., key input, gesture input, etc.) input by the doctor through the second XR device 270-2 or the interaction device (e.g., the sensor wearable device 271-3). Exemplarily, the interaction instruction may be a gesture operation instruction such as picking up, putting down, and moving one or more virtual surgical devices by the doctor 271 through the sensor wearable device 271-3.

[0262] In some embodiments, the processing device 210 can generate operation feedback in the virtual surgical scene based on the interactive instructions. Operation feedback refers to the virtual surgical operation on the virtual surgical site and / or one or more virtual surgical devices presented in the virtual surgical scene. For example, the processing device 210 can present the corresponding surgical operation in the virtual surgical scene based on the gesture operation instruction (for example, pick up the virtual scalpel and control the virtual scalpel to scratch the surgical site, pick up the virtual hemostat and control the virtual hemostat to clamp the blood vessel, etc.), thereby controlling one or more virtual surgical devices to complete the operation corresponding to the instruction to realize simulated surgery at the virtual surgical site. The gesture operation instruction may include the type of virtual surgical device used, the fixed position of the virtual surgical device, the moving direction, the moving amplitude (for example, the moving distance and angle), etc. obtained by the sensing wearable device 271-3.

[0263] In some embodiments, the processing device 210 can update the virtual surgical site and one or more virtual surgical devices in the virtual surgical scene based on the interactive instructions to generate operation feedback. Fig.10 Related description in .

[0264] In some embodiments, the processing device 210 can present an emergency situation during a simulated surgery to help doctors practice their ability to respond to emergencies. An emergency situation refers to a risk condition that may occur during a surgery, such as severe bleeding or cardiac arrest. The surgical emergency situation can be triggered at a preset time point, randomly, or according to interactive instructions of one or more virtual surgical devices input by the doctor. For example, the processing device 210 can present a corresponding emergency situation (e.g., severe bleeding caused by an incorrect incision position, etc.) in a virtual surgical scene according to interactive instructions of one or more virtual surgical devices input by the doctor 271.

[0265] In some embodiments, the processing device 210 can obtain the doctor's (e.g., doctor 271) processing data for surgical emergencies (e.g., reaction time, processing method, processing time, processing results, etc. after the emergency occurs). Exemplarily, the processing device 210 can preset that the virtual patient has cardiac arrest 10 minutes after the operation, and monitor the doctor's 271 processing method for cardiac arrest (e.g., cardiopulmonary resuscitation or injection of cardiac stimulant, etc.) and processing results (e.g., the time it takes for the patient's heartbeat to recover or the failure of heartbeat recovery).

[0266] In some embodiments, the processing device 210 may provide prompts based on the doctor's real-time operation data. For example, when the doctor 271 makes an operation error, the processing device 210 may provide prompts through the second XR device 270-2 and record the error operation.

[0267] In some embodiments, the number of repeated trainings of the simulated surgery can be adjusted based on the operation data of the simulated surgery. For example, the more wrong operations the doctor has, the more repeated trainings of the simulated surgery are required.

[0268] Step 930, generating a simulation evaluation report.

[0269] The simulation evaluation report may include a record of the simulated surgical process and / or surgical evaluation. For example, when the purpose of surgical simulation is to improve surgical skills, the content of the simulation evaluation report may include whether the surgical steps are accurate, whether the surgical tools are used correctly, the stability of the implant and its compatibility with surrounding tissues, the record and number of incorrect operations, progress evaluation (for example, increasing accuracy of surgical operations and fewer incorrect operations are considered progress), personalized learning suggestions, etc. For another example, when the purpose of surgical simulation is to improve the surgical plan, the content of the simulation evaluation report may include the handling of emergency situations and their accuracy, surgical complications, etc.

[0270] In some embodiments, the processing device 210 may generate a simulation evaluation report based on the simulation data of the virtual surgical site and one or more virtual surgical devices during the surgical simulation. The simulation data is the usage data of the virtual surgical device (e.g., the type of virtual surgical device used, the number of times used, the purpose of use, etc.), and the operation data of the virtual surgical site using the virtual surgical device (e.g., suturing, cutting, puncturing, implanting, etc.).

[0271] In some embodiments, the processing device 210 can determine the optimal surgical plan based on the simulation evaluation report. For example, when the surgical type is an implantation surgery, the processing device 210 can determine the optimal implantation plan (the implantation plan that best suits the patient) based on the simulation data of multiple implantation plans. The implantation plan may include the implant model, implant implantation location, implant fixation method, amount of tissue resection around the implant, etc.

[0272] In some embodiments, the processing device 210 may determine whether the surgical plan needs to be optimized based on the simulation data or the simulation evaluation report. In response to determining that the surgical plan needs to be optimized, the processing device 210 may update the surgical plan based on the simulation data. For more details, see Fig.10 Related description in .

[0273] Fig.10 1000 is a flowchart of an exemplary surgical simulation according to some embodiments of the present specification. In some embodiments, the process 1000 may be executed by the processing device 210 or the surgical simulation module 420.

[0274] In some embodiments, Fig.10As shown, in step 1010, the processing device 210 or the surgical simulation module 420 can obtain a surgical plan. More descriptions of surgical plans can be found elsewhere in this specification (e.g., Figure 5 ) related description.

[0275] In step 1020, the processing device 210 or the surgical simulation module 420 may generate a virtual surgical scene for surgical simulation based on the surgical plan. For example, the surgical plan may be used as a simulated surgical plan for surgical simulation. In combination with the above, the virtual surgical scene may include a virtual surgical site and one or more virtual surgical devices. For more detailed descriptions, see Fig. 9 and its related description.

[0276] In some embodiments, the processing device 210 or the surgery simulation module 420 may present a virtual surgery scene to the doctor 271 through the second terminal device 270 (e.g., the second XR device 270-2). The doctor 271 may enter the virtual surgery scene through the second terminal device 270 (e.g., the second XR device 270-2), and perform a virtual surgery on the virtual surgery site through the second terminal device 270 or an interactive device corresponding to the virtual surgery device (e.g., the sensory wearable device 271-3).

[0277] In step 1030, the processing device 210 or the surgical simulation module 420 may obtain interactive instructions of one or more virtual surgical devices input by the doctor 271 through the second terminal device 270 (e.g., the second XR device 270-2) or an interactive device corresponding to one or more virtual surgical devices (e.g., the sensory wearable device 271-3).

[0278] In some embodiments, the processing device 210 or the surgery simulation module 420 may update the virtual surgery site in the virtual surgery scene and one or more virtual surgery devices in the virtual surgery scene based on the interaction instruction.

[0279] Updating the virtual surgical site and one or more virtual surgical devices in the virtual surgical scene means updating the form or position of the virtual surgical site and the virtual surgical device based on the content of the interactive instruction after obtaining the interactive instruction. For example, taking the interactive instruction of "lowering the scalpel tip until it contacts the skin, then moving downward to cut into the skin 1cm, and then cutting 3cm to the left" as an example, the processing device 210 can present a virtual incision 1cm deep and 3cm long in the left-right direction on the skin surface of the virtual surgical site in the virtual operating room; assuming that the height of the virtual scalpel from the skin of the virtual surgical site before the update is 5cm, then after the update, the virtual scalpel moves 6cm downward and 3cm to the left compared to before the update.

[0280] In step 1035, the processing device 210 or the surgery simulation module 420 may determine an emergency situation that may occur in the virtual surgery scene based on the interactive instruction. Based on the emergency situation that may occur, the processing device 210 or the surgery simulation module 420 may update the virtual surgery site in the virtual surgery scene and one or more virtual surgery devices in the virtual surgery scene.

[0281] In conjunction with the above, an emergency situation refers to a sudden adverse situation that may occur during surgery. For example, an emergency situation may be heavy bleeding at a virtual surgical site.

[0282] In some embodiments, the processing device 210 or the surgery simulation module 420 may process the interaction instruction using an emergency situation determination model to determine an emergency situation. The emergency situation determination model may be a machine learning model, the input of which may be a surgery plan and an interaction instruction, and the output of which may be whether an emergency situation is caused and / or the type of emergency situation (e.g., heavy bleeding, decreased blood pressure, cardiac arrest, etc.).

[0283] Exemplarily, taking the emergency situation of heavy bleeding as an example, the method for updating the virtual surgical site and one or more virtual surgical devices based on the emergency situation may be: adding a large amount of virtual blood at the contact position between the virtual scalpel and the virtual surgical site through the second XR device 270 - 2 .

[0284] In step 1040 , the processing device 210 or the surgery simulation module 420 may obtain simulation data of the virtual surgery site and the one or more virtual surgery devices during the surgery simulation process.

[0285] In conjunction with the above, simulation data refers to the usage data of virtual surgical equipment and the operation data of virtual surgical sites using one or more virtual surgical equipment. For example, simulation data may include the incision depth, incision size, amount of bleeding, movement path of the virtual scalpel, implantation path and implantation position of the virtual implant, etc.

[0286] In step 1050, the processing device 210 or the surgery simulation module 420 may determine whether the surgery plan needs to be optimized based on the simulation data. In response to the need to optimize the surgery plan, the processing device 210 or the surgery simulation module 420 may update the surgery plan based on the simulation data.

[0287] In some embodiments, the processing device 210 or the surgical simulation module 420 can determine whether the preset update conditions are met based on the simulation data, thereby determining whether the surgical plan needs to be optimized. If the preset update conditions are met, it is determined that the surgical plan needs to be optimized; otherwise, the surgical plan does not need to be optimized. The preset update conditions may be complications caused by the surgery, the postoperative recovery time is greater than the preset recovery time, and the postoperative vital signs are not within the preset range.

[0288] In some embodiments, the processing device 210 or the surgical simulation module 420 can update the surgical time, surgical steps, estimated surgical duration, anesthesia dosage, incision location and cutting depth, cutting path, implant specifications, etc. of the surgical plan based on the simulation data. For example, the processing device 210 can update the cutting depth in the surgical plan from 3.6 cm to 3 cm.

[0289] In some embodiments of the present specification, simulated surgery can be used to simulate complex surgical operations, implant schemes, and emergency rehearsals. The surgical team can practice and refine medical skills in a risk-free environment, which helps reduce errors in real operations, thereby improving patient safety and the success rate of operations. On the other hand, simulated surgery can be used for doctor training. As an efficient and economical training method, simulated surgery can quickly improve the surgical capabilities of trainees. The training content can also be adjusted according to the trainees' personal learning progress and needs to achieve personalized learning and further improve training efficiency.

[0290] Fig.11 is a schematic diagram of exemplary preoperative patient education according to some embodiments of the present specification.

[0291] In conjunction with the above, preoperative patient education is the process of explaining the patient's condition, surgical plan, and simulating the patient's postoperative recovery to the patient and / or the patient's family.

[0292] In some embodiments, Fig.11 As shown, the processing device 210 can generate a virtual patient education space 1120 and explain the patient's condition and surgical plan to the patient and / or the patient's family through the virtual patient education space 1120.

[0293] The virtual patient education space 1120 is a virtual space that can be viewed through a terminal device (e.g., a first terminal device, a second terminal device, a third terminal device). In some embodiments, a virtual region of interest 1121, a surgical video 1122, and a surgical consent form 1123 may be displayed in the virtual patient education space 1120. For example, Fig.11 As shown, the doctor 271 (for example, the surgeon in charge or the doctor who makes the surgical plan, etc.), the patient 261 and the patient's family 1133 can access the virtual patient education space 1120 through the terminal devices they wear (for example, XR devices), and can then view the virtual area of ​​interest 1121, the surgical video 1122 and the surgical consent form 1123, and can communicate with each other.

[0294] The region of interest refers to the lesion and / or its surrounding tissues or organs. The virtual region of interest 1121 can be a three-dimensional anatomical model corresponding to the patient's lesion and its surrounding tissues or organs, which is used to visualize the morphology of the patient's lesion, the relationship between the lesion and the surrounding tissues or organs, and analyze the progression of the disease (for example, the impact of the lesion on the patient's physiological function).

[0295] The surgical video 1122 is used to visualize the process of performing surgery at the surgical site (e.g., region of interest) of the patient based on the surgical plan. For example, the surgical video 1122 can show the complete surgical operation process, possible emergency situations, and postoperative recovery process, etc. For example, the surgical video 1122 can show the content of different surgical plans, the wound size, lesion resection location, cutting depth, cutting path, resection risk or implantation path of different surgical plans in the virtual region of interest 1121, and the recovery progress and nursing requirements of the patient after discharge in the form of video.

[0296] The surgical consent form 1123 refers to the electronic informed consent form regarding the surgical plan that the patient and his / her family need to sign simultaneously after the patient and his / her family communicate with the doctor (for example, remote communication in the virtual patient education space 1120) to jointly determine the optimal surgical plan, and the doctor has informed the patient and his / her family of precautions during the operation.

[0297] In some embodiments, the doctor 271 can use the virtual region of interest 1121 to show the patient's lesion condition and explain the progression of the disease to the patient 261 and the patient's family 1133. In some embodiments, the processing device 210 can simulate the preoperative lesion morphology, possible spread range of the lesion, and the relationship between the lesion and surrounding tissues or organs of the virtual region of interest 1121 in the virtual patient education space 1120. For example, the doctor 271 can use the virtual region of interest 1121 and the virtual lesion located therein to explain to the patient 261 and the patient's family 1133 the location, size, degree of harm, and current progression stage of the lesion (e.g., whether the lesion is in the early / middle / late stage, whether metastasis has occurred, etc.) of the lesion, as well as the prediction of the later development of the disease.

[0298] In some embodiments, the doctor 271 can explain to the patient 261 and the patient's family 1133 the process of performing surgery based on the surgical plan at the patient's surgical site, as well as the patient's postoperative recovery process, based on the surgical video 1122. For example, the doctor 271 can explain to the patient 261 and the patient's family 1133 the surgical operation process of each surgical plan, possible emergencies, and postoperative recovery progress (for example, the patient's postoperative awakening time, time to get out of bed, discharge time, recovery status at discharge, precautions and nursing needs after discharge, full recovery time, etc.) based on the surgical video 1122 presented in the virtual patient education space 1120.

[0299] In some embodiments, after understanding the condition and various surgical plans, the doctor 271, the patient 261, and the patient's family 1133 can discuss together through the virtual patient education space 1120 to determine the final surgical plan (for example, the surgical plan used when the patient is subsequently operated on, hereinafter referred to as the target surgical plan). In some embodiments, after determining the target surgical plan, the processing device 210 or the doctor 271 can inform the patient 261 and the patient's family 1133 of the precautions during the operation through the virtual patient education space 1120. Furthermore, the doctor 271, the patient 261, and the patient's family 1133 can sign the surgical consent form 1123 in the virtual patient education space 1120. For more details, please refer to Fig.12 and 13 Get the relevant description.

[0300] Fig.12 FIG. 1 is a flow chart of an exemplary preoperative patient education process according to some embodiments of this specification. Fig.12 As shown, in some embodiments, process 1200 may include the following steps. In some embodiments, process 1200 may be performed by processing device 210 or preoperative preparation module 430. In some embodiments, process 1200 may be performed by a preoperative patient education agent. For more information about the agent, see Fig.17 and its related description.

[0301] Step 1210, generating explanation materials for explaining the surgical plan to the patient.

[0302] The explanation materials are used to explain information related to the surgical plan, such as the explanation of the surgical plan, the execution process of the surgical plan at the surgical site of the patient, the postoperative recovery process of the patient after using the surgical plan, etc. In some embodiments, the explanation materials may include text, pictures, audio or video materials. For example, the explanation materials may include pictures of the surgical site of the patient, text annotations or voice explanations related to the surgical site (for example, lesion descriptions, introductions to the surgical execution process, etc.), image sequences of the patient's postoperative recovery process (for example, multiple pictures with time information showing the progress of postoperative recovery at each stage) and their text annotations or voice explanations.

[0303] In some embodiments, the determined surgical plan may include multiple surgical plans. The processing device 210 may generate explanation materials corresponding to each surgical plan.

[0304] In some embodiments, the explanation material is generated based on a digital twin model (e.g., a three-dimensional anatomical model) of the patient's surgical site. For example, the processing device 210 can simulate the process and results of the surgery (e.g., the size of the postoperative incision, etc.) based on the surgical plan on the three-dimensional anatomical model of the patient's surgical site.

[0305] In some embodiments, the explanation material includes a surgical video (e.g., surgical video 1122). In combination with the above, the surgical video shows the process of performing surgery on the surgical site of the patient based on the surgical plan. For example, taking the surgical type in the surgical plan as an open surgery, the surgical video shows the appearance of the surgical site of the patient before the surgery, the process of the surgical site being cut open with a scalpel, the process of the lesion being removed, the process of suturing the incision, and the appearance after suturing, etc.

[0306] As described above, the explanation material may include the patient's postoperative recovery process. In some embodiments, the processing device 210 or the preoperative preparation module 430 may predict the patient's postoperative recovery process based on the patient's patient data.

[0307] The recovery process reflects the patient's vital signs and wound recovery process after surgery. In some embodiments, the recovery process may include the wound healing speed, whether the patient's vital signs are normal, whether there are complications, and the expected recovery time. For example, the recovery process may be "the patient's wound healing speed is 1 mm / day, the postoperative vital signs are normal, there are no complications, and the expected recovery time is 1 month."

[0308] In some embodiments, the processing device 210 or the preoperative preparation module 430 may process the patient data and the surgical plan using a rehabilitation prediction model to determine the rehabilitation process. The rehabilitation prediction model may be a machine learning model, the input of the rehabilitation prediction model may include the patient data and the surgical plan, and the output may include the rehabilitation process.

[0309] In some embodiments, the rehabilitation prediction model can be obtained by training. In some embodiments, the rehabilitation prediction model can be obtained by training based on multiple fifth training samples with fifth labels. For example, the processing device 210 or the preoperative preparation module 430 can input multiple fifth training samples with fifth labels into the initial rehabilitation prediction model, determine the value of the loss function through the fifth label and the result of the initial rehabilitation prediction model, and update the parameters of the initial rehabilitation prediction model based on the value iteration of the loss function. When the loss function of the initial rehabilitation prediction model meets the fifth end condition of the end of training, the model training is completed, and a trained rehabilitation prediction model is obtained. Among them, the fifth training sample may include sample patient data and sample surgical plan, and the sample patient data and sample surgical plan can be obtained based on the historical patient data and historical surgical plan of the historical patient, respectively. The fifth label can be the historical rehabilitation process of the historical patient corresponding to the fifth training sample, and the fifth label can be determined by manual annotation or automatic annotation by the system. The fifth end condition can be the convergence of the loss function (for example, the mean square error of the loss function is less than the fifth error threshold), the number of iterative training reaches the fifth number threshold, etc.

[0310] In some embodiments, the surgical video may further demonstrate the patient's recovery process after surgery.

[0311] In some embodiments, the surgical video may further display the risk conditions faced by the patient during or after the operation (ie, intraoperative risk conditions and postoperative risk conditions).

[0312] In some embodiments, the processing device 210 can use a risk assessment model to predict the patient's intraoperative risk status. The input of the risk assessment model includes the surgical plan and at least part of the patient data, and the output may include the risk assessment result (intraoperative risk status). In some embodiments, the output of the risk assessment model may also include a postoperative risk status. Postoperative risk status refers to harmful conditions that the patient may face after the operation, such as fever, surgical site infection, etc. For more information about the risk assessment model, please refer to Figure 7 or Figure 8 and its related description.

[0313] Step 1220: Display the explanation materials to the patient and the doctor simultaneously through at least one terminal device. The terminal device refers to a device worn by a user (e.g., a patient, a doctor, etc.) that can access, view, and interact with a virtual reality space (e.g., a virtual surgical scene). For example, the terminal device can be an extended reality device (e.g., XR glasses, etc.).

[0314] In some embodiments, the at least one terminal device includes a first terminal device of the patient, a second terminal device of the doctor, and a third terminal device of a family member.

[0315] In some embodiments, the at least one terminal device includes a first XR device worn by the patient and a second XR device worn by the doctor, that is, the first terminal device is the first XR device and the second terminal device is the second XR device.

[0316] In some embodiments, Fig.12 As shown, the processing device 210 or the preoperative preparation module 430 can display the explanatory materials (e.g., the surgical video 1122) to the patient and the doctor respectively through a first terminal device (e.g., the first XR device 260-2) worn by the patient (e.g., the patient 261) and a second terminal device (e.g., the second XR device 270-2) worn by the doctor (e.g., the doctor 271).

[0317] In some embodiments, at least one terminal device includes a third terminal device worn by a patient's family member. In some embodiments, the processing device 210 or the preoperative preparation module 430 can display the explanation materials to the patient's family member through the third terminal device (e.g., a third XR device) of the patient's family member. Fig.12As shown, the processing device 210 can display explanatory materials (e.g., virtual area of ​​interest 1121, surgical video 1122, surgical consent form 1123, etc.) to the patient's family members through a third XR device 1133-2 worn by the patient's family members (e.g., patient's family member 1133).

[0318] In some embodiments, the processing device 210 or the preoperative preparation module 430 can simultaneously display the explanation materials to the patient, the patient's family members, and the doctor through at least one terminal device, and the doctor can explain the explanation materials. For example, when the explanation materials are pictures, the processing device 210 or the preoperative preparation module 430 can simultaneously display the picture sequence to the patient, the patient's family members, and the doctor through the first XR device 260-2, the second XR device 270-2, and the third XR device 1133-2, and the doctor can explain the picture content currently displayed in the first XR device 260-2, the second XR device 270-2, and the third XR device 1133-2 to the patient and / or the patient's family members through the voice of the second XR device 270-2 worn.

[0319] Step 1230, determining first feedback information about the surgical plan based on second perception information collected by one or more second perception devices during the explanation of the surgical plan.

[0320] The second perception device refers to a perception device for receiving input information (e.g., text information, voice information, etc.) from a user (e.g., a doctor or a patient, etc.). For example, the second perception device may be an acoustic sensor (e.g., a microphone) to receive voice information input by the user. For example, the microphone on the XR glasses worn by the doctor can obtain the doctor's voice information and send the above voice information to the speaker on the XR glasses worn by the patient (and vice versa) to achieve communication between the doctor and the patient.

[0321] In some embodiments, at least one of the one or more second sensing devices is part of at least one terminal device. For example, the second sensing device can be a microphone on the first terminal device or the second terminal device.

[0322] In some embodiments, the second perception information includes a voice signal collected by one or more acoustic sensors (e.g., a microphone on a terminal device). For example, the second perception information may include at least one of a patient's voice signal collected by the first XR device 260-2, a doctor's voice signal collected by the second XR device 270-2, and a patient's family member's voice signal collected by the third XR device 1133-2.

[0323] The first feedback information includes the feedback of the preoperative patient education participants (e.g., doctors, patients, and patients' families) on the surgical plan. For example, the first feedback information includes at least the selection information for the surgical plan. The selection information refers to the selection result of the surgical plan. For example, taking the number of surgical plans as 5 as an example, the selection information may be "select the third surgical plan".

[0324] In some embodiments, the first feedback information also includes modification information of the surgical plan corresponding to the selected information. For example, the modification information may be "reduce the incision depth by 0.5 cm".

[0325] In some embodiments, the processing device 210 or the preoperative preparation module 430 may determine the first feedback information by analyzing and processing the second perception information.

[0326] Step 1240: confirm or update the surgical plan based on the first feedback information.

[0327] In some embodiments, the processing device 210 or the preoperative preparation module 430 may determine a target surgical plan from a plurality of surgical plans based on the first feedback information. For example, the processing device 210 or the preoperative preparation module 430 may determine the surgical plan selected by the doctor, the patient, and the patient's family as the target surgical plan. The target surgical plan will be used to perform surgery on the surgical site of the patient.

[0328] In some embodiments, the processing device 210 or the preoperative preparation module 430 may update the surgical plan of the patient based on the first feedback information. For example, if the incision depth in the surgical plan is 4 cm and the first feedback information is "reduce the incision depth by 0.5 cm", the incision depth in the surgical plan updated based on the first feedback information is 3.5 cm.

[0329] Step 1250, generate a video explaining the surgical plan.

[0330] In some embodiments, Fig.12 As shown, the processing device 210 or the preoperative preparation module 430 can generate an explanation annotation of the surgical plan based on the second perception information, and generate an explanation video of the surgical plan based on the surgical video and the explanation annotation of the surgical plan.

[0331] The explanation annotation may be a doctor's explanation of the corresponding operation of one or more frames in the surgical video. For example, the explanation annotation may include an explanation of the selection of the incision position, the incision path, the incision length, etc. For example, the processing device 210 may obtain the doctor's voice signal through the microphone on the second XR device 270-2, and use the content of the voice signal as the explanation annotation of the surgical plan. The explanation annotation may be merged into the surgical video in the form of text or audio to obtain an explanation video.

[0332] In some embodiments, the processing device 210 or the preoperative preparation module 430 can display the explanatory video to the patient, the doctor, and the patient's family through a first terminal device (e.g., the first XR device 260-2) worn by the patient (e.g., the patient 261), a second terminal device (e.g., the second XR device 270-2) worn by the doctor (e.g., the doctor 271), and a third terminal device (e.g., the third XR device 1133-2) of the patient's family.

[0333] In some embodiments, the processing device 210 or the preoperative preparation module 430 may obtain an interactive instruction regarding the explanation material from at least one terminal device; and update the explanation material displayed by the at least one terminal device based on the interactive instruction.

[0334] The interactive command refers to a control or modification command for the explanation material input by the patient through the first terminal device, or by the doctor through the second terminal device. For example, taking the explanation material as a surgical video, the interactive command may be "rewind the video by 15 seconds".

[0335] The above-mentioned update refers to adjusting the explanation materials based on the interactive instruction. For example, taking the interactive instruction in the previous example as an example, the result of the update may be: rewinding the playback progress of the surgical video displayed by the first terminal device and the second terminal device by 15 seconds.

[0336] In some embodiments of the present specification, by explaining the patient's disease progression and surgical plan to the patient and his / her family in the form of video in a virtual patient education space, doctor-patient communication can be achieved remotely, helping the patient and his / her family to quickly understand and improve the efficiency of doctor-patient communication; it can also help the patient and his / her family to fully understand their own current situation and postoperative risks, reduce the patient's fear and anxiety, and improve the success rate of the operation.

[0337] Fig.13 It is a flowchart of an exemplary agreement signing process according to some embodiments of this specification. In some embodiments, process 1300 can be executed by the processing device 210 or the preoperative preparation module 430.

[0338] In combination with the above, the processing device 210 or the preoperative preparation module 430 can display the explanatory materials to the patient, the doctor, and the patient's family members through a first terminal device worn by the patient (e.g., patient 261), a second terminal device worn by the doctor (e.g., doctor 271), and a third terminal device worn by the patient's family members (e.g., patient's family member 1133), respectively, and determine or update the patient's surgical plan based on the first feedback information.

[0339] In some embodiments, Fig.13As shown, after determining the target surgical plan from multiple surgical plans, the processing device 210 or the preoperative preparation module 430 can further obtain a first confirmation instruction and a second confirmation instruction. The first confirmation instruction is an instruction about the surgical plan input by the patient through the first terminal device (e.g., the first XR device 260-2). The second confirmation instruction is an instruction about the surgical plan input by the patient's family through the third terminal device (e.g., the third XR device 1133-2). In response to receiving the first confirmation instruction and the second confirmation instruction, the processing device 210 or the preoperative preparation module 430 can cause the first terminal device, the second terminal device (e.g., the second XR device 270-2) and the third terminal device to present the surgical consent form respectively. The processing device 210 or the preoperative preparation module 430 can obtain the signature information of the surgical consent form (e.g., the surgical consent form 1123) from the first terminal device, the second terminal device and the third terminal device respectively.

[0340] The first confirmation instruction and the second confirmation instruction are confirmation information for the patient and the patient's family to determine that the currently determined surgical plan is the final surgical plan (i.e., the target surgical plan). The input method of the confirmation instruction may include key input, gesture input, voice input, etc. For example, the patient can say "confirm the surgical plan" to the first terminal device (e.g., the first XR device 260-2), and the patient's family can say "confirm the surgical plan" to the third terminal device. The first terminal device (e.g., the first XR device 260-2) and the third terminal device 1133-2 respectively obtain the voice signal through the microphone thereon and send it to the processing device 210. In some embodiments, the input method of the first confirmation instruction may be fingerprint input, and the processing device 210 or the preoperative preparation module 430 may verify the patient's identity based on the fingerprint input by the patient, and generate the first confirmation instruction in response to the correct identity verification. The patient may input a fingerprint through a fingerprint sensor of the first terminal device or other terminal devices (e.g., a touch screen device used by the patient). In some embodiments, the method of verifying the patient's identity based on the patient's fingerprint may be through blockchain verification. For example, the processing device 210 or the preoperative preparation module 430 can generate the patient's first identity through a preset algorithm (e.g., a hash algorithm) based on the patient's fingerprint data (e.g., fingerprint data collected when the patient is admitted to the hospital) and store it in the blockchain. When patient identity authentication is required, the first terminal device or other terminal devices can obtain the current patient's fingerprint (e.g., the patient inputs the fingerprint through a touch screen device used by the patient) and initiate a verification request to the processing device 210 or the preoperative preparation module 430. The processing device 210 or the preoperative preparation module 430 generates the current patient's second identity through a preset algorithm based on the current patient's fingerprint data. The processing device 210 or the preoperative preparation module 430 retrieves the first identity of the patient stored in the blockchain (e.g., based on the current patient number, searching for the first identity of the patient with the same number stored in the blockchain), and compares it with the second identity to determine whether the first identity and the second identity are consistent. In response to determining that the first identity identifier and the second identity identifier are consistent, the identity verification is correct, otherwise the identity verification fails, and the processing device 210 or the preoperative preparation module 430 can feedback the verification failure information to the patient, the doctor and the family (for example, the verification failure information in the form of voice is sent to the first terminal device, the second terminal device and the third terminal device respectively). The identity identifier (including the first identity identifier and the second identity identifier) ​​can be in the form of a code, a picture, etc.

[0341] like Fig.13As shown, the patient, the doctor and the patient's family can view the surgical consent form and sign the surgical consent form through the first terminal device, the second terminal device and the third terminal device respectively. For example, the processing device 210 or the preoperative preparation module 430 can generate signature information (e.g., signed text or pattern) on the surgical consent form based on the signature information of the surgical consent form obtained from the first XR device 260-2, the second XR device 270-2 and the third XR device 1133-2. In some embodiments, the patient inputs the signature information based on fingerprint input, and the processing device 210 or the preoperative preparation module 430 can verify the patient's identity based on the fingerprint input by the patient, and generate the patient's signature information in response to the correct identity verification. The patient can input the fingerprint through the fingerprint sensor of the first terminal device or other terminal devices (e.g., a touch screen device used by the patient). The way to verify the patient's identity can be through blockchain verification, and the specific process of blockchain verification can be referred to the above description.

[0342] In some embodiments, the processing device 210 can obtain the voice data (i.e., second perception information) of the doctor and the patient / patient's family through a perception device (e.g., one or more second perception devices), and generate and store a preoperative patient education record (e.g., patient education content, doctor-patient interaction content, etc.) based on the voice data. By generating a preoperative patient education record, it can be convenient for the patient and / or the patient's family to study and review, as well as to archive the record.

[0343] Fig.14 is a schematic diagram of an exemplary preoperative guidance according to some embodiments of this specification. In some embodiments, process 1400 can be performed by processing device 210 or preoperative preparation module 430. In some embodiments, process 1400 can be performed by a preoperative guidance agent. For more information about the agent, see Fig.17 and its related description.

[0344] In combination with the above, preoperative guidance may include patient guidance, patient verification, preoperative care, preoperative cleaning, establishment of intravenous access, etc.

[0345] Patient access refers to the process of transporting patients from their current location to the waiting area of ​​the operating room. The waiting area refers to the area where patients wait before surgery, such as the area outside the operating room or the sterile waiting area.

[0346] Patient verification refers to checking whether the patient meets the conditions for performing the operation. The conditions for performing the operation include: whether the patient is the target patient who needs to perform the current operation (for example, verify whether the identity information of the patient (current patient) (for example, number, name, gender, age, fingerprint, etc.) matches the target patient who needs to perform the current operation), whether the operation that the patient needs to undergo is the operation that is currently being scheduled (or to be performed), whether the patient's current physical condition (for example, vital signs such as heart rate, blood pressure, and respiratory rate) meets the requirements of the operation, whether the patient fasts and abstains from water according to the doctor's orders, whether the patient does not carry items that affect the safety of the operation (for example, metal products, electronic products, etc.), and whether the patient's mental state is normal (for example, the patient does not feel anxious, the tension is low, etc.), etc. Among them, the mental state of the verification object can be determined based on the recognition of the expression of the verification object. It can be understood that in response to the verification object not meeting any one or more of the conditions for performing the operation, the operation on the patient is suspended or postponed.

[0347] In some embodiments, the processing device 210 can collect the patient's sensory information (e.g., biological information) through one or more third sensory devices in the waiting area; and perform patient identity verification based on the sensory information. Fig.14 As shown, after the patient is transported to the waiting area 1410, the processing device 210 can collect the biometric information of the patient 261 through one or more sensing devices 1411 (e.g., image sensor, microphone, fingerprint sensor, etc.) in the waiting area 1410, and verify the identity of the patient 261 based on the biometric information. The biometric information may include the patient's facial information, voice information, fingerprint information, etc.

[0348] In some embodiments, the processing device 210 or the preoperative preparation module 430 can verify the patient's identity by comparing the collected patient's biological information with the patient's pre-stored biological information. In some embodiments, the processing device 210 can use the nurse agent to verify the patient's identity. For example, the nurse agent can verify the collected biological information, or verify the patient's identity by voice interaction with the patient (for example, asking the patient's age, name, gender, etc.).

[0349] Preoperative care includes preoperative comfort and preoperative education. Preoperative comfort refers to the preoperative preparation work to reduce the patient's negative emotions (such as anxiety, tension, fear, etc.) before the operation through verbal communication, video, music, etc. Preoperative education refers to the preoperative preparation work to help patients understand the surgical process.

[0350] Preoperative cleaning refers to the preoperative preparation work of cleaning the patient's body, shaving (e.g., hair, body hair, etc.), changing surgical clothes, etc. to reduce the risk of infection.

[0351] Intravenous access establishment refers to establishing an intravenous channel for drug injection in the patient's body to ensure that drugs can be effectively injected into the patient during surgery. For example, intravenous access establishment can be inserting a retention needle into the patient's hand vein.

[0352] In some embodiments, the processing device 210 can determine a planned route from the patient's current location to the waiting area, and control the smart chair to transport the patient to the waiting area along the planned route. Fig.14 As shown, before performing surgery on a patient based on the surgical plan, the processing device 210 can determine a planned route from the current location of the patient 261 (e.g., the ward 1403) to the waiting area 1410. The processing device 210 can control the smart chair 240-5 to transport the patient 261 from the ward 1403 to the waiting area 1410 along the planned route.

[0353] In some embodiments, the processing device 210 may determine a planned route from the current location to the waiting area based on a hospital map.

[0354] The smart chair is a wheelchair with an automatic driving function, which can transport a load (eg, a patient) to a designated destination (eg, a waiting area) along a planned route (eg, a planned route).

[0355] In some embodiments, the processing device 210 may be configured with a corresponding nurse agent. The nurse agent refers to an agent that replaces a nurse to complete part of her work, and it may present a virtual nurse character. In some embodiments, the first terminal device (e.g., the first XR device 260-2) may present a virtual nurse character that provides preoperative care, and the patient 261 may see the virtual nurse character in a virtual space by wearing the first terminal device (e.g., the first XR device 260-2). The virtual nurse character may interact with the patient by voice, for example, answering the patient's questions, asking for and confirming the patient's identity information, and providing the patient with preoperative care materials. For more information about agents, see Fig.17 and its related description.

[0356] In some embodiments, the processing device 210 can use the nurse agent to control the smart chair to transport the patient from the current location to the waiting area.

[0357] In some embodiments, after the patient is transported to the waiting area, the processing device 210 or the preoperative preparation module 430 may perform patient verification. For more information about patient verification, please refer to the above description.

[0358] In some embodiments, the processing device 210 or the preoperative preparation module 430 can determine the patient's preoperative care materials based on the patient data and the surgical plan. During the process of transporting the patient to the waiting area, the processing device 210 or the preoperative preparation module 430 can use the patient's first terminal device to perform preoperative care on the patient based on the preoperative care materials.

[0359] Preoperative care materials may include videos, music, images, texts, etc. related to surgical explanations and / or emotional relaxation. For example, preoperative care materials may be light music that can soothe the patient's nervousness, videos or audios recorded by family members, or videos explaining the surgery that the patient is about to undergo.

[0360] In some embodiments, the processing device 210 or the preoperative preparation module 430 may process the patient data and the surgical plan using the material determination model to determine the preoperative care materials. The material determination model may be a trained machine learning model, the input of the material determination model may include the patient data and the surgical plan, and the output may include the preoperative care materials. In some embodiments, the material determination model may be integrated in the processing device 210. After the processing device 210 determines the preoperative care materials using the material determination model, it may be displayed to the patient through the first terminal device worn by the patient.

[0361] In some embodiments, the material determination model can be obtained by training. In some embodiments, the material determination model can be obtained by training based on multiple sixth training samples with sixth labels. For example, the processing device 210 or the preoperative preparation module 430 can input multiple sixth training samples with sixth labels into the initial material determination model, determine the value of the loss function by the sixth label and the result of the initial material determination model, and iteratively update the parameters of the initial material determination model based on the value of the loss function. When the loss function of the initial material determination model meets the sixth end condition of the end of training, the model training is completed, and the trained material determination model is obtained. Among them, the sixth training sample can include sample patient data and sample surgical plan, and the sample patient data and sample surgical plan can be obtained based on the historical patient data and historical surgical plan of the historical patient, respectively. The sixth label can be the historical preoperative care material corresponding to the sixth training sample, and the sixth label can be determined by manual annotation or automatic annotation by the system. The sixth end condition can be the convergence of the loss function (for example, the mean square error of the loss function is less than the sixth error threshold), the number of iterative training reaches the sixth number threshold, etc.

[0362] In some embodiments, the processing device 210 can use the nurse agent to provide preoperative care to the patient. Fig.14As shown, the processing device 210 can present a virtual nurse character 1423 in the first terminal device (e.g., the first XR device 260-2) worn by the patient 261, and the virtual nurse character 1423 explains the preoperative care materials to the patient 261. In some embodiments, the virtual nurse character 1423 can interact with the patient 261 by voice to relieve the patient's negative emotions or answer the patient's questions through communication. In some embodiments, the processing device 210 can determine whether the patient's emotions need to be relieved by collecting the patient's expression, physical state, tone of voice, etc.

[0363] In some embodiments, the processing device 210 can use the nurse agent to guide the nurse to perform preoperative cleaning and / or intravenous access to the patient. For example, the nurse agent can be presented by a three-dimensional nurse image, and the three-dimensional nurse image can voice-guide the nurse to perform intravenous access for drug delivery, body cleaning and shaving on the patient, and / or voice-guide the patient to change surgical clothes, body cleaning, etc.

[0364] In some embodiments, in the process of sending the patient to the waiting area, the processing device 210 or the preoperative preparation module 430 can obtain the third perception information related to the portion of the planned route from the current position of the smart chair to the waiting area (i.e., the portion of the planned route where the smart chair has not yet traveled) through one or more fourth perception devices in the hospital. Based on the third perception information, the processing device 210 or the preoperative preparation module 430 can determine the potential risk of the portion of the planned route that has not been traveled, and update the portion of the planned route that has not been traveled based on the potential risk.

[0365] The fourth sensing device may include an image sensor (e.g., an infrared surveillance camera 1413), a laser radar, etc. One or more fourth sensing devices may be installed in a smart chair, a hospital ceiling, or a hospital wall, etc.

[0366] The third perception information may reflect the environmental conditions of the area through which the planned route passes (for example, whether there are obstacles, whether the ground is flat, whether there are people or other moving objects passing by, etc.). For example, taking the fourth perception device as an image sensor, the third perception information may be image data of the area through which at least part of the planned route passes (for example, outdoor walkways, corridors, building platforms, etc. in a hospital).

[0367] Potential risks refer to the problems that the smart chair may encounter when passing a certain location or part of the planned route, such as collision, rollover, etc.

[0368] In some embodiments, the processing device 210 or the preoperative preparation module 430 may process the third perception information using a risk identification model to determine potential risks. The risk identification model may be a trained machine learning model, the input of the risk identification model may include the third perception information, and the output may include whether there is a potential risk (e.g., 1 indicates that there is a risk, and 0 indicates that there is no risk) and / or the type of risk (e.g., a collision may occur). In some embodiments, the risk identification model may be integrated in the processing device 210.

[0369] In some embodiments, the risk identification model can be obtained by training. In some embodiments, the risk identification model can be obtained by training based on multiple seventh training samples with the seventh label. For example, the processing device 210 or the preoperative preparation module 430 can input multiple seventh training samples with the seventh label into the initial risk identification model, determine the value of the loss function by the seventh label and the result of the initial risk identification model, and iteratively update the parameters of the initial risk identification model based on the value of the loss function. When the loss function of the initial risk identification model meets the seventh end condition of the end of training, the model training is completed, and a trained risk identification model is obtained. Among them, the seventh training sample can include sample sensor data, and the sample sensor data can be determined based on the historical third perception information obtained when the historical patient is transported to the waiting area. The seventh label can be the historical potential risk corresponding to the seventh training sample, and the seventh label can be determined by manual labeling or automatic labeling by the system. The seventh end condition can be the convergence of the loss function (for example, the mean square error of the loss function is less than the seventh error threshold), the number of iterative training reaches the seventh number threshold, etc.

[0370] Updating the portion of the planned route means that when there is a potential risk, the processing device 210 or the preoperative preparation module 430 can adjust the position or line shape of the portion of the planned route so that the planned route does not pass through the location of the potential risk or avoids the risk. For example, taking the potential risk of road flooding as an example, the update of the planned route can be: modifying the portion of the planned route that passes through the flooded area so that it bypasses the flooded area. For another example, when someone passes by in front, the update of the planned route can be: modifying the portion of the planned route that coincides with the location of the pedestrian so that it avoids the pedestrian.

[0371] In some embodiments of the present specification, through the above-mentioned preoperative guidance process, a more humane, transparent and efficient preoperative preparation process can be provided and preoperative preparation matters can be dynamically adjusted based on patient feedback to improve the efficiency of preoperative preparation; through the above-mentioned methods of transporting patients and verifying identities, it helps to avoid human errors and enhance the safety of the entire surgical process; by using a virtual nurse image to assist in completing many preoperative preparation tasks, labor costs can be saved.

[0372] Fig.15is a schematic diagram of an exemplary surgical procedure performed according to some embodiments of the present specification.

[0373] After sending the patient to the waiting area and completing the corresponding preparations (e.g., preoperative patient education, preoperative guidance, etc.), the medical team can perform surgery on the patient based on the target surgical plan.

[0374] In some embodiments, the medical team may include local surgery participants 1510, and / or remote surgery participants 1531 (e.g., remote medical experts, experts from other departments, etc.). Fig.15 As shown, surgical participants 1510 may include a surgeon 1511 , an anesthesiologist 1513 , and a surgical nurse 1515 .

[0375] The surgeon in charge is responsible for performing surgery on the patient according to the surgical plan, making immediate decisions and adjustments based on the actual progress of the surgery, communicating with the surgical nurse about the needs for surgical instruments and supplies, and communicating with the anesthesiologist about the patient's physiological condition.

[0376] The surgical nurse is responsible for the preparation and management of surgical instruments and consumables, as well as delivering instruments according to the doctor's needs during the operation, recording key events during the operation, etc. In some embodiments, before the operation is performed, the processing device 210 or the surgical execution module 440 can control the intelligent robot nurse (for example, the intelligent robot nurse 240-6) to prepare surgical tools in the operating room based on the surgical plan; and based on the first perception information collected by one or more first perception devices in the operating room during the operation, identify the instructions for the target surgical tools issued by the surgical participant (for example, the local surgical participant 1510); based on the instructions, control the intelligent robot nurse to pass the target surgical tools to the surgical participant. For more information about this embodiment, see Fig.17 and its related description.

[0377] The anesthesiologist is responsible for injecting anesthetics into the patient. In some embodiments, the anesthesiologist 1513 can use the display screen of the operating room (e.g., display screen 1523) to retrieve the patient's digital twin in the UHR, and confirm the effective anesthetic dosage and assess the patient's anesthesia depth and predict awakening based on the digital twin. In some embodiments, patient anesthesia can be achieved through an anesthesia machine. For example, the processing device 210 can confirm the effective anesthetic dosage and assess the patient's anesthesia depth and predict awakening based on the patient's digital twin, and control the intelligent anesthesia machine to perform general anesthesia or local anesthesia on the patient based on the evaluation results.

[0378] The remote surgery participant 1531 can observe the surgical process in real time (e.g., observe the lesion removal operation, hemostasis operation, response operation to risk conditions, etc. of the surgeon) through the virtual surgery space 1530 presented by the extended reality device (e.g., the fourth terminal device 1531-2 worn by the surgeon), and / or provide remote surgery guidance (e.g., the treatment method for emergency situations, action guidance for difficult operations, etc.). For example, the remote surgery participant 1531 can make annotations and voice interactions in the virtual surgery space through the fourth terminal device 1531-2 to guide the local surgery participant 1510 (e.g., the surgeon 1511) to operate on the patient.

[0379] In some embodiments, the medical team can perform surgery using the smart surgical device. Fig.15 As shown, the intelligent surgical device may include an intelligent surgical terminal 240-3 and an intelligent robot nurse 240-6. For more information about the intelligent surgical terminal 240-3, see Figure 2 In some embodiments, the medical equipment used in performing the surgery may also include vital signs monitoring equipment (e.g., ventilator, pulse oximeter, ECG and blood pressure monitor, etc.), imaging equipment (e.g., ultrasound equipment, endoscope, subtraction angiography equipment, etc.) and other auxiliary equipment (e.g., ultrasonic scalpel, interventional robot, various surgical knives, syringes, hemostatic forceps, etc.).

[0380] In some embodiments, information interaction between the local surgical participant 1510 and the remote surgical participant 1531 can be achieved through the interactive device 1520 (e.g., display screen 1523 and / or terminal device 1525). For example, the processing device 210 or the surgical execution module 440 can obtain the surgical scene from the doctor's perspective through the terminal device (e.g., the second terminal device) worn by the surgeon 1511, and send it to the display screen 1523 and / or the fourth terminal device 1531-2 for display. For another example, the surgical participant can display three-dimensional images of different sections in the interactive device 1520 through gesture interaction to analyze the anatomical structure of the patient's organs and lesions, and / or drag the real-time ultrasound image to any angle for viewing through gestures.

[0381] In some embodiments, the processing device 210 or the surgery execution module 440 can switch the display content of the display screen 1523 and / or the terminal device 1525 in real time based on the surgical process. For example, the processing device 210 can display the department's operating room view, the current operating room real-time view (including personnel, events, resources and operating room status monitoring), the patient's key surgical information (for example, the target surgical plan, anesthesia plan, expected risks and treatment methods, etc.), the patient's intraoperative imaging data (for example, the surgical instrument arrival position, etc.), the patient's real-time physiological data (for example, heart rate, heart rhythm, blood pressure, blood oxygen saturation, respiratory status), etc. on the display screen 1523.

[0382] In some embodiments, Fig.15 As shown, the processing device 210 or the surgical execution module 440 can monitor the environmental data and personnel behavior through the sensing device 250. Fig.15 As shown, the sensing device 250 may include an image sensor 250-1 (e.g., an infrared camera), a sound sensor 250-2 (e.g., a microphone), a temperature and humidity sensor 250-3 (including a temperature sensor and a humidity sensor), and a behavior tracking device 250-4 (e.g., a gesture recognition device or a motion recognition device, etc.). Among them, the image sensor 250-1 is used to obtain images of the surgical participant 1510 and / or the patient; the sound sensor 250-2 is used to obtain voice information of the surgical participant 1510 and / or the patient; the temperature and humidity sensor 250-3 is used to monitor the temperature and humidity in the operating room; and the behavior tracking device 250-4 is used to obtain behavior information of the surgical participant 1510 (including position, gesture, action type, and action amplitude, etc.). In some embodiments, the behavior tracking device 250-4 may be a wearable device, such as a wearable camera, wearable gloves, wearable clothing, etc.

[0383] In some embodiments, the sensing device 250 may also include a device for monitoring the operating room environment, such as a pressure difference sensor, an air quality sensor, and the like.

[0384] In some embodiments, the processing device 210 or the surgical execution module 440 can send the monitoring information collected by the sensing device to the display screen 1523 for display.

[0385] In some embodiments, the processing device 210 can generate a surgical record (eg, an initial surgical record) and a medical order (eg, an initial medical order) based on the first perception information collected during the surgical procedure.

[0386] For more details on surgical performance, see Fig.16 and Fig.17 Described in.

[0387] Fig.1616 is a flowchart of an exemplary surgical execution according to some embodiments of the present specification. In some embodiments, the surgical execution process 1600 can be executed by the processing device 210 or the surgical execution module 440. In some embodiments, steps 1611, 1613, and 1615 in the process 1600 can be executed by a preoperative preparation agent. For more information about the agent, see Fig.17 and its related description.

[0388] In combination with the above, the surgical execution process may include preoperative preparation matters, intraoperative matters and postoperative matters.

[0389] like Fig.16 As shown, the process before the operation (i.e., preoperative preparations) may include step 1611, step 1613, and step 1615.

[0390] Step 1611, start the operating room.

[0391] Starting the operating room may include opening the door of the operating room, starting the surgical equipment in the operating room (for example, surgical lights, vital signs monitoring instruments, intelligent robot nurses, intelligent surgical terminals, imaging equipment, etc.), sensing equipment, adjusting parameters in the operating room, verifying the status of surgical equipment, etc.

[0392] Indoor parameter adjustment includes adjusting the temperature, humidity, air quality, brightness, etc. in the operating room.

[0393] Surgical equipment status verification refers to the process by which the surgical equipment verifies and reports whether it can operate normally based on a self-test procedure.

[0394] In some embodiments, the processing device 210 can control the intelligent robot nurse 240-6 to start the operating room, or guide the nurse to start the operating room. For example, the processing device 210 can control the intelligent robot nurse 240-6 to automatically start the equipment in the operating room at the scheduled time of the operating room, and adjust the temperature, humidity and air quality in the room.

[0395] Step 1613, prepare surgical tools.

[0396] Surgical tools may include surgical instruments and surgical consumables. For example, surgical instruments may be non-consumable items such as scalpels, surgical forceps, and suture needles of various specifications, and surgical consumables may be consumable items such as syringes, anesthetics, blood bags, gauze, and drugs. In some embodiments, before the operation, the processing device 210 may control the intelligent robot nurse to prepare surgical tools in the operating room based on the surgical plan. For example, the processing device 210 may control the intelligent robot nurse 240-6 to prepare the scalpels, surgical forceps, hemostats, suture needles, blood bags, etc. required for the operation based on the surgical plan, and sterilize the surgical instruments. In some embodiments, the processing device 210 may control the intelligent robot nurse to sterilize the operating table and arrange the operating table (for example, arrange the positions of various surgical tools on the operating table).

[0397] Step 1615, identify and / or anesthetize the patient.

[0398] Confirming the patient means confirming the identity of the patient, and anesthetizing the patient means injecting anesthetic into the patient.

[0399] In some embodiments, after the patient enters the operating room, the processing device or the surgical participant can authenticate the patient, and confirm the surgical site and anesthesia plan. For example, after the anesthesiologist 1513 and the surgical nurse 1515 bring the patient into the operating room, they can verify the patient's identity and confirm the surgical site and anesthesia plan with the patient. For another example, the processing device 210 can present a virtual nurse character through the first terminal device worn by the patient, authenticate the patient through the virtual nurse character, and confirm the surgical site and anesthesia plan.

[0400] The anesthesia plan may include anesthesia type and anesthesia mode. Anesthesia type may include local anesthesia or general anesthesia. Anesthesia mode may include manual anesthesia (manually performed by an anesthesiologist) and intelligent anesthesia (performed by an intelligent anesthesia device).

[0401] Step 1620, perform the surgery.

[0402] In some embodiments, Fig.16 As shown, matters during surgical execution (i.e., intraoperative matters) may include remote collaboration, tool delivery, image interaction, intraoperative planning navigation, and real-time early warning.

[0403] Remote collaboration means remote participation and / or guidance of surgery. For example, the processing device 210 can display the surgical operation process from the perspective of the local surgical participant (e.g., the surgeon 1511) to the remote surgical participant 1531 (e.g., the remote medical expert) through the fourth terminal device 1531-2, and send prompt information to the local surgical participant through the fourth terminal device 1531-2 via the second terminal device (e.g., the second XR device 270-2).

[0404] Tool transfer refers to the transfer of surgical tools (e.g., surgical instruments, surgical consumables, etc.) to the surgeon (e.g., surgeon 1511) during the operation. For example, the processing device 210 can control the intelligent robot nurse 240-6 to transfer surgical tools to the surgeon 1511. In some embodiments, the processing device 210 can identify instructions for target surgical tools issued by surgical participants (e.g., local surgical participants 1510) based on first perception information collected by one or more first perception devices in the operating room during the operation. Based on the instructions, the processing device 210 can control the intelligent robot nurse (e.g., intelligent robot nurse 240-6) to transfer the target surgical tools to the surgical participant. For more details, see Fig.17 Described in.

[0405] Image interaction refers to displaying the patient's digital body model (e.g., a three-dimensional anatomical model of the surgical site), the patient's electronic health record, the surgical plan of the current surgery, the real-time image of the patient's surgical site, etc. to the surgical participants (e.g., local surgical participants 1510, remote surgical participants 1531) and / or patients through the interactive devices in the operating room (e.g., the display screen 1523 in the operating room, the terminal device 1525 worn by the doctor). The real-time image of the patient's surgical site can be acquired in real time through the image sensor on the terminal device 1525 worn by the doctor and sent to the processing device 210, and the processing device 210 can process the above real-time image (e.g., amplify, reduce noise, calibrate, etc.), and control the interactive devices in the operating room to display the processed real-time image.

[0406] Intraoperative planning and navigation refers to the fusion of patient lesion images (e.g., CT scan images of the lesion) with the digital body model of the patient during surgery, or the positioning tracking of superimposed surgical tools, to guide surgical participants to perform the surgery (e.g., accurately placing an implant (e.g., a heart stent) in a predetermined position).

[0407] Real-time warnings can include behavioral warnings for surgical participants, warnings for patients' vital signs, warnings for equipment operating status, etc.

[0408] Behavior warning refers to monitoring and warning the operation behavior of the surgical participants during the operation. In some embodiments, the processing device 210 can learn the pattern of a specific surgical procedure, perceive the operation of the surgeon in real time through the sensing device during the operation, and provide an operation risk warning in the form of voice broadcast based on the operation data. For example, the processing device 210 can monitor the behavior of the local surgical participant 1510 during the operation in real time through the sensing device 250 (for example, the image sensor 250-1, the sound sensor 250-2, the behavior tracking device 250-4, etc.). When the local surgical participant 1510 performs abnormal behavior (for example, the surgical nurse 1515 is in the wrong station or stays in the same position for too long) or the operation has surgical risks (for example, the surgical operation of the surgeon 1511 may cause heavy bleeding or damage to other organs), a warning (for example, a voice warning message, etc.) is issued to the surgical participant.

[0409] The patient's vital sign warning can be a warning when the patient's vital signs (for example, the patient's electrocardiogram, blood pressure, etc.) are lower than a preset value.

[0410] The early warning of the equipment operation status refers to the early warning when the operation status of the surgical equipment (for example, surgical lights, vital signs monitoring instruments, intelligent surgical terminals 240-3, intelligent mechanical nurses 240-6, etc.) is abnormal. The equipment operation status can be reflected based on the equipment operation parameters, which include equipment operation power, equipment operation temperature, etc. When the equipment operation parameters are abnormal (for example, the equipment operation temperature is higher than the preset temperature threshold, etc.), the processing device 210 can send equipment early warning information (for example, equipment failure prompt information, etc.) to the surgical participant 1510.

[0411] like Fig.16 As shown, the process after the operation is performed (i.e., postoperative matters) may include step 1631, step 1633 and step 1635.

[0412] Step 1631, patient transfer.

[0413] Patient transfer refers to the process of transferring a patient (wake up or in coma) from an operating room to a recovery room (e.g., a ward or intensive care unit) after surgery. In some embodiments, patient transfer can be performed by a caregiver (e.g., a nurse). In some embodiments, patient transfer can be performed by a caregiver with the assistance of an intelligent robotic nurse.

[0414] Step 1633, tidy up the operating room.

[0415] Operating room tidying refers to the process of cleaning or clearing surgical equipment and surgical tools. For example, operating room tidying may include cleaning discarded surgical tools, counting the number of remaining surgical tools, sterilizing surgical equipment and surgical tools, etc. In some embodiments, the processing device 210 may control the intelligent robot nurse to tidy the operating room.

[0416] Step 1635, generate a surgical record.

[0417] Surgical records may include surgery-related information, patient-related records, participant-related records, etc.

[0418] In some embodiments, the processing device 210 or the surgery execution module 440 can generate an initial surgery record based on data collected during the surgery (for example, first perception information collected by one or more first perception devices in the operating room). In some embodiments, the processing device 210 or the surgery execution module 440 can generate an initial surgery record based on the initial surgery record and feedback information about the initial surgery record input by the doctor. In some embodiments, the processing device 210 or the surgery execution module 440 can generate an initial medical order based on data collected during the surgery. In some embodiments, the processing device 210 or the surgery execution module 440 can generate a medical order based on the initial medical order and feedback information about the initial medical order input by the doctor. For more details, see Fig.17 Described in.

[0419] Fig.17 FIG. 1 is a flowchart of an exemplary operation performed according to other embodiments of this specification. Fig.17 As shown, in some embodiments, the process 1700 may include the following steps. In some embodiments, the process 1700 may be executed by the processing device 210 or the surgery execution module 440.

[0420] Step 1710, based on the surgical plan, control the intelligent robot nurse to prepare surgical tools in the operating room.

[0421] In some embodiments, the processing device 210 can determine the type and quantity of surgical tools required based on the surgical plan, and control the intelligent robot nurse to prepare the surgical tools according to the type and quantity of surgical tools, and place them at a predetermined position on the operating table.

[0422] Step 1721, obtaining first sensing information collected by one or more first sensing devices in the operating room during the patient's surgery.

[0423] The first sensing device refers to a sensing device installed in the operating room and / or worn on the surgical participant. For example, the first sensing device may be a sound sensor (e.g., a microphone on the second XR device), a gesture recognition device, an action recognition device, a force sensing device, an image sensor, etc. installed in the operating room or configured on the doctor's second terminal device.

[0424] The first perception information may include voice data, motion data, and / or image data. For example, the first perception information may be a voice uttered by a surgical participant. For another example, the first perception information may be an image of a surgical site. In some embodiments, the first perception information includes an image of a surgical tool acquired by an image sensor.

[0425] like Fig.17 As shown, in some embodiments, the processing device 210 may perform event of interest (EOI) detection on the first perception information (step 1722). In response to detecting the occurrence of EOI, one or more preset operations corresponding to EOI are performed (step 1723). Specifically, as follows:

[0426] Step 1722, perform event of interest (EOI) detection on the first perception information.

[0427] EOI refers to events that may occur during surgery that may affect the progress of the surgery or pose a threat to the patient, such as risk conditions for the patient during surgery, incorrect operations by surgical participants, insufficient surgical tools or consumables, etc.

[0428] In some embodiments, EOI includes at least one of the following: an instruction issued by a surgical participant (for example, the surgeon issues an instruction for a blood transfusion to the patient); a surgical risk is detected (for example, the patient experiences cardiac arrest or massive bleeding, etc.); the patient's physiological state is abnormal (for example, the patient's blood pressure is too low); the number of surgical tools is less than a preset value (for example, the number of scalpels is less than a scalpel number threshold); the operation is completed (i.e., the surgical procedure is over), etc.

[0429] EOI detection refers to identifying whether an event of interest occurs during surgery based on the first perception information. For example, the processing device 210 can perform voice recognition on the voice information in the first perception information to determine whether the voice information is an instruction issued by a participant in the surgery (for example, an instruction issued by a participant in the surgery regarding a target surgical tool or a target surgical consumable). For another example, the processing device 210 can perform image recognition on the surgical tool image in the first perception information to determine the quantity of each surgical tool, and then determine whether the quantity of a certain surgical tool is less than the corresponding preset value.

[0430] EOI detection may include voice detection or gesture detection, image detection, and physical sign detection, etc.

[0431] Voice or gesture detection refers to the recognition of voice information or gesture information expressed by a surgical participant (e.g., the surgeon) to determine whether the voice information or gesture information is a command issued by the surgical participant, or whether the event corresponding to the voice information or gesture information is an event of interest. For example, if the voice content is "I need hemostatic forceps", the processing device 210 or the surgical execution agent can recognize the voice information as a command voice for surgical tools (combined with the above, the surgical participant issuing a command is an event of interest). Based on this, the processing device 210 or the surgical execution agent detects an event of interest and determines that the event of interest is a command issued by the surgical participant. For another example, the doctor makes an "OK" gesture, and the processing device 210 or the surgical execution agent can recognize the event corresponding to the gesture information as "surgery completed" (combined with the above, the operation is completed as an event of interest). Based on this, the processing device 210 or the surgical execution agent detects an event of interest and determines that the event of interest is the completion of the operation. For more information about the surgical execution agent, please refer to the following description.

[0432] Image detection refers to image recognition of images (e.g., images of the patient's surgical site, images of surgical tools, etc.) collected by the first sensing device to determine whether an event of interest has occurred. For example, the processing device 210 or the surgical execution agent can perform image recognition on the surgical tool images collected by the first sensing device, and then determine the number of each surgical tool and compare the number with the number threshold of each surgical tool. In response to determining that the number of at least one surgical tool is less than its number threshold, the processing device 210 or the surgical execution agent believes that an event of interest has been detected, and determines that the event of interest is that the number of surgical tools is less than a preset value. For another example, the processing device 210 or the surgical execution agent can perform image recognition on the image of the patient's surgical site collected by the first sensing device, and then determine the amount of bleeding at the patient's surgical site. In response to determining that the amount of bleeding is greater than the bleeding threshold, the processing device 210 or the surgical execution agent believes that an event of interest has been detected, and determines that the event of interest is a detected surgical risk, and the surgical risk is a massive bleeding of the patient.

[0433] Sign detection refers to monitoring the patient's vital signs (e.g., heart rate, blood pressure, respiratory rate, etc.) during surgery to determine whether an event of interest (EOI) has occurred. For example, the processing device 210 or the surgery execution agent can monitor the patient's heart rate during surgery in real time and compare it with the normal heart rate range in real time. In response to the patient's heart rate not being within the normal heart rate range, the processing device 210 or the surgery execution agent considers that an event of interest has been detected, and determines that the event of interest is an abnormal physiological state of the patient, specifically an abnormal heart rate of the patient.

[0434] In some embodiments, the computing device to which the processing device 210 belongs is configured with an agent, and the process 1700 can be executed by the agent (hereinafter referred to as the surgery execution agent). For example, the surgery execution agent can perform event of interest (EOI) detection on the first perception information.

[0435] An agent is a program (e.g., a machine learning model with self-evolving capabilities) that can make decisions or provide services based on environmental information (e.g., first-person perception information), user input (e.g., voice information input by a doctor), or experience data. These programs can be used to autonomously collect information and make decisions regularly, at a scheduled time, or in real time when prompted by the user.

[0436] The intelligent agent can be equipped with supporting devices to realize the functions of the intelligent agent. The supporting devices may include sensors (for collecting information), effectors (for responding to the information collected by the sensors) and actuators (for outputting information). For example, taking the surgical execution intelligent agent as an example, the surgical execution intelligent agent can use sensors (for example, sound sensors and image sensors in the first sensor device, etc.) to collect environmental information or user input. The surgical execution intelligent agent controls the effector (for example, a robotic arm, etc.) to complete the operation corresponding to the instruction based on the doctor's instruction (for example, delivering the target surgical tool to the doctor based on the doctor's instruction that the target surgical tool is needed). The surgical execution intelligent agent outputs information (for example, outputs reminder information of incorrect operation to the doctor) through the actuator (for example, a terminal device, a display screen, etc.).

[0437] In some embodiments, the computing device to which the surgery execution agent belongs can be integrated into the operating table, the intelligent surgery terminal or the intelligent robot nurse. That is, the surgery execution agent can be configured in the operating table, the intelligent surgery terminal or the intelligent robot nurse to make decisions or provide services during the surgery execution process.

[0438] In some embodiments, a hospital agent may be deployed on the hospital's main server. A hospital agent refers to an agent that makes decisions, guides, or provides services for various in-hospital medical behaviors such as patient visits, hospitalization, surgery (including preoperative preparation, preoperative patient education, surgery execution, postoperative review, surgery simulation), rehabilitation, and physical examinations. For example, a hospital agent can provide services such as consultation guidance, answering consultation questions, and generating consultation records for patients, while collecting patients' consultation feedback and consultation evaluations.

[0439] In some embodiments, the hospital agent may be comprised of multiple sub-agents.

[0440] Based on the application scenarios, multiple sub-agents may include medical consultation agents, hospitalization agents, surgical agents, rehabilitation agents, physical examination agents, etc., which are used to make decisions, guide or provide services for various in-hospital medical behaviors such as patient consultation, hospitalization, surgery (including preoperative preparation, preoperative patient education, surgical execution, postoperative review, surgical simulation), rehabilitation, physical examination, etc. Among them, the surgical agent may include a preoperative preparation agent, a preoperative patient education agent, a surgical execution agent, a postoperative review agent, and a surgical simulation agent. For example, as mentioned above, the surgical execution agent can be used to perform event of interest (EOI) detection on the first perception information during the execution of the operation. For another example, the preoperative patient education agent can be used to provide preoperative patient education to the patient before the operation. For more information about preoperative patient education, please refer to Figures 11 to 13 Related description.

[0441] Based on the role division, multiple sub-agents may include nurse agent, doctor agent, etc. Nurse agent refers to an agent that replaces the nurse to complete part of the work (e.g., preoperative patient education, postoperative care, etc.). Doctor agent refers to an agent that replaces the doctor to complete part of the work (e.g., preoperative patient education, postoperative care, etc.). For example, the doctor agent can explain the surgical plan and rehabilitation process to the patient based on the explanation materials. For more information about the nurse agent, see Fig.14 and its related description.

[0442] In some embodiments, EOI detection is performed based on EOI detection rules learned by the agent from historical records.

[0443] Historical records refer to the recorded data of historical surgical events and collected historical first perception information during historical surgical processes. The historical first perception information may include historical voice data, historical action data (e.g., cutting action, pressing action, injection action, suturing action, etc.), historical image data, and the corresponding time points of each data during historical surgical processes. Historical surgical events may include doctor operations (e.g., cutting the surgical site, stopping bleeding at the surgical site, suturing the surgical site, etc., and the corresponding time points of each operation), surgical risk conditions and occurrence time points, and patient vital signs records at multiple time points during the operation (e.g., recorded every 5 seconds). Doctor operations and surgical risk conditions in historical surgical events may be manually recorded by surgical participants, or automatically recorded after being collected or determined by the processing device 210.

[0444] The EOI detection rule refers to the constraint conditions that need to be met when determining an event of interest based on the first perception information. For example, the patient's bleeding volume satisfies the constraint conditions (for example, the patient's bleeding volume is greater than the bleeding volume threshold), the processing device 210 or the surgical execution agent can determine that the patient has heavy bleeding (such as heavy bleeding is an event of interest), otherwise, the processing device 210 or the surgical execution agent does not believe that the patient has heavy bleeding (the processing device 210 or the surgical execution agent does not believe that an event of interest has occurred). For another example, the patient's blood pressure value satisfies the constraint conditions (the patient's blood pressure value is not within the normal blood pressure range and the difference with the boundary value of the normal blood pressure range is greater than the difference threshold), the processing device 210 or the surgical execution agent can determine that the patient's blood pressure is abnormal (such as abnormal blood pressure is an event of interest), otherwise, the processing device 210 or the surgical execution agent does not believe that the patient's blood pressure is abnormal (the processing device 210 or the surgical execution agent does not believe that an event of interest has occurred).

[0445] In some embodiments, the processing device 210 or the surgery execution agent can determine the EOI detection rule based on the historical first perception information and historical surgery events in the historical records. For example, in the historical records, among the patients with severe bleeding, the bleeding volume of the patient with the smallest bleeding volume is 522 ml; among the patients without severe bleeding, the bleeding volume of the patient with the largest bleeding volume is 354 ml, then the processing device 210 or the surgery execution agent can set the bleeding volume threshold to a value between 354 ml and 522 ml, for example, the bleeding volume threshold is set to the average of 326 ml and 522 ml (424 ml). For another example, in the historical records, among the patients with high blood pressure, the minimum excess amount exceeding the upper limit of the normal blood pressure range is 7.1 mmHg; among the normal patients (patients who are not recorded as having high blood pressure), the maximum excess amount exceeding the upper limit of the normal blood pressure range is 2.5 mmHg, then the processing device 210 or the surgery execution agent can set the blood pressure difference threshold to a value between 3.5 mmHg and 7.1 mmHg. For example, the bleeding volume threshold is set to the average of 2.5 mmHg and 7.1 mmHg (4.8 mmHg). That is, when the subsequent processing device 210 or the surgery execution intelligent body performs EOI detection, if the blood pressure value exceeds the upper limit of the normal blood pressure range by more than 4.8 mmHg, the processing device 210 or the surgery execution intelligent body will determine that EOI is detected (EOI means that the patient's blood pressure is too high).

[0446] In some embodiments, Fig.17 As shown, EOI detection can be further performed based on the patient data of the patient. For different patients, EOI detection rules can be the same or different.

[0447] Exemplarily, if the patient only suffers from the target disease (e.g., tumor), when judging whether the patient has a massive hemorrhage during the operation, the judgment is made according to the preset bleeding volume threshold. If it is known from another patient data that the patient suffers from other diseases (e.g., anemia) in addition to the target disease (e.g., tumor), resulting in the poor physical condition of the patient, the bleeding volume threshold can be lowered when judging whether the patient has a massive hemorrhage during the operation. For example, taking the preset bleeding volume threshold of 500ml as an example (generally, the bleeding volume of the patient is greater than 500ml and will be judged as a massive hemorrhage), the bleeding volume threshold of the patient can be lowered to 300ml (i.e., the bleeding volume of the patient is greater than 300ml and is considered as a massive hemorrhage). In another example, for patients who do not suffer from hypertension, the patient's blood pressure is higher than the normal range and will be judged as an event of interest. If it is known from the patient data that the patient suffers from hypertension, the patient's blood pressure is higher than the normal range and will not be judged as an event of interest.

[0448] Step 1723: In response to detecting the occurrence of EOI, one or more preset operations corresponding to the EOI are executed.

[0449] The preset operation refers to the preset operation for responding to the event of interest. For example, if the event of interest is heavy bleeding of the patient, the preset operation may be a hemostasis operation (e.g., blood transfusion, suture hemostasis, etc.). For another example, if the event of interest is cardiac arrest of the patient, the preset operation may be a cardiac recovery operation (e.g., injecting a cardiac stimulant into the patient, performing cardiopulmonary resuscitation, etc.).

[0450] In some embodiments, one or more preset operations corresponding to the EOI are determined based on the correspondence between the EOI and the preset operation. The correspondence is learned by the agent from historical records. For example, taking the event of interest as a patient with severe bleeding and a learning trigger value of 90%, if the event of interest occurs 51 times in the historical records, and the doctor transfused the patient 49 times after the event of interest, then the corresponding ratio of the patient's severe bleeding to the transfusion operation is 49 / 51=96.1%>90%, then the surgical execution agent can establish a correspondence between severe bleeding (event of interest) and transfusion (preset operation).

[0451] In some embodiments, Fig.17 As shown, one or more preset operations corresponding to the EOI are further determined according to the patient data of the patient. For example, the surgery execution agent learns that there is a corresponding relationship between cardiac arrest (corresponding to the event of interest) and cardiac injection (corresponding to the preset operation). If the medical examination data in the patient data show that the patient suffers from kidney disease (not suitable for cardiac injection), then when performing surgery on the patient, there is no corresponding relationship between cardiac arrest and cardiac injection.

[0452] In some embodiments, executing one or more preset operations corresponding to EOI may include: after detecting the occurrence of EOI, the processing device 210 or the surgery execution agent sends a reminder message to the surgical participant, or controls the surgical equipment (e.g., an intelligent robot nurse) to perform a preset operation. For example, if the event of interest is that the doctor makes an incorrect operation, the processing device 210 or the surgery execution agent sends a voice reminder about the incorrect operation to the doctor through the second terminal device worn by the doctor. For another example, if the event of interest is that the doctor issues an instruction for the target surgical tool, the processing device 210 or the surgery execution agent can control the intelligent robot nurse to pass the target surgical tool to the doctor.

[0453] In some embodiments, the EOI includes instructions issued by a surgical participant for a target surgical tool, and executing one or more preset operations corresponding to the EOI includes: controlling the intelligent robotic nurse to pass the target surgical tool to the surgical participant.

[0454] The target surgical tool refers to the surgical tool that the surgical participant currently wants to obtain. The instruction about the target surgical tool may include the type and / or specification of the target surgical tool. For example, the instruction about the target surgical tool may be "a small hemostat is required". The method for identifying the instruction about the target surgical tool may be voice recognition, gesture recognition, etc. For example, the surgeon may input the instruction of "small hemostat" by voice through the second terminal device worn by the surgeon, and the microphone on the second terminal device collects the voice signal and sends the voice signal to the processing device 210 or the surgical execution agent. The processing device 210 or the surgical execution agent may recognize the voice signal by voice recognition, confirm that the target surgical tool required by the surgeon is a small hemostat, and send the recognition result to the nurse through the interactive device (e.g., XR glasses) worn by the nurse, so that the nurse delivers the small hemostat to the surgeon. Or after confirming that the target surgical tool required by the doctor is a small hemostat, the processing device 210 generates an instruction to deliver the small hemostat to the surgeon and sends it to the intelligent mechanical nurse, controlling the intelligent mechanical nurse to deliver the small hemostat to the surgeon.

[0455] In some embodiments, the processing device 210 or the surgery execution agent can send the above instructions to the intelligent robot nurse, and the intelligent robot nurse can identify the target surgical tool from the operating table and pass it to the surgical participant (e.g., the surgeon 1511). For example, the intelligent robot nurse 240-6 can identify the first position of the target surgical tool on the operating table based on the image of the surgical tool on the operating table, and identify the second position of the surgeon based on the image of the surgeon, determine the moving route from the first position to the second position (including moving distance, moving angle, moving placement, etc.), and based on the moving route, transport the target surgical tool from the first position to the second position to pass it to the surgeon.

[0456] The image may be acquired through an image sensor in the operating room and / or an image sensor installed on the intelligent robot nurse.

[0457] In some embodiments, the EOI includes a number of surgical tools that is less than a preset value, and executing one or more preset operations corresponding to the EOI includes: controlling the intelligent robotic nurse to replenish surgical tools.

[0458] In some embodiments, for each surgical tool, the processing device 210 or the surgical execution agent can determine whether the number of surgical tools is less than a preset value based on the surgical tool image. In this case, the first perception information includes the surgical tool image.

[0459] In some embodiments, the processing device 210 or the surgery performing agent may determine the quantity of each surgical tool through an image recognition algorithm.

[0460] The above preset value is the preset number of surgical tools. Each surgical tool can correspond to the same or different preset values. For example, the preset value of a blood bag is 4, the preset value of a syringe is 5, and so on.

[0461] In some embodiments, the processing device 210 or the surgery execution agent can predetermine the preset values ​​corresponding to various surgical tools based on the surgical plan. In some embodiments, the processing device 210 or the surgery execution agent can adjust the preset values ​​corresponding to various surgical tools in real time according to the progress of the operation. For example, at the beginning of the operation, the preset value can be increased, and when the operation is about to end, the preset value can be reduced. In some embodiments, the processing device 210 or the surgery execution agent can adjust the preset values ​​corresponding to various surgical tools based on the real-time situation of the operation. For example, if severe bleeding occurs during the operation, the preset value of the blood transfusion bag can be increased.

[0462] The number of surgical tools to be replenished may be the difference between the current number of surgical tools and the corresponding preset value, or may be greater than the difference. For example, if the scalpel count is 3 and the preset value is 5, the processing device 210 or the surgery execution agent may control the intelligent robot nurse to replenish at least 2 scalpels.

[0463] In some embodiments, the EOI includes detecting a surgical risk, and performing one or more preset operations corresponding to the EOI includes: providing a reminder about the surgical risk. The reminder may be in the form of a voice reminder, a text reminder, or the like.

[0464] For example, if the surgical risk is heavy bleeding of the patient, when the processing device 210 or the surgical execution agent detects the risk, the processing device 210 or the surgical execution agent can generate a corresponding voice reminder message (for example, the content of the voice reminder can be "Please note that the patient is bleeding too much") and send the voice reminder message to the second terminal device worn by the surgical participant (for example, the doctor), and the speaker on the second terminal device can broadcast the voice reminder content to the surgical participant. For another example, if the surgical risk is low blood pressure of the patient, when the processing device 210 or the surgical execution agent detects the risk, the processing device 210 or the surgical execution agent can generate a corresponding text reminder message (for example, the content of the voice reminder can be "Please note that the patient's blood pressure is too low") and send the text reminder message to the display screen in the operating room, and the display screen can display the text reminder message in large font and eye-catching color.

[0465] In some embodiments, the EOI includes completion of a surgery, and performing one or more preset operations corresponding to the EOI includes: generating a surgery record based on the first perception information.

[0466] In some embodiments, at the end of the operation, the operation participant (e.g., doctor) may send out operation completion information (e.g., by voice or gesture). The sensor (e.g., sound sensor or gesture sensor) on the second terminal device worn by the operation participant may collect the information and send it to the processing device 210 or the operation execution intelligent agent, and the processing device 210 or the operation execution intelligent agent may recognize the information to determine that the operation is completed.

[0467] In some embodiments, Fig.17 As shown, the processing device 210 or the surgery execution agent can generate an initial surgery record based on the first perception information (step 1730); show the initial surgery record to the doctor (step 1740); and generate a surgery record based on the initial surgery record and the feedback information input by the doctor about the initial surgery record. The details are as follows:

[0468] Step 1730, generating an initial surgical record based on the first perception information.

[0469] In some embodiments, the processing device 210 or the surgery execution agent can generate an initial surgery record based on the first sense information collected. In this case, the first sense information can include all types of data during the surgery, such as gesture data, image data, voice data, etc.

[0470] Surgical records refer to the original recorded data of surgical related information and events that occurred during the operation. For example, the initial surgical record may include surgical related information, patient related records, participant related records, etc.

[0471] The information related to the surgery includes the start / end time of the surgery, the duration of the surgery, the risk conditions that occurred, the types and quantities of surgical tools before the surgery and after the surgery (to prevent the situation where surgical tools are left in the patient's body), the types and quantities of surgical consumables before the surgery and after the surgery, etc. For example, the information related to the surgery may be "the surgery started at 14:00, ended at 17:12, and lasted 3 hours and 12 minutes; the patient had heavy bleeding during the surgery; the surgical tools before the surgery included 2 scalpels and 2 hemostatic forceps, and after the surgery, they were counted and included 2 scalpels and 2 hemostatic forceps; the surgical consumables before the surgery included 2 blood transfusion packs (400ml each), 1 dose of cardiac tonic, and 5 packs of hemostatic gauze, and after the surgery, there were 0 blood transfusion packs remaining (indicating that the blood transfusion pack has been used up), 1 dose of cardiac tonic remaining (indicating that the cardiac tonic has not been used), and 1 pack of hemostatic gauze remaining".

[0472] Patient-related records include the duration of anesthesia (the duration from the start of anesthesia to awakening), the patient's vital signs during surgery (heart rate, blood pressure, respiratory rate, etc.), blood loss, etc. For example, the patient-related records can be in the form of a patient-related record sheet, which records the duration of anesthesia (e.g., 2 hours), blood loss (e.g., 350 ml), and the patient's heart rate, blood pressure, respiratory rate, etc. at multiple time points during the operation (e.g., recorded every 10 seconds during the operation).

[0473] Participant-related records include the surgeon's action record, force record, and standing position record, as well as the nursing staff's (nurse's) standing position record, action record, etc. The action record includes the action type of each action of the surgical participant (for example, cutting, pressing, lifting, etc.) and the time corresponding to each action. The force record includes the force magnitude of the surgical participant at multiple moments during the operation (for example, recorded every 1 second). The standing position record includes the standing position of the surgical participant at multiple moments during the operation. The action record can be obtained through the action recognition device, and the force record can be obtained through the force sensing device.

[0474] In some embodiments, the processing device 210 or the surgical execution agent can use a record generation model (e.g., a preset record template) to generate an initial surgical record based on the first perception information.

[0475] Step 1740, display the initial surgical record to the physician.

[0476] In some embodiments, the processing device 210 or the surgical execution agent may display the initial surgical record to the doctor (eg, the surgeon 1511 ) via a second terminal device (eg, the second XR device 270 - 2 ).

[0477] Step 1750: Generate an operation record based on the initial operation record and feedback information (hereinafter referred to as third feedback information) input by the doctor regarding the initial operation record.

[0478] The third feedback information refers to the doctor's modification or confirmation information of the initial operation record. The doctor can input the third feedback information in the form of text input, voice input, etc. For example, the third feedback information can be "change the operation time to 4 hours".

[0479] In some embodiments, the processing device 210 or the surgery execution agent can obtain the third feedback information input by the doctor through the perception device. The perception device may include a sound sensor. For example, the doctor can verbally express the modification information or confirmation information of the initial surgery record, and the microphone on the second terminal device can collect the doctor's voice signal and send it to the processing device 210 or the surgery execution agent.

[0480] The above-mentioned operation record refers to the operation record obtained after the initial operation record is updated based on the third feedback information. For example, if the operation duration in the initial operation record is 4.5 hours and the third feedback information is "change the operation duration to 4 hours", the operation duration in the operation record is 4 hours.

[0481] In some embodiments, the processing device 210 or the surgery execution agent may generate an initial doctor's order at the same time, before, or after generating the initial surgery record. In some embodiments, the processing device 210 or the surgery execution agent may generate an initial doctor's order based on the first perception information (e.g., the voice data of the surgeon).

[0482] Initial medical orders refer to the original postoperative medical orders. Initial medical orders may include normal postoperative vital signs range, medication prescriptions, expected complications and countermeasures, nursing requirements, etc. Normal postoperative vital signs range, medication prescriptions (including medication type, frequency and total amount) and expected complications (e.g., fever, infection, etc.) may be determined based on patient data and surgical records or doctor voice signals.

[0483] In some embodiments, the processing device 210 or the surgery execution agent can show the initial medical order to the doctor. In some embodiments, the processing device 210 or the surgery execution agent can show the initial medical order to the doctor (e.g., the surgeon 1511) through the second terminal device (e.g., the second XR device 270-2).

[0484] In some embodiments, the processing device 210 or the surgery execution agent can obtain feedback information related to the doctor's order (hereinafter referred to as the fourth feedback information) input by the doctor through a sensing device (for example, a microphone on the second terminal device). The fourth feedback information includes the doctor's modification or confirmation of the initial doctor's order (for example, confirmation that the initial doctor's order is correct). For example, the fourth feedback information also includes "reduce the frequency of medication in the medication prescription by 1 / 3".

[0485] In some embodiments, the processing device 210 or the surgery execution agent can simultaneously display the initial doctor's order and the initial surgery record to the doctor. In this case, the processing device 210 or the surgery execution agent can obtain the third feedback information related to the initial surgery record and the fourth feedback information related to the initial doctor's order input by the doctor.

[0486] In some embodiments, the processing device 210 or the surgery execution agent can generate a medical order based on the initial medical order and the fourth feedback information input by the doctor. The medical order refers to the medical order obtained by updating the initial medical order based on the fourth feedback information. For example, taking the example that the frequency of medication in the medication prescription in the initial medical order is three times a day, and the fourth feedback information includes "reduce the frequency of medication in the medication prescription by 1 / 3", the frequency of medication in the medication prescription in the medical order is twice a day.

[0487] Fig.18 1800 is a flowchart of an exemplary surgical review according to some embodiments of this specification. In some embodiments, process 1800 may include the following steps. In some embodiments, process 1800 may be performed by processing device 210 or postoperative review module 450. In some embodiments, process 1800 may be performed by a postoperative review agent. For more information about agents, see Fig.17 and its related description.

[0488] For more information on the generation of medical orders and surgical records, please refer to Fig.17 In some embodiments, as described in Fig.18 As shown in step 1810 to step 1830, the processing device 210 or the postoperative review module 450 can perform postoperative care based on the surgical records and doctor's orders. Specifically:

[0489] Step 1810, monitoring the patient's postoperative vital signs.

[0490] In some embodiments, the processing device 210 or the postoperative review module 450 can also monitor the patient's postoperative signs through a vital sign monitoring instrument (e.g., an electrocardiogram, a blood pressure monitor, etc.) in the ward to determine whether the patient's postoperative vital signs are within the normal postoperative vital sign range, whether there are any abnormalities, or whether the recovery progress is normal. For example, if the patient should wake up 24 hours after the operation, but does not wake up, it means that the recovery progress is abnormal; or the blood pressure should return to the standard value after one week, but it does not reach it, indicating that the recovery progress is abnormal.

[0491] Step 1820, update the doctor's order.

[0492] In some embodiments, the processing device 210 or the postoperative review module 450 can update the doctor's order based on the patient's postoperative physical signs. For example, if the patient's blood pressure is too high after surgery, antihypertensive drugs can be added to the medication prescription in the doctor's order. For another example, the processing device 210 can add treatment or prevention measures for complications to the doctor's order when it is determined that the patient has complications or is at risk of complications based on the patient's postoperative physical signs.

[0493] In some embodiments, the processing device 210 can update the doctor's order based on the doctor's instruction. For example, the processing device 210 can display the patient's real-time vital sign monitoring data to the doctor through the second terminal device or the display terminal of the doctor's workstation. When the doctor feedbacks that the doctor's order needs to be updated, the doctor's order is updated based on the doctor's feedback instruction.

[0494] In some embodiments, the processing device 210 may send the updated medical order to a display device of a nurse workstation and / or a display device of a doctor workstation.

[0495] Step 1830, determine the postoperative care plan.

[0496] In some embodiments, the processing device 210 or the postoperative review module 450 can determine the postoperative care plan based on the updated doctor's order. The postoperative care plan refers to the care tasks that the care staff (e.g., nurses, care workers, etc.) need to perform during the patient's postoperative hospitalization. The postoperative care plan may include basic postoperative care, postoperative patient education, abnormal reports, etc. Basic postoperative care includes incision management (e.g., dressing changes, etc.), diet management, rehabilitation training, etc. Postoperative patient education includes education on postoperative precautions, patient emotional comfort, etc. Abnormal reports include reports of abnormal vital signs of patients, reports of abnormal emotions (e.g., patients are too irritable or pessimistic), reports of incision infections, etc.

[0497] In some embodiments, the processing device 210 can control the intelligent medical device (e.g., the intelligent nursing cart 240-4) to provide care for the patient based on the postoperative care plan. For example, the processing device 210 can send the patient's postoperative care plan to the nursing agent corresponding to the intelligent nursing cart 240-4. Based on the postoperative care plan, the nursing agent sends a reminder message to the nurse station when the patient needs treatment or care. In response to determining that the nurse has received the reminder message, the nursing agent controls the intelligent nursing cart 240-4 to guide the nurse to the ward to provide care for the patient. The nursing agent refers to the agent configured for the intelligent nursing cart 240-4. For more information about the agent, see Fig.17 and its related description.

[0498] In some embodiments, the processing device 210 may send the postoperative care plan to a nurse so that the nurse can provide postoperative care to the patient.

[0499] In some embodiments, the processing device 210 can update the postoperative care plan in real time according to the patient status during the care process.

[0500] In some embodiments, Fig.18 As shown in steps 1813 and 1823, the processing device 210 or the postoperative review module 450 can generate the doctor's surgical results and operation records based on the surgical records and doctor's orders, so that the doctor can review the surgical process. The details are as follows:

[0501] Step 1813, generate the doctor's surgical results and operation records.

[0502] The surgical results refer to data reflecting the results of the operation. For example, the surgical results may be a successful operation (the operation achieved the expected results) or a failed operation (the operation did not achieve the expected results). In some embodiments, the doctor's surgical results also include summary data of the doctor's surgical results within a predetermined time period (e.g., one month). For example, the doctor's surgical results may also include "the doctor participated in 15 operations in the past month, of which 14 were successful, with a surgical success rate of 93.3%."

[0503] Operation records refer to the records of the doctor's behavior during the operation, which may include action records, force records, standing position records, etc.

[0504] In some embodiments, the processing device 210 can generate the doctor's surgical achievements and operation records based on the surgical records and medical orders. In some embodiments, the processing device 210 can generate the doctor's surgical achievements and corresponding operation records of successful cases based on the doctor's historical surgical records and historical medical orders, surgical records and medical orders. In some embodiments, the processing device 210 can generate the doctor's surgical achievements and corresponding operation records of failed cases based on the doctor's historical surgical records and historical medical orders, surgical records and medical orders.

[0505] Step 1823, review the surgical process.

[0506] In some embodiments, the processing device 210 can display the generated doctor's surgical results and operation records to the doctor (for example, through the display terminal of the doctor's workstation or the second terminal device) so that the doctor can review the surgical process. By reviewing the surgical process, the doctor can summarize and look forward to the surgical work and improve his skills from his own practice.

[0507] In some embodiments, postoperative review may also include calculating postoperative operational data after multiple surgeries, including three-party (doctor, patient, and patient's family) verification records (records of reviewing surgical records), surgical success rate, average surgical time, postoperative medical order update time, operating room drug consumption and inventory, and average operating room turnover time (the time required for the operating room to be available for the next surgery from the end of the previous surgery).

[0508] Fig.19 1900 is a schematic diagram of an exemplary process of generating a surgical plan according to other embodiments of the present specification. In some embodiments, process 1900 may include the following steps. In some embodiments, process 1900 may be executed by processing device 210 or surgical planning module 410.

[0509] Step 1910, based on the patient data of the patient who needs to undergo surgery, determine whether a specialist meeting is needed.

[0510] In some embodiments, the processing device 210 or the surgical planning module 410 may determine the difficulty coefficient of the operation based on the patient data of the patient. The processing device 210 or the surgical planning module 410 may determine whether to hold an expert meeting based on the difficulty coefficient of the operation. For more information about the above embodiments, see Figure 7 and its related description.

[0511] Step 1920, in response to determining that an expert conference needs to be held, controlling the terminal device of the doctor corresponding to the patient and the terminal device of the remote expert to respectively present a virtual conference space.

[0512] In some embodiments, the processing device 210 or the surgical planning module 410 may determine whether the surgical difficulty coefficient is greater than the difficulty coefficient threshold to determine whether an expert meeting is required. In response to determining that the surgical difficulty coefficient is greater than the difficulty coefficient threshold, it is determined that an expert meeting is required; in response to determining that the surgical difficulty coefficient is not greater than the difficulty coefficient threshold, the processing device 210 or the surgical planning module 410 may directly generate a surgical plan (for example, generating a surgical plan using the process described in step 620). The difficulty coefficient threshold may be manually preset or automatically determined by the system.

[0513] In some embodiments, the processing device 210 or the surgical planning module 410 can present the virtual conference space on the doctor's second terminal device and the remote expert's fourth terminal device, respectively. Figure 7 and its related description.

[0514] Step 1930, obtaining the perception information collected by the doctor's terminal device and the remote expert's terminal device during the expert meeting.

[0515] In some embodiments, the processing device 210 or the surgical planning module 410 may obtain the fourth perception information collected by the second terminal device and the fourth terminal device during the expert meeting. Figure 7 and its related description.

[0516] Step 1940, generate a surgical plan based on the patient data and sensory information.

[0517] In some embodiments, the processing device 210 or the surgical planning module 410 can use the plan generation model to process and analyze the patient data and the fourth sensory information to generate a surgical plan. For more information about the above embodiments, please refer to Figure 6 or Figure 7 and its related description.

[0518] Fig. 20 2000 is a flowchart of exemplary preoperative preparation and surgical execution according to other embodiments of the present specification. In some embodiments, process 2000 may include the following steps. In some embodiments, process 2000 may be executed by processing device 210 or surgical planning module 410 or preoperative preparation module 430 or surgical execution module 440.

[0519] Step 2010, generating a surgical plan for the patient.

[0520] In some embodiments, the processing device 210 or the surgical planning module 410 can determine the surgical difficulty coefficient based on the patient data of the patient. The processing device 210 or the surgical planning module 410 can determine whether an expert meeting is needed based on the surgical difficulty coefficient; in response to the need to hold an expert meeting, the processing device 210 or the surgical planning module 410 can control the doctor's second terminal device and the remote expert's fourth terminal device to respectively present a virtual meeting space. The processing device 210 or the surgical planning module 410 can obtain the fourth perception information collected by the second terminal device and the fourth terminal device during the expert meeting. The processing device 210 or the surgical planning module 410 can generate a surgical plan based on the patient data and the fourth perception information. For more information about the above embodiments, please refer to Figure 7 and its related description.

[0521] In some embodiments, the processing device 210 or the surgical planning module 410 may generate an initial surgical plan based on the patient data of the patient. The processing device 210 or the surgical planning module 410 may display the initial surgical plan to the doctor. The processing device 210 or the surgical planning module 410 may generate a surgical plan based on the initial surgical plan and the second feedback information about the initial surgical plan input by the doctor through the second terminal device. For more information about the above embodiments, see Figure 8 and its related description.

[0522] Step 2020, provide preoperative care to the patient based on the surgical plan.

[0523] In some embodiments, the processing device 210 or the preoperative preparation module 430 may determine the patient's preoperative care materials based on the patient data and the surgical plan. The processing device 210 or the preoperative preparation module 430 may control the patient's first terminal device to perform preoperative care on the patient according to the preoperative care materials during the process of transporting the patient to the waiting area. For more information about the above embodiments, please refer to Fig.14 and its related description.

[0524] Step 2030, based on the surgical plan, control the intelligent robot nurse to prepare surgical tools in the operating room before the operation.

[0525] In some embodiments, the processing device 210 or the preoperative preparation module 430 can determine the type and quantity of surgical tools required based on the surgical plan, and control the intelligent robot nurse to prepare the surgical tools according to the type and quantity of surgical tools, and place them at a predetermined position on the operating table.

[0526] Step 2040: During the operation, the intelligent robot nurse is controlled to assist the operation based on the sensing information collected by one or more sensing devices in the operating room.

[0527] In some embodiments, the processing device 210 or the surgery execution module 440 can control the intelligent robot nurse to replenish the surgical tools. In some embodiments, the processing device 210 or the surgery execution module 440 can control the intelligent robot nurse to pass the target surgical tools to the surgical participants. For more information about the above embodiments, please refer to Fig.17 and its related description.

[0528] Some embodiments of the present specification also provide a system, which includes at least one storage medium, including a set of instructions; and one or more processors communicating with the at least one storage medium. When executing the instructions, the one or more processors are used to execute the above-mentioned method (such as process 500 to process 2000).

[0529] In some embodiments of the present specification, a computer-readable storage medium is further provided, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the aforementioned method (such as process 500 to process 2000).

[0530] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0531] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0532] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0533] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0534] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0535] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0536] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A method for surgical planning and execution, implemented on a computing device having at least one processing device and at least one storage device, wherein the computing device is configured with an intelligent agent capable of self-evolving based on artificial intelligence technology, and the method is executed by the intelligent agent, the method comprising: Acquiring sensory information collected by sensory equipment in an operating room during surgery on a patient; performing an event of interest (EOI) detection on the sensed information based on an EOI detection rule, wherein the EOI detection rule is learned by the agent from historical records, and the EOI includes at least one of an instruction issued by a surgical participant, a surgical risk detected, an abnormal physiological state of a patient, a number of surgical tools being less than a preset value, and completion of the surgery; as well as In response to detecting the occurrence of an EOI, a preset operation corresponding to the EOI is executed, wherein the preset operation corresponding to the EOI is determined based on a correspondence between the EOI and the preset operation, and the correspondence is learned by the agent from historical records.

2. The method according to claim 1, characterized in that The intelligent agent is integrated into one of the operating bed, intelligent surgical terminal and intelligent robot nurse in the operating room.

3. The method according to claim 1, characterized in that For patients with different diseases, the EOI detection rules are different.

4. The method according to claim 1, characterized in that The preset operation corresponding to the EOI is further determined based on the patient data of the patient.

5. The method according to claim 1, characterized in that In response to detecting that an EOI occurs, executing a preset operation corresponding to the EOI includes: In response to detecting an instruction issued by a surgical participant for a target surgical tool, the intelligent robotic nurse is controlled to pass the target surgical tool to the surgical participant.

6. The method according to claim 1, characterized in that The perception information includes a surgical tool image, and in response to detecting the occurrence of the EOI, executing a preset operation corresponding to the EOI includes: For each surgical tool, in response to detecting based on the surgical tool image that the number of the surgical tools is less than the corresponding preset value, the intelligent robot nurse is controlled to replenish the surgical tools.

7. The method according to claim 1, characterized in that In response to detecting that an EOI occurs, executing a preset operation corresponding to the EOI includes: In response to detecting a surgical risk during a surgical procedure, providing a reminder regarding the surgical risk.

8. The method according to claim 2, characterized in that In response to detecting that an EOI occurs, executing a preset operation corresponding to the EOI includes: In response to detecting that the surgery is complete, generating an initial surgery record based on the sensed information; showing said initial surgical record to the physician; A surgical record is generated based on the feedback information input by the doctor regarding the initial surgical record.

9. The method according to claim 1, characterized in that Also includes: Monitor patients’ postoperative signs; Update the doctor's orders based on the patient's postoperative signs; Determine postoperative care plan based on updated physician orders; as well as Based on the postoperative care plan, the intelligent nursing cart is controlled to provide care for the patient.

10. A computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the method according to any one of claims 1 to 9.

Images