Surgery simulation method, device, electronic device and storage medium

Abstract

The present application relates to the field of artificial intelligence technology, and specifically discloses a surgical simulation method, device, electronic device and storage medium, wherein the method comprises: constructing an initial virtual pet according to pet information and examination information of a pet to be operated on, so as to simulate the operation; displaying the virtual pet to a surgeon, and obtaining a real-time surgical video of the surgeon on the initial virtual pet; performing multiple parameter adjustment processing on the initial virtual pet according to the real-time surgical video, and obtaining simulation results and a surgical log, wherein the surgical log is used to record the surgical operation process of the surgeon on the initial virtual pet and the impact of the surgical operation process on the initial virtual pet; generating a feasibility rate of the operation according to the simulation result, and displaying the simulation result, the surgical log and the feasibility rate to the surgeon.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a surgical simulation method, device, electronic equipment and storage medium. Background Art

[0002] Currently, surgery is widely used to treat diseases in pets. However, for life-threatening diseases, since different pets have different conditions, the feasibility of surgery can only be determined based on the surgeon's experience, and then the surgery can be attempted after informing the owner of the risks. Therefore, there is an urgent need for a highly interpretable method for determining the feasibility of surgery to determine the feasibility of the surgical plan and possible problems before the surgery is performed. Summary of the invention

[0003] In order to solve the above-mentioned problems existing in the prior art, the embodiments of the present application provide a surgical simulation method, device, electronic device and storage medium, which can obtain the feasibility and possible problems of the surgical plan before the operation, thereby improving the success rate of the surgical plan.

[0004] In a first aspect, an embodiment of the present application provides a surgical simulation method, the method comprising:

[0005] constructing an initial virtual pet according to pet information and examination information of the pet to be operated on, so as to simulate the operation;

[0006] Showing the virtual pet to the surgeon and obtaining the real-time surgical video of the surgeon on the initial virtual pet;

[0007] According to the real-time surgical video, the parameters of the initial virtual pet are adjusted multiple times to obtain simulation results and surgical logs, wherein the surgical logs are used to record the surgical operation process of the surgeon on the initial virtual pet and the impact of the surgical operation process on the initial virtual pet;

[0008] The feasibility rate of the surgery is generated based on the simulation results, and the simulation results, surgery log and feasibility rate are presented to the surgeon.

[0009] In a second aspect, an embodiment of the present application provides a surgical simulation device, comprising:

[0010] A modeling module, used for constructing an initial virtual pet according to pet information and examination information of the pet to be operated on, so as to simulate the operation;

[0011] A simulation module is used to show the virtual pet to the surgeon, obtain the surgeon's real-time surgical video of the initial virtual pet, and perform multiple parameter adjustment processing on the initial virtual pet according to the real-time surgical video to obtain simulation results and a surgical log, wherein the surgical log is used to record the surgeon's surgical operation process on the initial virtual pet and the impact of the surgical operation process on the initial virtual pet;

[0012] The display module is used to generate the feasibility rate of the operation according to the simulation results, and to display the simulation results, operation log and feasibility rate to the surgeon.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, the processor is connected to a memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the method of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program enables a computer to execute the method of the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and is computer-operable to cause the computer to execute the method of the first aspect.

[0016] The implementation of the present application has the following beneficial effects:

[0017] In the embodiment of the present application, an initial virtual pet with the same state as the pet to be operated is constructed by using the pet information of the pet to be operated on and the examination information reflecting its current disease state. Then, by showing the virtual pet to the surgeon, the surgeon can perform the operation on the initial virtual pet, and then realize the simulation of the operation plan. At the same time, during the operation, the surgeon obtains the real-time operation video of the initial virtual pet, and then performs multiple parameter adjustment processing on the initial virtual pet according to the real-time operation video to obtain the simulation result and the operation log. Specifically, by real-time recognition of the surgical action in the real-time operation video, and then by adjusting the parameters of the virtual pet, the effect of the action on the virtual pet is displayed in real time in front of the surgeon. For example, if the surgeon makes a dissection action, the position, depth and size of the dissection are identified through the video, and the corresponding position of the virtual pet is adjusted so that the position shows the appearance after the dissection. At the same time, the surgical operation process of the surgeon on the initial virtual pet and the effect of the surgical operation process on the initial virtual pet are recorded through the operation log. Finally, the feasibility rate of the operation can be generated according to the simulation result, and the simulation result, the operation log and the feasibility rate can be displayed to the surgeon. Therefore, by building a virtual pet, the surgical plan can be simulated before the operation, which can not only make the surgeon familiar with the surgical process, but also preview the results of the operation, and then derive the feasibility rate of the surgical plan with high interpretability. In addition, by recording the surgical operation process, the risks that may arise during the operation can be predicted, and then the doctor can be assisted in optimizing the surgical plan, thereby improving the success rate of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the implementation modes of the present application, the drawings required for use in the description of the implementation modes will be briefly introduced below. Obviously, the drawings described below are some implementation modes of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of the hardware structure of a surgical simulation device provided in an embodiment of the present application;

[0020] Figure 2 A system framework diagram of a surgical simulation method in a scenario of surgical simulation for pets provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a surgical simulation method provided in an embodiment of the present application;

[0022] Figure 4A schematic diagram of displaying a virtual pet to a surgeon via a flat-panel display provided in an embodiment of the present application;

[0023] Figure 5 A flowchart of a method for performing multiple parameter adjustment processing on an initial virtual pet according to a real-time surgical video to obtain simulation results and surgical logs provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of an embodiment of the present application providing a method of grouping at least one second key working image according to a screening process to obtain at least one set of image sets;

[0025] Figure 7 A functional module composition block diagram of a surgical simulation device provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0029] Reference to "embodiment" herein means that a particular feature, result, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] First, see Figure 1 , Figure 1The hardware structure diagram of a surgical simulation device provided in the embodiment of the present application is as follows: The surgical simulation device 100 includes at least one processor 101 , a communication line 102 , a memory 103 and at least one communication interface 104 .

[0031] In this embodiment, the processor 101 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0032] The communication link 102 may include a path to transmit information between the above components.

[0033] The communication interface 104 may be any transceiver-like device (eg, antenna, etc.) for communicating with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc.

[0034] The memory 103 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0035] In this embodiment, the memory 103 can exist independently and be connected to the processor 101 via the communication line 102. The memory 103 can also be integrated with the processor 101. The memory 103 provided in the embodiment of the present application can generally be non-volatile. Among them, the memory 103 is used to store computer-executable instructions for executing the scheme of the present application, and the execution is controlled by the processor 101. The processor 101 is used to execute the computer-executable instructions stored in the memory 103, so as to implement the method provided in the following embodiment of the present application.

[0036] In an optional implementation, the computer-executable instructions may also be referred to as application code, which is not specifically limited in this application.

[0037] In an optional embodiment, the processor 101 may include one or more CPUs, such as Figure 1 CPU0 and CPU1 in.

[0038] In an optional embodiment, the surgical simulation device 100 may include multiple processors, such as Figure 1 1 and 107. Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0039] In an optional embodiment, if the surgical simulation device 100 is a server, for example, it can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Then the surgical simulation device 100 can also include an output device 105 and an input device 106. The output device 105 communicates with the processor 101 and can display information in a variety of ways. For example, the output device 105 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 106 communicates with the processor 101 and can receive user input in a variety of ways. For example, the input device 106 can be a mouse, a keyboard, a touch screen device, or a sensor device.

[0040] The above-mentioned surgery simulation device 100 can be a general device or a special device. The embodiment of the present application does not limit the type of the surgery simulation device 100.

[0041] Secondly, it should be noted that the simulation method provided by the present application can be applied to various scenarios of planned simulation, such as surgery simulation, training simulation, etc. for pets. In this embodiment, the simulation method provided by the present application will be described by taking the scenario of surgery simulation for pets as an example. The simulation methods in other scenarios are similar to the simulation methods in the scenario of surgery simulation for pets, and will not be described in detail here.

[0042] at last, Figure 2 A system framework diagram of a surgical simulation method in a scenario of surgical simulation of pets provided in an embodiment of the present application. Specifically, the system may include: a physician device 201, a simulation device 202, and a database 203. Among them, the physician device 201 may be a smart phone (such as an Android phone, an iOS phone, a Windows Phone phone, etc.), a tablet computer, a PDA, a laptop computer, a mobile Internet device MID (Mobile Internet Devices, referred to as: MID), etc. Specifically, the physician device 201 is used to obtain pet information and examination information of the pet to be operated on, and send the obtained information to the simulation device 202. At the same time, the simulation device 202 may further include a simulation processing device 204 and a simulation display device 205. Among them, the simulation processing device 204 may be a server, which is used to receive the pet information and examination information sent by the physician device 201, and then retrieve the data in the database 203 according to the received information to construct a virtual pet, and display the constructed virtual pet to the surgeon through the simulation display device 205. At the same time, the simulation processing device 204 is also used to receive the real-time surgical video fed back by the simulation display device 205, and retrieve the data in the database 203 according to the real-time surgical video, and adjust the parameters of the virtual pet displayed by the simulation display device 205 in real time. In addition, the simulation processing device 204 is also used to determine the feasibility of the surgical plan according to the simulation results. The simulation display device 205 may include a display device 206 and a photographing device 207, wherein the display device 206 may be a flat-panel display and / or a 3D projector, which is used to receive and display the virtual pet, the feasibility of the surgical plan, the simulation results and the surgical log sent by the simulation processing device 204. The photographing device 207 may be a plurality of cameras with different viewing angles arranged around the display device, which is used to obtain the real-time surgical video of the surgeon on the virtual pet, and generate a corresponding surgical log, and send the video and the surgical log to the simulation processing device 204.

[0043] In this embodiment, by building a virtual pet, the surgical plan is simulated before the operation, which not only allows the surgeon to be familiar with the surgical process, but also allows the surgeon to preview the results of the operation, thereby obtaining a highly interpretable feasibility rate of the surgical plan. In addition, by recording the surgical log of the surgical operation process, the risks that may occur during the operation can be predicted, thereby assisting the surgeon in optimizing the surgical plan and improving the success rate of the operation.

[0044] The following will take the scenario of pet surgery simulation as an example to illustrate the surgery simulation method disclosed in this application:

[0045] See also Figure 3 , Figure 3 A schematic diagram of a surgical simulation method provided in an embodiment of the present application. The surgical simulation method comprises the following steps:

[0046] 301: Construct an initial virtual pet according to the pet information and examination information of the pet to be operated on, so as to simulate the operation.

[0047] In this embodiment, the pet information may include: breed information, age information, body shape information, weight information, appearance information, etc. The surgeon may obtain the pet information from the database by inputting the medical record number of the pet when it was treated in the surgeon device 201, or the surgeon device 201 may generate a QR code of the information filling interface, and the owner of the pet to be operated on may scan the QR code and fill in and upload the information by himself.

[0048] In this embodiment, the examination information may include information generated by various auxiliary devices after examining the pet undergoing surgery. Similarly, the surgeon may obtain the examination information from the database by inputting the medical record number of the pet during the surgery in the physician device 201 .

[0049] Based on this, in this embodiment, the standard body model library can be queried through the breed information and age information in the pet information to obtain the body model of the pet to be operated on in the standard development state of the same breed and age. Then, the matched body model is adjusted through the actual body shape information of the pet to be operated on, such as body length, limb length, body width, weight information, and examination information. For example, the muscle-to-bone ratio, visceral fat content, etc. are adjusted through body shape information and weight information. At the same time, the corresponding disease characteristics are added to the body model according to the examination information, thereby obtaining a virtual pet that is the same as the pet to be operated on.

[0050] 302: Show the virtual pet to the surgeon, and obtain the real-time surgical video of the surgeon on the initial virtual pet.

[0051] In this embodiment, the virtual pet can be displayed to the surgeon via the display device 206. For example, the display device 206 can be a flat-panel display and / or a 3D projector. Figure 4 As shown, the virtual pet can be presented in the display in the form of a 3D model, and the surgeon can rotate, move and enlarge the 3D model in the display by touching. At the same time, different surgical instruments are switched by selecting the toolbar, and then the corresponding surgical operation is performed on the position of the virtual pet corresponding to the display area by touching. At this time, in addition to the cameras with multiple different viewing angles surrounding the display device 206, the corresponding photographic device 207 can also include a screen recording software set in the display device 206. In addition to obtaining the surgeon's touch operation video on the display device 206, the screen recording video of the flat-panel display can also be obtained, and the touch operation video and the screen recording video are used together as the surgeon's real-time surgical video of the initial virtual pet, which improves the accuracy and credibility of subsequent analysis.

[0052] In this embodiment, when the display device 206 is a 3D projector, the virtual pet can be projected on the display table in the form of a 3D stereoscopic image, and then the surgeon can directly perform the surgery on the projected 3D stereoscopic image, thereby more realistically restoring the surgical environment.

[0053] 303: Perform multiple parameter adjustments on the initial virtual pet according to the real-time surgical video to obtain simulation results and surgical logs.

[0054] In this embodiment, the surgical log is used to record the surgical operation process of the surgeon on the initial virtual pet and the impact of the surgical operation process on the initial virtual pet. Figure 5 As shown, the following will use the scenario where the display device 206 is a 3D projector to illustrate a method provided by this embodiment for performing multiple parameter adjustment processing on an initial virtual pet according to a real-time surgical video to obtain a simulation result and a surgical log, the method comprising:

[0055] 501: In the i-th parameter adjustment process, the first real-time video A i Perform feature extraction to obtain the first action feature B i .

[0056] In this embodiment, i is an integer greater than or equal to 1, and when i=1, the first real-time video A i For example, first, the first real-time video A iPerform frame-by-frame analysis to obtain at least one first key action image. Then, based on the image content of each first key action image in the at least one first key action image, screen the at least one first key action image to obtain at least one second key working image. Specifically, since the extraction of action features is to clarify what kind of surgical operation the surgeon has performed on the virtual pet, images that only contain surgeons or pets to be operated on cannot meet the above purpose. Therefore, the first key action images that only contain surgeons or pets to be operated on in the image content can be screened out, and then the remaining first key action images are used as the at least one second key working image.

[0057] Then, in this embodiment, at least one second key working image can be grouped according to the screening process to obtain at least one set of images. Specifically, the at least one first key action image is arranged in the order of time of the video frames corresponding to each first key action image. Therefore, the at least one second key working image can be grouped according to the position of the first key action image to be screened out in the at least one first key action image as a separation point. For example, Figure 6 As shown, there are 7 first key action images: A, B, C, D, E, F and G. After screening, image C and image F are screened out. Then, the remaining second key working images A, B, D, E and G can be divided into three groups according to the positions of image C and image F in the original sequence, namely: image A and image B, image D and image E, and image G.

[0058] Then, in this embodiment, image feature extraction can be performed on each image set in at least one image set to obtain at least one image feature that corresponds to at least one image set. Specifically, feature extraction can be performed on each second key working image in each image set to obtain at least one key image feature. Then, at least one key image feature is vertically spliced ​​according to the chronological order of occurrence of each corresponding second key working image to obtain the image feature of each image set. That is, the image features of each image set are aggregated to obtain the at least one image feature.

[0059] Finally, at least one image feature can be vertically spliced ​​to obtain the first action feature B i .

[0060] 502: According to the first real-time video A i and the first action feature Bi for the first virtual pet C i Adjust the parameters and get the first simulation result D i .

[0061] In this embodiment, when i=1, the first virtual pet C i The initial virtual pet is the initial virtual pet. That is, in the first adjustment, the object of adjustment is the initial virtual pet, and in the subsequent adjustment process, the object of adjustment is the result of the previous adjustment. Specifically, first, according to the first real-time video A i Determine the implementation location and operation information of the surgical operation currently performed by the surgeon. That is, identify the surgical action currently performed by the surgeon and the real-time location of the action. For example, if the surgical action currently performed by the surgeon is to open the pet's abdomen, then the implementation location is the abdomen and the operation information is the opening operation. Furthermore, the implementation location and operation information can be more accurately identified, such as: which area in the abdomen, the direction, depth, opening size, and opening technique of the opening, etc.

[0062] Then, the first action feature Bi can be matched in the action library to obtain the surgical action information corresponding to the first action feature Bi, and then the impact information of the surgical operation on the implementation position can be determined based on the operation information and the surgical action information. Specifically, the impact information is used to identify the state change of the implementation position before and after the surgical operation. Taking the above-mentioned dissection example, when performing this action, the corresponding position will present a dissection opening along the dissection direction, and at the same time, the muscle texture or internal organs presented under the opening will be determined according to the dissection depth, opening size and dissection position, and presented. By classifying this information, the impact information can be obtained.

[0063] Then, the adjustment parameters of the implementation position can be determined according to the impact information, and then the first virtual pet C can be adjusted according to the adjustment parameters. i Adjust the parameters and get the first simulation result D i Taking the above-mentioned dissection example as an example, the impact information records the opening position, opening direction, opening depth, and the content presented under the opening (muscle texture or internal organs). Therefore, the first virtual pet C can be adjusted according to this information. i The model parameters of the corresponding parts make it present the image recorded in the impact information. Alternatively, the corresponding map can be matched in the database according to the impact information, and the first virtual pet C i The texture of the corresponding part in the first virtual pet C is replaced, and then the first virtual pet C is i The parameters are adjusted to obtain the first simulation result D i .

[0064] 503: Add the process of parameter adjustment for the i-th time into the first operation log E i , get the second operation log F i .

[0065] In this embodiment, when i=1, the first operation log E iEmpty log.

[0066] 504: The first simulation result D i As the first virtual pet C in the (i+1)th parameter adjustment process i+1 , the second operation log F i As the first operation log E in the (i+1)th parameter adjustment process i+1 , and obtain the first real-time video A corresponding to the i+1th parameter adjustment process i+1 , perform the i+1th parameter adjustment process, until multiple parameter adjustment processes are performed to obtain the simulation results and surgery logs.

[0067] In an optional embodiment, the process of the i-th parameter adjustment process is added to the first surgical log E i , get the second operation log F i Afterwards, it can be determined whether the current result meets the end condition. If so, the multiple parameter adjustment processes can be terminated directly, and the simulation results and surgical logs can be output, thereby improving the processing efficiency. Specifically, the first action feature B can be obtained. i Then, when the action tag is the end action, the multiple parameter adjustment processes are terminated directly, and the first simulation result D i As a result of the simulation, the second operation log F i As the adjustment log; and when the action label is not the end action, continue the parameter adjustment process for the i+1th time. Alternatively, a confirmation interface can be displayed to the surgeon, and the surgeon's confirmation instruction can be received. Then, when the confirmation instruction is the end operation instruction, the multiple parameter adjustment processes are terminated, and the first simulation result D i As a result of the simulation, the second operation log F i As an adjustment log; when the action label is the instruction to continue the surgery, the parameter adjustment process for the i+1th time will be continued.

[0068] 304: Generate a feasibility rate of the surgery according to the simulation result, and present the simulation result, the surgery log and the feasibility rate to the surgeon.

[0069] In this embodiment, the simulation results can be subjected to disease-related inspections to determine their health status after surgery. Exemplarily, the inspection process for the simulation results can be determined based on the disease information of the pet to be operated on, and relevant inspections can be performed on the simulation results to determine the various inspection indicators of the simulation results, which are then compared with standard health indicators to determine the treatment result information. Then, the simulation results are subjected to relevant sequelae inspections based on the disease information to determine the sequelae information. The feasibility rate of the surgical plan is then determined based on the treatment result information and the sequelae information. Specifically, the treatment efficiency of the surgical plan can be determined based on the treatment result information, and at the same time, the severity level of each sequelae contained therein can be determined based on the sequelae information, and then the feasibility rate of the surgical plan can be determined based on the treatment efficiency and severity level. The feasibility rate can be expressed by formula ①:

[0070]

[0071] Wherein, P represents the feasibility of the surgical plan, Q represents the treatment efficiency, and max(R) represents the maximum severity level among the severity levels of various sequelae contained in the sequelae information.

[0072] In an optional implementation, the risks that may be encountered during the operation can also be determined based on the operation log, and then the risk rate of the operation plan can be determined based on the number of occurrences and probability of occurrence of the risk, and then the feasibility rate of the operation plan can be determined based on the risk rate, treatment efficiency and severity level. At this time, the feasibility rate can be expressed by formula ②:

[0073]

[0074] Where O represents the risk rate.

[0075] In summary, in the surgical simulation method provided by the present invention, an initial virtual pet with the same state as the pet to be operated is constructed through the pet information of the pet to be operated and the examination information reflecting its current disease state. Then, by showing the virtual pet to the surgeon, the surgeon can perform the surgical operation on the initial virtual pet, and then realize the simulation of the surgical plan. At the same time, during the surgical operation, the surgeon obtains the real-time surgical video of the initial virtual pet, and then performs multiple parameter adjustment processing on the initial virtual pet according to the real-time surgical video to obtain the simulation result and the surgical log. Specifically, by real-time recognition of the surgical action in the real-time surgical video, and then by adjusting the parameters of the virtual pet, the effect of the action on the virtual pet is displayed in real time in front of the surgeon. For example, if the surgeon makes a dissection action, the position, depth and size of the dissection are identified through the video, and the corresponding position of the virtual pet is adjusted so that the position shows the appearance after the dissection. At the same time, the surgical operation process of the surgeon on the initial virtual pet and the effect of the surgical operation process on the initial virtual pet are recorded through the surgical log. Finally, the feasibility of the operation can be generated based on the simulation results, and the simulation results, operation logs and feasibility rates can be presented to the surgeon. Therefore, by building a virtual pet and rehearsing the operation plan before the operation, the surgeon can not only become familiar with the operation process, but also rehearse the results of the operation, and then obtain a highly interpretable feasibility rate of the operation plan. In addition, by recording the operation log of the operation process, the risks that may arise during the operation can be predicted, and then the surgeon can be assisted in optimizing the operation plan, thereby improving the success rate of the operation.

[0076] See also Figure 7 , Figure 7 This is a functional module block diagram of a surgical simulation device provided in the embodiment of the present application. Figure 7 As shown, the surgical simulation device 700 includes:

[0077] A modeling module 701 is used to construct an initial virtual pet based on the pet information and examination information of the pet to be operated on, so as to simulate the operation;

[0078] The simulation module 702 is used to show the virtual pet to the surgeon, obtain the real-time surgical video of the surgeon on the initial virtual pet, and perform multiple parameter adjustment processing on the initial virtual pet according to the real-time surgical video to obtain simulation results and surgical logs, wherein the surgical logs are used to record the surgical operation process of the surgeon on the initial virtual pet and the impact of the surgical operation process on the initial virtual pet;

[0079] The display module 703 is used to generate the feasibility rate of the operation according to the simulation result, and to display the simulation result, the operation log and the feasibility rate to the surgeon.

[0080] In the embodiment of the present invention, in terms of performing multiple parameter adjustment processing on the virtual pet according to the real-time surgical video to obtain simulation results and surgical logs, the simulation module 702 is specifically used to:

[0081] In the i-th parameter adjustment process, the first real-time video A i Perform feature extraction to obtain the first action feature B i , where i is an integer greater than or equal to 1, and when i=1, the first real-time video A i For real-time surgical video;

[0082] According to the first real-time video A i and the first action feature Bi for the first virtual pet C i Adjust the parameters and get the first simulation result D i , when i=1, the first virtual pet C i For the initial virtual pet;

[0083] Add the process of parameter adjustment for the i-th time into the first surgery log E i , get the second operation log F i , where, when i=1, the first surgery log E i Empty log;

[0084] The first simulation result D i As the first virtual pet C in the (i+1)th parameter adjustment process i+1 , the second operation log F i As the first operation log E in the (i+1)th parameter adjustment process i+1 , and obtain the first real-time video A corresponding to the i+1th parameter adjustment process i+1 , perform the i+1th parameter adjustment process, until multiple parameter adjustment processes are performed to obtain the simulation results and surgery logs.

[0085] In the embodiment of the present invention, when the first real-time video A i Perform feature extraction to obtain the first action feature B i In terms of simulation module 702, it is specifically used for:

[0086] For the first real-time video A i Perform frame-by-frame analysis to obtain at least one first key action image;

[0087] According to the image content of each first key action image in the at least one first key action image, screening the at least one first key action image to obtain at least one second key working image;

[0088] According to the screening process, at least one second key working image is grouped to obtain at least one set of images;

[0089] Extracting image features from each image set in at least one image set to obtain at least one image feature, wherein the at least one image feature corresponds one-to-one to the at least one image set;

[0090] Vertically splice at least one image feature to obtain the first action feature B i .

[0091] In the embodiment of the present invention, in terms of extracting image features from each image set in at least one image set to obtain at least one image feature, the simulation module 702 is specifically configured to:

[0092] Performing feature extraction on each second key working image in each group of images to obtain at least one key image feature;

[0093] Vertically splicing at least one key image feature according to the chronological order of occurrence of each corresponding second key working image to obtain image features of each group of image sets;

[0094] The image features of each group of image sets are aggregated to obtain at least one image feature.

[0095] In an embodiment of the present invention, according to the first real-time video A i and the first action feature Bi for the first virtual pet C i Adjust the parameters and get the first simulation result D i In terms of simulation module 702, it is specifically used for:

[0096] According to the first real-time video A i Determine the location and operation information of the surgical operation currently performed by the surgeon;

[0097] Matching the first action feature Bi in the action library to obtain surgical action information corresponding to the first action feature Bi;

[0098] Determine, based on the operation information and the surgical action information, the impact information of the surgical operation on the implementation position, wherein the impact information is used to identify the state change of the implementation position before and after the surgical operation;

[0099] Determine adjustment parameters for implementation locations based on impact information;

[0100] According to the adjustment parameters, the first virtual pet C i Adjust the parameters and get the first simulation result D i .

[0101] In the embodiment of the present invention, when the process of the i-th parameter adjustment process is added to the first operation log E i , get the second operation log F i Afterwards, the simulation module 702 is further used to:

[0102] Get the first action feature B i The action tag;

[0103] When the action label is the end action, the multiple parameter adjustment processes are terminated and the first simulation result D is i As a result of the simulation, the second operation log F i As an adjustment log;

[0104] When the action label is not an end action, the parameter adjustment process is performed for the (i+1)th time.

[0105] In the embodiment of the present invention, when the process of the i-th parameter adjustment process is added to the first operation log E i , get the second operation log F i Afterwards, the simulation module 702 is further used to:

[0106] Display a confirmation interface to the surgeon and receive the surgeon's confirmation instruction;

[0107] When the confirmation instruction is an instruction to end the operation, the multiple parameter adjustment processes are terminated, and the first simulation result D is i As a result of the simulation, the second operation log F i As an adjustment log;

[0108] When the action label is a continue surgery instruction, the (i+1)th parameter adjustment process is performed.

[0109] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the electronic device 800 includes a transceiver 801, a processor 802 and a memory 803. They are connected via a bus 804. The memory 803 is used to store computer programs and data, and can transmit the data stored in the memory 803 to the processor 802.

[0110] The processor 802 is used to read the computer program in the memory 803 and perform the following operations:

[0111] constructing an initial virtual pet according to pet information and examination information of the pet to be operated on, so as to simulate the operation;

[0112] Showing the virtual pet to the surgeon and obtaining the real-time surgical video of the surgeon on the initial virtual pet;

[0113] According to the real-time surgical video, the parameters of the initial virtual pet are adjusted multiple times to obtain simulation results and surgical logs, wherein the surgical logs are used to record the surgical operation process of the surgeon on the initial virtual pet and the impact of the surgical operation process on the initial virtual pet;

[0114] The feasibility rate of the surgery is generated based on the simulation results, and the simulation results, surgery log and feasibility rate are presented to the surgeon.

[0115] In an embodiment of the present invention, in performing multiple parameter adjustment processing on a virtual pet according to a real-time surgical video to obtain simulation results and surgical logs, the processor 802 is specifically configured to perform the following operations:

[0116] In the i-th parameter adjustment process, the first real-time video A i Perform feature extraction to obtain the first action feature B i , where i is an integer greater than or equal to 1, and when i=1, the first real-time video A i For real-time surgical video;

[0117] According to the first real-time video A i and the first action feature Bi for the first virtual pet C i Adjust the parameters and get the first simulation result D i , when i=1, the first virtual pet C i For the initial virtual pet;

[0118] Add the process of parameter adjustment for the i-th time into the first surgery log E i , get the second operation log F i , where, when i=1, the first surgery log E i Empty log;

[0119] The first simulation result D i As the first virtual pet C in the (i+1)th parameter adjustment process i+1 , the second operation log F i As the first operation log E in the (i+1)th parameter adjustment process i+1 , and obtain the first real-time video A corresponding to the i+1th parameter adjustment process i+1 , perform the i+1th parameter adjustment process, until multiple parameter adjustment processes are performed to obtain the simulation results and surgery logs.

[0120] In the embodiment of the present invention, when the first real-time video A i Perform feature extraction to obtain the first action feature B i In one aspect, the processor 802 is specifically configured to perform the following operations:

[0121] For the first real-time video A i Perform frame-by-frame analysis to obtain at least one first key action image;

[0122] According to the image content of each first key action image in the at least one first key action image, screening the at least one first key action image to obtain at least one second key working image;

[0123] According to the screening process, at least one second key working image is grouped to obtain at least one set of images;

[0124] Extracting image features from each image set in at least one image set to obtain at least one image feature, wherein the at least one image feature corresponds one-to-one to the at least one image set;

[0125] Vertically splice at least one image feature to obtain the first action feature B i .

[0126] In an embodiment of the present invention, in extracting image features from each image set in at least one image set to obtain at least one image feature, the processor 802 is specifically configured to perform the following operations:

[0127] Performing feature extraction on each second key working image in each group of images to obtain at least one key image feature;

[0128] Vertically splicing at least one key image feature according to the chronological order of occurrence of each corresponding second key working image to obtain image features of each group of image sets;

[0129] The image features of each group of image sets are aggregated to obtain at least one image feature.

[0130] In an embodiment of the present invention, according to the first real-time video A i and the first action feature Bi for the first virtual pet C i Adjust the parameters and get the first simulation result D i In one aspect, the processor 802 is specifically configured to perform the following operations:

[0131] According to the first real-time video A i Determine the location and operation information of the surgical operation currently performed by the surgeon;

[0132] Matching the first action feature Bi in the action library to obtain surgical action information corresponding to the first action feature Bi;

[0133] Determine, based on the operation information and the surgical action information, the impact information of the surgical operation on the implementation position, wherein the impact information is used to identify the state change of the implementation position before and after the surgical operation;

[0134] Determine adjustment parameters for implementation locations based on impact information;

[0135] According to the adjustment parameters, the first virtual pet C i Adjust the parameters and get the first simulation result D i .

[0136] In the embodiment of the present invention, when the process of the i-th parameter adjustment process is added to the first operation log E i , get the second operation log F i Afterwards, the processor 802 is further configured to perform the following operations:

[0137] Get the first action feature B i The action tag;

[0138] When the action label is the end action, the multiple parameter adjustment processes are terminated and the first simulation result D is i As a result of the simulation, the second operation log F i As an adjustment log;

[0139] When the action label is not an end action, the parameter adjustment process is performed for the (i+1)th time.

[0140] In the embodiment of the present invention, when the process of the i-th parameter adjustment process is added to the first operation log E i , get the second operation log F i Afterwards, the processor 802 is further configured to perform the following operations:

[0141] Display a confirmation interface to the surgeon and receive the surgeon's confirmation instruction;

[0142] When the confirmation instruction is an instruction to end the operation, the multiple parameter adjustment processes are terminated, and the first simulation result D is i As a result of the simulation, the second operation log F i As an adjustment log;

[0143] When the action label is a continue surgery instruction, the (i+1)th parameter adjustment process is performed.

[0144] It should be understood that the surgical simulation device in the present application may include a smart phone (such as an Android phone, an iOS phone, a Windows Phone phone, etc.), a tablet computer, a PDA, a laptop computer, a mobile Internet device MID (Mobile Internet Devices, referred to as: MID), a robot or a wearable device, etc. The above-mentioned surgical simulation device is only an example, not an exhaustive list, including but not limited to the above-mentioned surgical simulation device. In practical applications, the above-mentioned surgical simulation device may also include: a smart vehicle terminal, a computer device, etc.

[0145] Through the description of the above implementation modes, those skilled in the art can clearly understand that the present invention can be implemented by combining software with a hardware platform. Based on such an understanding, all or part of the contribution of the technical solution of the present invention to the background technology can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various implementation modes of the present invention or some parts of the implementation modes.

[0146] Therefore, the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement some or all of the steps of any surgical simulation method described in the above method implementation. For example, the storage medium may include a hard disk, a floppy disk, an optical disk, a tape, a disk, a USB flash drive, a flash memory, etc.

[0147] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any surgical simulation method recorded in the above method embodiments.

[0148] It should be noted that, for the sake of simplicity, the aforementioned method implementations are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the implementations described in the specification are all optional implementations, and the actions and modules involved are not necessarily required by the present application.

[0149] In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device implementation described above is only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0152] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software program module.

[0153] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0154] A person skilled in the art may understand that all or part of the steps in the various methods of the above-mentioned embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable memory, and the memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0155] The above is a detailed introduction to the implementation methods of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above implementation methods is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A surgical simulation method, characterized in that: The method comprises: constructing an initial virtual pet according to pet information and examination information of the pet to be operated on, so as to simulate the operation; Showing the virtual pet to a surgeon, and obtaining a real-time surgical video of the surgeon on the initial virtual pet; Performing multiple parameter adjustment processing on the initial virtual pet according to the real-time surgical video to obtain simulation results and a surgical log, wherein the surgical log is used to record the surgical operation process of the surgeon on the initial virtual pet and the impact of the surgical operation process on the initial virtual pet; generating a feasibility rate of the surgery according to the simulation result, and presenting the simulation result, the surgery log and the feasibility rate to the surgeon; The step of performing multiple parameter adjustment processes on the virtual pet according to the real-time surgical video to obtain simulation results and surgical logs includes: In the i-th parameter adjustment process, the first real-time video A i Perform feature extraction to obtain the first action feature B i , wherein i is an integer greater than or equal to 1, and when i=1, the first real-time video A1 is the real-time surgical video; According to the first real-time video A i and the first action feature Bi for the first virtual pet C i Adjust the parameters and get the first simulation result D i , when i=1, the first virtual pet C1 is the initial virtual pet; Add the process of the i-th parameter adjustment process to the first surgery log E i , get the second operation log F i , where, when i=1, the first operation log E1 is an empty log; The first simulation result D i As the first virtual pet C in the (i+1)th parameter adjustment process i+1 , the second operation log F i As the first operation log E in the (i+1)th parameter adjustment process i+1 , and obtain the first real-time video A corresponding to the i+1th parameter adjustment process i+1 , performing the i+1th parameter adjustment process, until the simulation result and the surgery log are obtained after performing the multiple parameter adjustment processes.

2. The method according to claim 1, characterized in that The first real-time video A i Perform feature extraction to obtain the first action feature B i ,include: The first real-time video A i Perform frame-by-frame analysis to obtain at least one first key action image; According to the image content of each first key action image in the at least one first key action image, the at least one first key action image is screened to obtain the at least one second key working image; According to the screening process, the at least one second key working image is grouped to obtain at least one set of images; Performing image feature extraction on each image set in the at least one image set to obtain at least one image feature, wherein the at least one image feature corresponds one-to-one to the at least one image set; The at least one image feature is vertically spliced ​​to obtain the first action feature B i .

3. The method according to claim 2, characterized in that The step of extracting image features from each image set in the at least one image set to obtain at least one image feature comprises: Performing feature extraction on each second key working image in each group of images to obtain at least one key image feature; Vertically splicing the at least one key image feature according to the chronological order of occurrence of each corresponding second key working image to obtain image features of each group of image sets; The image features of each group of image sets are aggregated to obtain the at least one image feature.

4. The method according to claim 1, characterized in that: The first real-time video A i and the first action feature Bi for the first virtual pet C i Adjust the parameters and get the first simulation result D i ,include: According to the first real-time video A i Determine the implementation location and operation information of the surgical operation currently performed by the surgeon; Matching the first action feature Bi in an action library to obtain surgical action information corresponding to the first action feature Bi; Determine, according to the operation information and the surgical action information, the impact information of the surgical operation on the implementation location, wherein the impact information is used to identify the state change of the implementation location before and after the surgical operation is performed; Determining adjustment parameters of the implementation location according to the impact information; The first virtual pet C is adjusted according to the adjustment parameters i Adjust the parameters to obtain the first simulation result D i .

5. The method according to any one of claims 1 to 4, characterized in that: In the process of adding the i-th parameter adjustment process into the first surgery log E i , get the second operation log F i Afterwards, the method further comprises: Get the first action feature B i The action tag; When the action tag is an end action, the multiple parameter adjustment processes are ended, and the first simulation result D i As the simulation result, the second operation log F i as said adjustment log; When the action tag is not an end action, the (i+1)th parameter adjustment process is performed.

6. The method according to any one of claims 1 to 4, characterized in that: In the process of adding the i-th parameter adjustment process into the first surgery log E i , get the second operation log F i Afterwards, the method further comprises: Displaying a confirmation interface to the surgeon and receiving a confirmation instruction from the surgeon; When the confirmation instruction is an instruction to end the operation, the multiple parameter adjustment processes are ended, and the first simulation result D is i As the simulation result, the second operation log F i as said adjustment log; When the action label is a continue surgery instruction, the (i+1)th parameter adjustment process is performed.

7. A surgical simulation device, characterized in that: The device comprises: A modeling module, used for constructing an initial virtual pet according to pet information and examination information of the pet to be operated on, so as to simulate the operation; A simulation module, used for showing the virtual pet to the surgeon, obtaining the real-time surgical video of the surgeon on the initial virtual pet, and performing multiple parameter adjustment processing on the initial virtual pet according to the real-time surgical video to obtain simulation results and a surgical log, wherein the surgical log is used to record the surgical operation process of the surgeon on the initial virtual pet and the impact of the surgical operation process on the initial virtual pet; A presentation module, used for generating a feasibility rate of the surgery according to the simulation result, and presenting the simulation result, the surgery log and the feasibility rate to the surgeon; Wherein, in the aspect of performing multiple parameter adjustment processing on the virtual pet according to the real-time surgical video to obtain simulation results and surgical logs, the simulation module is specifically used for: In the i-th parameter adjustment process, the first real-time video A i Perform feature extraction to obtain the first action feature B i , wherein i is an integer greater than or equal to 1, and when i=1, the first real-time video A1 is the real-time surgical video; According to the first real-time video A i and the first action feature Bi for the first virtual pet C i Adjust the parameters and get the first simulation result D i , when i=1, the first virtual pet C1 is the initial virtual pet; Add the process of the i-th parameter adjustment process to the first surgery log E i , get the second operation log F i , where, when i=1, the first operation log E1 is an empty log; The first simulation result D i As the first virtual pet C in the (i+1)th parameter adjustment process i+1 , the second operation log F i As the first operation log E in the (i+1)th parameter adjustment process i+1 , and obtain the first real-time video A corresponding to the i+1th parameter adjustment process i+1 , performing the i+1th parameter adjustment process, until the simulation result and the surgery log are obtained after performing the multiple parameter adjustment processes.

8. An electronic device, characterized in that: The method comprises a processor, a memory, a communication interface and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the one or more programs include instructions for executing the steps in any one of the methods of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 6.

Images