Method and system for guiding path of hysteroscopic surgery

Abstract

The invention provides a hysteroscopic surgery path guiding method and system, and the method comprises the steps: obtaining multi-modal scanning data of a target region, and constructing a three-dimensional structure model of the target region based on the multi-modal scanning data of the target region; generating abnormal path guidance information of each abnormal region through a path guidance generation strategy based on the abnormal type of each abnormal region and the position range of each abnormal region; on the basis of the current position information of the hysteroscope and the current picture information of the hysteroscope, current advancing information of the hysteroscope is generated in the three-dimensional structure model; and based on the abnormal path guidance information of each abnormal area and the current advancing information of the hysteroscope, in the current picture information of the hysteroscope, generating the current path guidance content of the hysteroscope to each abnormal area, and performing picture display on the current path guidance content of the hysteroscope to each abnormal area to a worker. By adopting the scheme, the real-time path guiding effect on the hysteroscopic surgery can be improved.

Description

Technical Field

[0001] The present invention relates to the field of structural modeling and medical assistance technology, and in particular to a path guidance method and system for hysteroscopic surgery. Background Art

[0002] Hysteroscopic surgery refers to minimally invasive surgery performed using a hysteroscope. A hysteroscope is an advanced device for diagnosing and treating diseases within the uterine cavity. It can assist medical staff in clearly observing various changes within the uterine cavity, making it easier for them to perform surgery. However, in order to achieve the purpose of minimally invasive surgery, hysteroscopic surgery often cannot be directly observed with the naked eye. Instead, medical staff are required to perform the actual surgery according to their own work experience. This often has a poor guidance effect on medical staff with different surgical experience. Therefore, how to improve the guidance of medical staff on the surgical path during the actual operation of hysteroscopic surgery is the current research focus.

[0003] The traditional technical solution is to mark the surgical path on the display interface through manual annotation based on hysteroscopic surgery. However, the manually marked screen can only be used for fixed screens. In the real-time changing screens of the surgery, the accuracy of manual annotation is low, and it is impossible to adapt to the changes in annotations. It is also easy for medical staff to misunderstand the specific path of the marked area, resulting in poor real-time path guidance for hysteroscopic surgery. Summary of the Invention

[0004] The main purpose of the present invention is to provide a path guidance method and system for hysteroscopic surgery, aiming to solve the problem of the existing technology that since manually labeled images can only be used for fixed images, the accuracy of manual labeling is low in the real-time changing images of the operation, and it is impossible to adapt to the labeling changes, and it is easy for medical staff to misunderstand the specific path of the labeled area, thereby resulting in poor real-time path guidance effect for hysteroscopic surgery.

[0005] To achieve the above object, the present invention provides a path guidance method for hysteroscopic surgery, the method comprising:

[0006] Acquiring multimodal scanning data of a target area, and constructing a three-dimensional structural model of the target area based on the multimodal scanning data of the target area;

[0007] In response to the staff's annotation operation in the three-dimensional structural model, anomaly types of each abnormal area in the target area are collected, and based on the anomaly types and location ranges of each abnormal area, abnormal path guidance information for each abnormal area is generated through a path guidance generation strategy;

[0008] Collecting current position information of the hysteroscope and current image information of the hysteroscope, and generating current movement information of the hysteroscope in the three-dimensional structure model based on the current position information of the hysteroscope and the current image information of the hysteroscope;

[0009] Based on the abnormal path guidance information of each abnormal area and the current travel information of the hysteroscope, the current path guidance content of the hysteroscope to each abnormal area is generated in the current screen information of the hysteroscope, and the current path guidance content of the hysteroscope to each abnormal area is displayed to the staff on the screen.

[0010] Optionally, constructing a three-dimensional structural model of the target area based on the multimodal scanning data of the target area includes:

[0011] Splitting the multimodal scan data into sub-scan data of each scan type, and constructing an initial three-dimensional structural model of the target area corresponding to each scan type based on the sub-scan data of each scan type;

[0012] Based on the initial three-dimensional structure model corresponding to each scan type, a three-dimensional structure model of the target area is generated through a model fusion strategy.

[0013] Optionally, the generating of abnormal path guidance information for each abnormal area by a path guidance generation strategy based on the abnormality type and the location range of each abnormal area includes:

[0014] For each abnormal area, based on the position range of the abnormal area, identifying the spatial position information of the abnormal area in the three-dimensional structural model and the regional abnormal structure data of the abnormal area, and generating a hysteroscopic route for the abnormal area through the three-dimensional structural model based on the spatial position information of the abnormal area;

[0015] Based on the regional abnormal structure data of the abnormal area, marking the abnormal structure model of the abnormal area in the three-dimensional structure model, and based on the abnormal type of the abnormal area, identifying the entry point position of the abnormal area in the abnormal structure model through an abnormal entry point identification network;

[0016] The hysteroscopic route of the abnormal area and the entry point position of the abnormal area are used as abnormal path guidance information of the abnormal area.

[0017] Optionally, generating current movement information of the hysteroscope in the three-dimensional structure model based on the current position information of the hysteroscope and the current image information of the hysteroscope includes:

[0018] Based on the current position information of the hysteroscope, the spatial position information of the hysteroscope is located in the three-dimensional structure model, and based on the current image information of the hysteroscope, the uterine wall structure data in the current image information is identified through an image recognition network;

[0019] Based on the uterine wall structure data, in an area where the spatial position information of the hysteroscope is located in the three-dimensional structure model, adapting the area range corresponding to the uterine wall structure data, and identifying the orientation information of the hysteroscope based on the range position information of the area range in the three-dimensional structure model and the spatial position information of the hysteroscope;

[0020] Based on the current position information of the hysteroscope and the orientation information of the hysteroscope, the current travel route of the hysteroscope is generated, and the current travel route of the hysteroscope is used as the current travel information of the hysteroscope.

[0021] Optionally, based on the abnormal path guidance information of each abnormal area and the current travel information of the hysteroscope, generating the current path guidance content of the hysteroscope to each abnormal area in the current picture information of the hysteroscope includes:

[0022] For each abnormal area, based on the current position information of the hysteroscope, the abnormal path guidance information of the abnormal area is adjusted to obtain the current route guidance information of the hysteroscope, and based on the current travel route of the hysteroscope and the current route guidance information of the hysteroscope, the current travel direction adjustment information of the hysteroscope is generated through the route adjustment network;

[0023] Based on the current travel direction adjustment information of the hysteroscope, a route adjustment guidance mark of the hysteroscope is generated in the current image information of the hysteroscope, and the hysteroscope adjustment guidance mark is used as the current path guidance content of the hysteroscope to the abnormal area.

[0024] Optionally, when the current image information of the hysteroscope includes the abnormal area, the method further includes:

[0025] Based on the current image information of the hysteroscope, identifying the current regional structure data of the abnormal area through an image recognition network;

[0026] identifying relative position information between the hysteroscope and the abnormal area based on the current area structure data of the abnormal area and the area abnormality structure data of the abnormal area;

[0027] Based on the relative position information, the entry point position of the abnormal area, and the abnormal structure model of the abnormal area, the entry point adjustment route of the hysteroscope is generated, and the entry point adjustment route, the entry point position, and the current area structure data of the abnormal area are marked and displayed in the current screen information to obtain the entry guidance information of the abnormal area.

[0028] In addition, to achieve the above-mentioned purpose, the present invention further provides a path guidance system for hysteroscopic surgery, the path guidance system for hysteroscopic surgery comprising:

[0029] an acquisition module, configured to acquire multimodal scanning data of a target area and construct a three-dimensional structural model of the target area based on the multimodal scanning data of the target area;

[0030] a generation module, configured to collect anomaly types of each abnormal area in the target area in response to a staff member's annotation operation in the three-dimensional structural model, and generate abnormal path guidance information for each abnormal area based on the anomaly type and the location range of each abnormal area using a path guidance generation strategy;

[0031] an acquisition module, configured to acquire current position information of the hysteroscope and current image information of the hysteroscope, and generate current movement information of the hysteroscope in the three-dimensional structure model based on the current position information of the hysteroscope and the current image information of the hysteroscope;

[0032] A display module is used to generate the current path guidance content of the hysteroscope to each of the abnormal areas based on the abnormal path guidance information of each of the abnormal areas and the current travel information of the hysteroscope in the current screen information of the hysteroscope, and to display the current path guidance content of the hysteroscope to each of the abnormal areas to the staff on the screen.

[0033] Optionally, the acquisition module is specifically configured to:

[0034] Splitting the multimodal scan data into sub-scan data of each scan type, and constructing an initial three-dimensional structural model of the target area corresponding to each scan type based on the sub-scan data of each scan type;

[0035] Based on the initial three-dimensional structure model corresponding to each scan type, a three-dimensional structure model of the target area is generated through a model fusion strategy.

[0036] Optionally, the generating module is specifically configured to:

[0037] For each abnormal area, based on the position range of the abnormal area, identifying the spatial position information of the abnormal area in the three-dimensional structural model and the regional abnormal structure data of the abnormal area, and generating a hysteroscopic route for the abnormal area through the three-dimensional structural model based on the spatial position information of the abnormal area;

[0038] Based on the regional abnormal structure data of the abnormal area, marking the abnormal structure model of the abnormal area in the three-dimensional structure model, and based on the abnormal type of the abnormal area, identifying the entry point position of the abnormal area in the abnormal structure model through an abnormal entry point identification network;

[0039] The hysteroscopic route of the abnormal area and the entry point position of the abnormal area are used as abnormal path guidance information of the abnormal area.

[0040] Optionally, the acquisition module is specifically used to:

[0041] Based on the current position information of the hysteroscope, the spatial position information of the hysteroscope is located in the three-dimensional structure model, and based on the current image information of the hysteroscope, the uterine wall structure data in the current image information is identified through an image recognition network;

[0042] Based on the uterine wall structure data, in an area where the spatial position information of the hysteroscope is located in the three-dimensional structure model, adapting the area range corresponding to the uterine wall structure data, and identifying the orientation information of the hysteroscope based on the range position information of the area range in the three-dimensional structure model and the spatial position information of the hysteroscope;

[0043] Based on the current position information of the hysteroscope and the orientation information of the hysteroscope, the current travel route of the hysteroscope is generated, and the current travel route of the hysteroscope is used as the current travel information of the hysteroscope.

[0044] Optionally, the display module is specifically used to:

[0045] For each abnormal area, based on the current position information of the hysteroscope, the abnormal path guidance information of the abnormal area is adjusted to obtain the current route guidance information of the hysteroscope, and based on the current travel route of the hysteroscope and the current route guidance information of the hysteroscope, the current travel direction adjustment information of the hysteroscope is generated through the route adjustment network;

[0046] Based on the current travel direction adjustment information of the hysteroscope, a route adjustment guidance mark of the hysteroscope is generated in the current image information of the hysteroscope, and the hysteroscope adjustment guidance mark is used as the current path guidance content of the hysteroscope to the abnormal area.

[0047] Optionally, the device further includes:

[0048] A first recognition module is configured to recognize current regional structural data of the abnormal region through an image recognition network based on current image information of the hysteroscope;

[0049] a second identification module, configured to identify relative position information between the hysteroscope and the abnormal area based on current area structure data of the abnormal area and area abnormality structure data of the abnormal area;

[0050] A marking module is used to generate an entry point adjustment route for the hysteroscope based on the relative position information, the entry point position of the abnormal area, and the abnormal structure model of the abnormal area, and mark and display the entry point adjustment route, the entry point position, and the current area structure data of the abnormal area in the current screen information to obtain entry guidance information for the abnormal area.

[0051] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods in the first aspect.

[0053] In a fifth aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the first aspect are implemented.

[0054] The present invention provides a path guidance method and system for hysteroscopic surgery, the method comprising: acquiring multimodal scanning data of a target area, and constructing a three-dimensional structural model of the target area based on the multimodal scanning data of the target area; in response to a staff member's annotation operation in the three-dimensional structural model, collecting the abnormality type of each abnormal area of ​​the target area, and based on the abnormality type of each abnormal area and the position range of each abnormal area, generating abnormal path guidance information for each abnormal area through a path guidance generation strategy; collecting the current position information of the hysteroscope and the current image information of the hysteroscope, and generating the current travel information of the hysteroscope in the three-dimensional structural model based on the current position information of the hysteroscope and the current image information of the hysteroscope; based on the abnormal path guidance information of each abnormal area and the current travel information of the hysteroscope, generating the current path guidance content of the hysteroscope to each abnormal area in the current image information of the hysteroscope, and displaying the current path guidance content of the hysteroscope to each abnormal area to the staff on the screen. This solution combines multimodal scanning data to construct a three-dimensional structural model of the target area, which facilitates the staff to efficiently identify and locate abnormal areas in the target area. Then, this sub-solution generates abnormal path guidance information for each abnormal area based on the abnormal type and position range of each located abnormal area, thereby providing path positioning guidance to each medical staff before performing hysteroscopic surgery, thereby improving the path guidance accuracy and guidance effect for medical staff. Then, this solution collects the current position information and current image information of the hysteroscope in real time, thereby intelligently analyzing the path guidance content of the hysteroscope at the current position and displaying the image to the staff, thereby eliminating the need for manual marking, and instead using real-time annotation, and being able to adjust the path guidance content even as the position and orientation of the hysteroscope change, thereby improving the timeliness and efficiency of guidance for staff, thereby comprehensively improving the real-time path guidance effect for hysteroscopic surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 This is a flow chart of a path guidance method for hysteroscopic surgery provided by an embodiment of the present invention;

[0057] Figure 2 Schematic diagram of the structure of a path guidance system for hysteroscopic surgery provided by an embodiment of the present invention;

[0058] Figure 3 This is a diagram of the internal structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The path guidance method for hysteroscopic surgery provided by an embodiment of the present invention is applied to the path guidance system for hysteroscopic surgery. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0060] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0061] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0062] The path guidance method for hysteroscopic surgery provided in the embodiments of the present application can be applied to the path guidance application environment of hysteroscopic surgery. The method can be applied to a terminal, a server, or a system including a terminal and a server, and implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptop computers, etc. Among them, the terminal combines multimodal scanning data to construct a three-dimensional structural model of the target area, which is convenient for the staff to efficiently identify and locate the abnormal areas of the target area. Then, this sub-scheme generates abnormal path guidance information for each abnormal area based on the abnormal type and position range of each located abnormal area, so as to provide path positioning guidance to each medical staff before performing hysteroscopic surgery, thereby improving the path guidance accuracy and guidance effect for medical staff. Then, this scheme collects the current position information and current screen information of the hysteroscope in real time, thereby intelligently analyzing the path guidance content of the hysteroscope at the current position and displaying the screen to the staff, so that there is no need for manual marking, but through real-time labeling, and can adjust the path guidance content even if the position and direction of the hysteroscope change, it improves the timeliness and efficiency of the guidance to the staff, thereby comprehensively improving the real-time path guidance effect of hysteroscopic surgery.

[0063] In one embodiment, Figure 1 As shown, a path guidance method for hysteroscopic surgery is provided, which is described by taking the application of the method to a terminal as an example, and includes the following steps:

[0064] Step S101 : acquiring multimodal scanning data of a target area, and constructing a three-dimensional structural model of the target area based on the multimodal scanning data of the target area.

[0065] In this embodiment, the terminal performs a three-dimensional scan of the user's target area using various instruments, including but not limited to ultrasound equipment and magnetic resonance imaging equipment, to obtain multimodal scan data of the target area. The terminal then constructs a three-dimensional structural model of the target area based on the multimodal scan data. The target area is an area treatable by hysteroscopy-assisted surgery, namely, the uterine region. The method for constructing the three-dimensional structural model of the target area is any method corresponding to a modeling program that constructs a 3D structural model based on structural data.

[0066] In step S102, in response to the staff's annotation operation in the three-dimensional structure model, the abnormal type of each abnormal area in the target area is collected, and based on the abnormal type of each abnormal area and the location range of each abnormal area, the abnormal path guidance information of each abnormal area is generated through the path guidance generation strategy.

[0067] In this embodiment, the terminal sends the 3D structural model to the staff client, then receives the staff's annotation operations on the 3D structural model, and obtains the abnormality type of each abnormal area in the target area and the location range of each abnormal area in the 3D structural model. The abnormality type includes, but is not limited to, the disease type of each uterine region, such as uterine septum, uterine fibroids (submucosal, intramural fibroids not adjacent to the serosal layer), intrauterine adhesions, retained products of pregnancy, intrauterine foreign bodies, and incarcerated intrauterine contraceptive rings. Based on the abnormality type and location range of each abnormal area, the terminal generates abnormal path guidance information for each abnormal area using a path guidance generation strategy. This abnormal path guidance information includes the hysteroscope route from the entrance of the uterine region to the abnormal area, as well as the entry point location of the abnormal area. The specific generation process will be described in detail later.

[0068] Step S103 : collecting the current position information of the hysteroscope and the current image information of the hysteroscope, and generating the current movement information of the hysteroscope in the three-dimensional structure model based on the current position information of the hysteroscope and the current image information of the hysteroscope.

[0069] In this embodiment, the terminal collects the current position information of the hysteroscope and the current image information of the hysteroscope, and based on the current position information and the current image information of the hysteroscope, generates the current movement information of the hysteroscope in the three-dimensional structure model. The current movement information of the hysteroscope is the current movement route of the hysteroscope. The specific generation process will be described in detail later.

[0070] Step S104, based on the abnormal path guidance information of each abnormal area and the current travel information of the hysteroscope, the current path guidance content of the hysteroscope to each abnormal area is generated in the current screen information of the hysteroscope, and the current path guidance content of the hysteroscope to each abnormal area is displayed to the staff on the screen.

[0071] In this embodiment, based on the abnormal path guidance information of each abnormal area and the current travel information of the hysteroscope, the terminal generates the current path guidance content of the hysteroscope to each abnormal area in the current image information of the hysteroscope, and displays the current path guidance content of the hysteroscope to each abnormal area to the staff on the screen. Among them, when showing it to the staff, half of the display interface of the hysteroscope displays the real-time simulated position information of the hysteroscope in the three-dimensional structural model of the abnormal area, as well as the image information captured by the hysteroscope. The current path guidance content includes the route adjustment guidance mark of the hysteroscope, that is, the arrow or guide line and other identification information that guides the adjustment of the route and direction of the hysteroscope.

[0072] Based on the above scheme, by combining multimodal scanning data, a three-dimensional structural model of the target area is constructed, which is convenient for staff to efficiently identify and locate abnormal areas in the target area. Then, this sub-scheme generates abnormal path guidance information for each abnormal area based on the abnormal type and position range of each located abnormal area, so as to provide path positioning guidance to each medical staff before performing hysteroscopic surgery, thereby improving the path guidance accuracy and guidance effect for medical staff. Then, this scheme collects the current position information and current picture information of the hysteroscope in real time, thereby intelligently analyzing the path guidance content of the hysteroscope at the current position and displaying the picture to the staff, so that there is no need for manual marking, but through real-time labeling, and can adjust the path guidance content even as the position and orientation of the hysteroscope change, thereby improving the timeliness and efficiency of guidance to staff, thereby comprehensively improving the real-time path guidance effect of hysteroscopic surgery.

[0073] Optionally, a three-dimensional structural model of the target area is constructed based on the multimodal scanning data of the target area, including: splitting the multimodal scanning data into sub-scan data of each scanning type, and constructing an initial three-dimensional structural model of the target area corresponding to each scanning type based on the sub-scan data of each scanning type; based on the initial three-dimensional structural model corresponding to each scanning type, a three-dimensional structural model of the target area is generated through a model fusion strategy.

[0074] In this embodiment, the terminal splits the multimodal scan data into sub-scan data for each scan type and, based on the sub-scan data for each scan type, constructs an initial 3D structural model of the target area corresponding to each scan type. The terminal then generates a 3D structural model of the target area based on the initial 3D structural models corresponding to each scan type using a model fusion strategy. This model fusion strategy involves merging the two initial 3D structural models using a 3D modeling program and then smoothing the fused model structure to create a resulting 3D structural model.

[0075] Based on the above solution, the sub-scan data of multiple scan types are modeled separately and then the models are fused, thereby improving the accuracy of the obtained three-dimensional structural model of the target area.

[0076] Optionally, based on the abnormal type of each abnormal area and the location range of each abnormal area, abnormal path guidance information for each abnormal area is generated through a path guidance generation strategy, including: for each abnormal area, based on the location range of the abnormal area, identifying the spatial location information of the abnormal area in the three-dimensional structure model and the regional abnormal structure data of the abnormal area, and based on the spatial location information of the abnormal area, generating a hysteroscopic route for the abnormal area through the three-dimensional structure model; based on the regional abnormal structure data of the abnormal area, marking the abnormal structure model of the abnormal area in the three-dimensional structure model, and based on the abnormal type of the abnormal area, identifying the entry point position of the abnormal area in the abnormal structure model through the abnormal entry point identification network; using the hysteroscopic route of the abnormal area and the entry point position of the abnormal area as the abnormal path guidance information of the abnormal area.

[0077] In this embodiment, for each abnormal area, the terminal identifies the spatial location information of the abnormal area in the three-dimensional structural model and the regional abnormal structure data of the abnormal area based on the location range of the abnormal area. Based on the spatial location information of the abnormal area, a route planning network is used to generate a hysteroscopic route for the abnormal area in the three-dimensional structural model. The route planning network is a self-organizing neural network (SONN) optimized based on the BP neural network.

[0078] Based on the regional abnormal structure data of the abnormal area, the terminal marks the abnormal structure model of the abnormal area in the three-dimensional structure model. Based on the abnormal type of the abnormal area, the abnormal entry point recognition network identifies the entry point location of the abnormal area in the abnormal structure model. The entry point recognition network is a reinforcement learning convolutional neural network obtained by training sample abnormal structure models of different abnormal types. The sample abnormal structure model is a three-dimensional structure model with entry point location information annotated.

[0079] The terminal uses the hysteroscopic route of the abnormal area and the entry point position of the abnormal area as abnormal path guidance information of the abnormal area.

[0080] Based on the above scheme, by identifying the hysteroscopic route of the abnormal area and the entry point position of the abnormal area, abnormal path guidance information of the abnormal area is generated, which improves the comprehensiveness and accuracy of the guidance for medical staff.

[0081] Optionally, based on the current position information of the hysteroscope and the current picture information of the hysteroscope, the current travel information of the hysteroscope is generated in the three-dimensional structural model, including: based on the current position information of the hysteroscope, locating the spatial position information of the hysteroscope in the three-dimensional structural model, and based on the current picture information of the hysteroscope, identifying the uterine wall structure data in the current picture information through an image recognition network; based on the uterine wall structure data, in the area where the spatial position information of the hysteroscope in the three-dimensional structural model is located, adapting the area range corresponding to the uterine wall structure data, and identifying the orientation information of the hysteroscope based on the range position information of the area range in the three-dimensional structural model and the spatial position information of the hysteroscope; based on the current position information of the hysteroscope and the orientation information of the hysteroscope, generating the current travel route of the hysteroscope, and using the current travel route of the hysteroscope as the current travel information of the hysteroscope.

[0082] In this embodiment, the terminal locates the spatial position information of the hysteroscope in the three-dimensional structure model based on the current position information of the hysteroscope, and identifies the uterine wall structure data in the current image information through the image recognition network based on the current image information of the hysteroscope.

[0083] Based on the uterine wall structure data, the terminal adapts the regional range corresponding to the uterine wall structure data within the region of the hysteroscope's spatial position information in the three-dimensional structural model. The adaptation method involves performing data distribution processing on the structural data of the hysteroscope's spatial position information within the three-dimensional structural model to obtain three-dimensional uterine cavity spatial data distribution information. The terminal then generates three-dimensional uterine cavity wall data distribution information based on the uterine wall structure data and the regional range corresponding to the uterine cavity wall structure data. Finally, based on the three-dimensional uterine cavity wall data distribution information, the terminal searches within the three-dimensional uterine cavity spatial data distribution information for sub-three-dimensional uterine cavity spatial data distribution information whose distribution overlaps with the three-dimensional uterine cavity wall data distribution information by greater than a predetermined overlap threshold (e.g., 90%, 95%, 99%, etc.) preset in the terminal. The distribution range within the three-dimensional structural model serves as the regional range corresponding to the adapted uterine wall structure data.

[0084] The terminal identifies the orientation of the hysteroscope based on the range position information of the region in the three-dimensional structural model and the spatial position information of the hysteroscope. The orientation information is a vector line segment of the hysteroscope, which is connected by the spatial position information of the hysteroscope as the origin and the middle position point of the range position information as the direction point.

[0085] Finally, the terminal generates the current route of the hysteroscope based on the current position information of the hysteroscope and the orientation information of the hysteroscope, and uses the current route of the hysteroscope as the current travel information of the hysteroscope. Specifically, the terminal constructs the historical forward trajectory information of the hysteroscope based on the current position information of the hysteroscope and the position information of the hysteroscope collected each time. Then, based on the historical forward trajectory and the orientation information of the hysteroscope, the terminal generates the current route of the hysteroscope through a trajectory prediction network. The trajectory prediction network can be a trajectory prediction network based on a Sequential Network.

[0086] Based on the above solution, by combining the images captured by the hysteroscope and the position information of the hysteroscope, the previous route of the hysteroscope can be identified, thereby improving the timeliness and accuracy of the analysis of the route of the hysteroscope.

[0087] Optionally, based on the abnormal path guidance information of each abnormal area and the current travel information of the hysteroscope, the current path guidance content of the hysteroscope to each abnormal area is generated in the current picture information of the hysteroscope, including: for each abnormal area, based on the current position information of the hysteroscope, adjusting the abnormal path guidance information of the abnormal area to obtain the current route guidance information of the hysteroscope, and based on the current travel route of the hysteroscope and the current route guidance information of the hysteroscope, generating the current travel direction adjustment information of the hysteroscope through the route adjustment network; based on the current travel direction adjustment information of the hysteroscope, generating a route adjustment guidance mark of the hysteroscope in the current picture information of the hysteroscope, and using the hysteroscope adjustment guidance mark as the current path guidance content of the hysteroscope to the abnormal area.

[0088] In this embodiment, for each abnormal area, the terminal adjusts the abnormal path guidance information of the abnormal area based on the current position information of the hysteroscope to obtain the current route guidance information of the hysteroscope, and generates the current travel direction adjustment information of the hysteroscope through the route adjustment network based on the current travel route of the hysteroscope and the current route guidance information of the hysteroscope. Among them, the current route guidance information is the current guidance route with the current position information of the hysteroscope as the starting point and the range where the abnormal area is located as the end point. Then, the terminal identifies the current travel route of the hysteroscope, the deviation direction and the deviation angle of the current guidance route of the hysteroscope, and adjusts the orientation information of the hysteroscope based on the deviation direction and the deviation angle to obtain the new orientation information of the hysteroscope. Finally, the terminal uses the new orientation information as the current travel direction adjustment information of the hysteroscope.

[0089] Based on the hysteroscope's current heading adjustment information, the terminal generates a route adjustment guidance mark within the hysteroscope's current image information, and uses the hysteroscope adjustment guidance mark as the current path guidance content for the hysteroscope to the abnormal area. The route adjustment guidance mark is the screen identification information obtained by projecting the new heading information onto the image information according to the identification type preset by the staff.

[0090] Based on the above scheme, by collecting the real-time position information of the hysteroscope, a visual hysteroscope adjustment guidance mark is generated, which facilitates the staff to effectively guide the current path of the abnormal area, improves the intuitiveness of the guidance information, the timeliness of the guidance to the staff, and the effectiveness of the guidance.

[0091] Optionally, when the current picture information of the hysteroscope includes an abnormal area, the method also includes: based on the current picture information of the hysteroscope, identifying the current area structure data of the abnormal area through an image recognition network; based on the current area structure data of the abnormal area and the area abnormal structure data of the abnormal area, identifying the relative position information of the hysteroscope and the abnormal area; based on the relative position information, the entry point position of the abnormal area, and the abnormal structure model of the abnormal area, generating the entry point adjustment route of the hysteroscope, and marking and displaying the entry point adjustment route, entry point position, and the current area structure data of the abnormal area in the current picture information to obtain entry guidance information for the abnormal area.

[0092] In this embodiment, the terminal identifies the relative position information between the hysteroscope and the abnormal area based on the current area structure data of the abnormal area and the area abnormal structure data of the abnormal area. The relative position information includes an angle deviation value and a distance deviation value between the hysteroscope and the abnormal area. For example, if the abnormal area is in the lower left corner of the image information, the angle deviation value is a deviation toward the lower left corner. The deviation angle is the angle between the line connecting the position of the abnormal area and the center point of the image and a line perpendicular to the image information.

[0093] Then, based on the relative position information, the entry point position of the abnormal area, and the abnormal structure model of the abnormal area, the terminal generates an entry point adjustment route for the hysteroscope, and marks and displays the entry point adjustment route, the entry point position, and the current regional structure data of the abnormal area in the current screen information to obtain the entry guidance information of the abnormal area. Among them, the entry point adjustment route is used to guide the hysteroscope to the route where the entry point is located. The route uses the entry point of the abnormal area as the target point, the position information of the hysteroscope as the starting point, and the relative position information as the hysteroscope adjustment amount to generate the route. Ensure that the staff can accurately locate the entry point position of the abnormal area. The marking methods of the entry point adjustment route, the entry point position, and the current regional structure data of the abnormal area are different. For example, the current regional structure data of the abnormal area can be marked in the screen information by a mesh surface, a line segment, or a dotted line, while the entry point position can be marked by a bright color symbol, such as a dot symbol, a triangle symbol, a cross, etc. The entry point adjustment route can be marked by curves or arrows, thereby improving the intuitiveness of medical staff, as well as the efficiency and effectiveness of guidance for medical staff.

[0094] Based on the above scheme, this scheme collects the current position information and current screen information of the hysteroscope in real time, thereby intelligently analyzing the path guidance content of the hysteroscope at the current position and displaying the screen to the staff, thereby eliminating the need for manual marking. Instead, through real-time labeling, the path guidance content can be adjusted as the position and orientation of the hysteroscope change, thereby improving the timeliness and efficiency of guidance to staff, thereby comprehensively improving the real-time path guidance effect of hysteroscopic surgery.

[0095] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0096] Based on the same inventive concept, the present application also provides a hysteroscopic surgery path guidance system for implementing the above-mentioned hysteroscopic surgery path guidance method. The solution provided by this system is similar to the solution described in the above-mentioned method. Therefore, the specific limitations of one or more hysteroscopic surgery path guidance system embodiments provided below can be found in the above-mentioned limitations of the hysteroscopic surgery path guidance method, and will not be repeated here.

[0097] Further references Figure 2 , as a response to the above Figure 1 To implement the method shown, the present application provides an embodiment of a path guidance system 200 for hysteroscopic surgery, which includes an acquisition module 210, a generation module 220, a collection module 230, and a display module 240, wherein:

[0098] An acquisition module 210 is configured to acquire multimodal scanning data of a target area and construct a three-dimensional structural model of the target area based on the multimodal scanning data of the target area;

[0099] A generation module 220 is configured to collect anomaly types of each abnormal area in the target area in response to the staff's annotation operation in the three-dimensional structural model, and generate abnormal path guidance information for each abnormal area based on the anomaly type and location range of each abnormal area using a path guidance generation strategy;

[0100] An acquisition module 230 is configured to acquire current position information of the hysteroscope and current image information of the hysteroscope, and generate current movement information of the hysteroscope in the three-dimensional structure model based on the current position information of the hysteroscope and the current image information of the hysteroscope;

[0101] The display module 240 is used to generate the current path guidance content of the hysteroscope to each of the abnormal areas based on the abnormal path guidance information of each of the abnormal areas and the current travel information of the hysteroscope in the current screen information of the hysteroscope, and to display the current path guidance content of the hysteroscope to each of the abnormal areas to the staff on the screen.

[0102] Optionally, the acquisition module 210 is specifically configured to:

[0103] Splitting the multimodal scan data into sub-scan data of each scan type, and constructing an initial three-dimensional structural model of the target area corresponding to each scan type based on the sub-scan data of each scan type;

[0104] Based on the initial three-dimensional structure model corresponding to each scan type, a three-dimensional structure model of the target area is generated through a model fusion strategy.

[0105] Optionally, the generating module 220 is specifically configured to:

[0106] For each abnormal area, based on the position range of the abnormal area, identifying the spatial position information of the abnormal area in the three-dimensional structural model and the regional abnormal structure data of the abnormal area, and generating a hysteroscopic route for the abnormal area through the three-dimensional structural model based on the spatial position information of the abnormal area;

[0107] Based on the regional abnormal structure data of the abnormal area, marking the abnormal structure model of the abnormal area in the three-dimensional structure model, and based on the abnormal type of the abnormal area, identifying the entry point position of the abnormal area in the abnormal structure model through an abnormal entry point identification network;

[0108] The hysteroscopic route of the abnormal area and the entry point position of the abnormal area are used as abnormal path guidance information of the abnormal area.

[0109] Optionally, the acquisition module 230 is specifically configured to:

[0110] Based on the current position information of the hysteroscope, the spatial position information of the hysteroscope is located in the three-dimensional structure model, and based on the current image information of the hysteroscope, the uterine wall structure data in the current image information is identified through an image recognition network;

[0111] Based on the uterine wall structure data, in an area where the spatial position information of the hysteroscope is located in the three-dimensional structure model, adapting the area range corresponding to the uterine wall structure data, and identifying the orientation information of the hysteroscope based on the range position information of the area range in the three-dimensional structure model and the spatial position information of the hysteroscope;

[0112] Based on the current position information of the hysteroscope and the orientation information of the hysteroscope, the current travel route of the hysteroscope is generated, and the current travel route of the hysteroscope is used as the current travel information of the hysteroscope.

[0113] Optionally, the display module 240 is specifically configured to:

[0114] For each abnormal area, based on the current position information of the hysteroscope, the abnormal path guidance information of the abnormal area is adjusted to obtain the current route guidance information of the hysteroscope, and based on the current travel route of the hysteroscope and the current route guidance information of the hysteroscope, the current travel direction adjustment information of the hysteroscope is generated through the route adjustment network;

[0115] Based on the current travel direction adjustment information of the hysteroscope, a route adjustment guidance mark of the hysteroscope is generated in the current image information of the hysteroscope, and the hysteroscope adjustment guidance mark is used as the current path guidance content of the hysteroscope to the abnormal area.

[0116] Optionally, the device further includes:

[0117] A first recognition module is configured to recognize current regional structural data of the abnormal region through an image recognition network based on current image information of the hysteroscope;

[0118] a second identification module, configured to identify relative position information between the hysteroscope and the abnormal area based on current area structure data of the abnormal area and area abnormality structure data of the abnormal area;

[0119] A marking module is used to generate an entry point adjustment route for the hysteroscope based on the relative position information, the entry point position of the abnormal area, and the abnormal structure model of the abnormal area, and mark and display the entry point adjustment route, the entry point position, and the current area structure data of the abnormal area in the current screen information to obtain entry guidance information for the abnormal area.

[0120] Each module in the above-mentioned hysteroscopic surgery path guidance system can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0121] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, memory, a communication interface, a display screen, and an input system connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a path guidance method for hysteroscopic surgery. The display screen of the computer device can be a liquid crystal display or an electronic ink display. The input system of the computer device can be a touch screen covering the display screen, buttons, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0122] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0123] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of any one of the methods in the first aspect are implemented.

[0124] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the methods in the first aspect are implemented.

[0125] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of any one of the methods of the first aspect when executed by a processor.

[0126] It should be noted that the patient information (including but not limited to patient device information, patient personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the patient or fully authorized by all parties.

[0127] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0128] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A path guidance method for hysteroscopic surgery, characterized in that: The method comprises: Acquiring multimodal scanning data of a target area, and constructing a three-dimensional structural model of the target area based on the multimodal scanning data of the target area; In response to the staff's annotation operation in the three-dimensional structural model, anomaly types of each abnormal area in the target area are collected, and based on the anomaly types and location ranges of each abnormal area, abnormal path guidance information for each abnormal area is generated through a path guidance generation strategy; Collecting current position information of the hysteroscope and current image information of the hysteroscope, and generating current movement information of the hysteroscope in the three-dimensional structure model based on the current position information of the hysteroscope and the current image information of the hysteroscope; Based on the abnormal path guidance information of each abnormal area and the current travel information of the hysteroscope, the current path guidance content of the hysteroscope to each abnormal area is generated in the current screen information of the hysteroscope, and the current path guidance content of the hysteroscope to each abnormal area is displayed to the staff on the screen.

2. The method according to claim 1, characterized in that The constructing of a three-dimensional structural model of the target area based on the multimodal scanning data of the target area includes: Splitting the multimodal scan data into sub-scan data of each scan type, and constructing an initial three-dimensional structural model of the target area corresponding to each scan type based on the sub-scan data of each scan type; Based on the initial three-dimensional structure model corresponding to each scan type, a three-dimensional structure model of the target area is generated through a model fusion strategy.

3. The method according to claim 1, characterized in that The generating of abnormal path guidance information for each abnormal area based on the abnormal type and the location range of each abnormal area by a path guidance generation strategy includes: For each abnormal area, based on the position range of the abnormal area, identifying the spatial position information of the abnormal area in the three-dimensional structural model and the regional abnormal structure data of the abnormal area, and generating a hysteroscopic route for the abnormal area through the three-dimensional structural model based on the spatial position information of the abnormal area; Based on the regional abnormal structure data of the abnormal area, marking the abnormal structure model of the abnormal area in the three-dimensional structure model, and based on the abnormal type of the abnormal area, identifying the entry point position of the abnormal area in the abnormal structure model through an abnormal entry point identification network; The hysteroscopic route of the abnormal area and the entry point position of the abnormal area are used as abnormal path guidance information of the abnormal area.

4. The method according to claim 1, wherein The generating of the current movement information of the hysteroscope in the three-dimensional structure model based on the current position information of the hysteroscope and the current image information of the hysteroscope includes: Based on the current position information of the hysteroscope, the spatial position information of the hysteroscope is located in the three-dimensional structure model, and based on the current image information of the hysteroscope, the uterine wall structure data in the current image information is identified through an image recognition network; Based on the uterine wall structure data, in an area where the spatial position information of the hysteroscope is located in the three-dimensional structure model, adapting the area range corresponding to the uterine wall structure data, and identifying the orientation information of the hysteroscope based on the range position information of the area range in the three-dimensional structure model and the spatial position information of the hysteroscope; Based on the current position information of the hysteroscope and the orientation information of the hysteroscope, the current travel route of the hysteroscope is generated, and the current travel route of the hysteroscope is used as the current travel information of the hysteroscope.

5. The method according to claim 4, characterized in that The generating of the current path guidance content of the hysteroscope to each of the abnormal areas based on the abnormal path guidance information of each of the abnormal areas and the current travel information of the hysteroscope in the current image information of the hysteroscope includes: For each abnormal area, based on the current position information of the hysteroscope, the abnormal path guidance information of the abnormal area is adjusted to obtain the current route guidance information of the hysteroscope, and based on the current travel route of the hysteroscope and the current route guidance information of the hysteroscope, the current travel direction adjustment information of the hysteroscope is generated through the route adjustment network; Based on the current travel direction adjustment information of the hysteroscope, a route adjustment guidance mark of the hysteroscope is generated in the current image information of the hysteroscope, and the hysteroscope adjustment guidance mark is used as the current path guidance content of the hysteroscope to the abnormal area.

6. The method according to claim 1, characterized in that When the abnormal area is included in the current image information of the hysteroscope, the method further includes: Based on the current image information of the hysteroscope, identifying the current regional structure data of the abnormal area through an image recognition network; identifying relative position information between the hysteroscope and the abnormal area based on the current area structure data of the abnormal area and the area abnormality structure data of the abnormal area; Based on the relative position information, the entry point position of the abnormal area, and the abnormal structure model of the abnormal area, the entry point adjustment route of the hysteroscope is generated, and the entry point adjustment route, the entry point position, and the current area structure data of the abnormal area are marked and displayed in the current screen information to obtain the entry guidance information of the abnormal area.

7. A path guidance system for hysteroscopic surgery, characterized in that: The system comprises: an acquisition module, configured to acquire multimodal scanning data of a target area and construct a three-dimensional structural model of the target area based on the multimodal scanning data of the target area; a generation module, configured to collect anomaly types of each abnormal area in the target area in response to a staff member's annotation operation in the three-dimensional structural model, and generate abnormal path guidance information for each abnormal area based on the anomaly type and the location range of each abnormal area using a path guidance generation strategy; an acquisition module, configured to acquire current position information of the hysteroscope and current image information of the hysteroscope, and generate current movement information of the hysteroscope in the three-dimensional structure model based on the current position information of the hysteroscope and the current image information of the hysteroscope; A display module is used to generate the current path guidance content of the hysteroscope to each of the abnormal areas based on the abnormal path guidance information of each of the abnormal areas and the current travel information of the hysteroscope in the current screen information of the hysteroscope, and to display the current path guidance content of the hysteroscope to each of the abnormal areas to the staff on the screen.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images