Operation effect generation method and device based on generative model

Abstract

The invention relates to an operation effect generation method and device based on a generative model. The method comprises the steps of obtaining a target medical image and a natural image of a target object; the target medical image is a multi-modal two-dimensional image registered with the natural image; performing superposition processing on the target medical image and the natural image to obtain a reference image, and outputting and displaying the reference image; obtaining a first text prompt and an operation position image corresponding to the reference image; inputting the operation position image, the first text prompt, the target medical image and the natural image into a conditional image generation network to obtain an operation effect simulation diagram, and outputting and displaying the operation effect simulation diagram; the problem that the effect generated by the operation on the target object cannot be visually simulated is solved, the effect of the operation effect picture is accurately predicted according to information such as text prompt, the operation position and the image of the target object, and therefore an operator is assisted in judging whether the current operation is matched with the target object or not.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for generating operation effects based on a generative model. Background Art

[0002] With the rapid development of modern medical technology, preoperative planning and simulation have become important means to improve surgical accuracy and success rate. Especially in the field of complex surgery, such as surgery, ensuring the accuracy of the surgical incision position is crucial to reducing patient trauma and promoting postoperative recovery. Traditional surgical incision positioning mainly relies on the doctor's experience, observation of preoperative images (such as CT, MRI and X-rays), and the navigation system used during surgery. However, there are still many challenges in the difference between the image and the patient's actual anatomical structure, the dimensional conversion of the image, and the doctor's interpretation of the image data during the operation and the connection between the implementation of the operation.

[0003] In the prior art, some methods have been proposed to more accurately locate the surgical incision. For example, the center point of the brain tumor is calculated by acquiring a three-dimensional model, and the point with the shortest distance on the scalp is located. The shortest point in the location area is located; the edge point of the tumor is obtained; the location of the surgical incision is determined, and the surgical approach is designed. For another example, the target holographic image and CT vascular image of the target object at the current moment are used; the target holographic image is fused based on the image of the target object and the three-dimensional image model. According to the CT vascular image, the vascular spatial distribution information of the target holographic image is fitted. Taking the lesion location point as the foot of the perpendicular, the shortest line segment from the foot of the perpendicular to the surface of the target holographic image is determined to determine the optimal surgical incision location point.

[0004] However, with the above methods, doctors can only predict the effects of the determined surgical location based on their own experience, and cannot intuitively simulate the final effect of the incision or the postoperative suturing situation. Summary of the invention

[0005] Based on this, it is necessary to provide a method and device for generating operation effects based on a generative model that can predict operation effect diagrams in order to address the above technical problems.

[0006] In a first aspect, the present application proposes a method for generating an operation effect based on a generative model, the method comprising:

[0007] Acquire a target medical image and a natural image of a target object; the target medical image is a multimodal two-dimensional image that has been registered with the natural image;

[0008] Performing superposition processing on the target medical image and the natural image to obtain a reference image, and outputting and displaying the reference image;

[0009] Acquire a first text prompt and an operation position image corresponding to the reference image;

[0010] The operation position image, the first text prompt, the target medical image and the natural image are input into a pre-trained image generation network to obtain an operation effect simulation diagram, and the operation effect simulation diagram is output and displayed.

[0011] In one embodiment, obtaining a target medical image and a natural image of a target object includes:

[0012] Acquire original medical image data and natural images of a target object, wherein the original medical image data includes original two-dimensional image data and original three-dimensional image data; project the original three-dimensional image data onto a two-dimensional plane to obtain a projected two-dimensional image; and obtain a plurality of two-dimensional medical images based on the original two-dimensional image data and the projected two-dimensional image;

[0013] Each of the two-dimensional medical images is registered with the natural image to obtain a target medical image.

[0014] In one embodiment, registering each of the two-dimensional medical images with the natural image to obtain a target medical image includes:

[0015] Performing a coarse alignment process based on contour matching on each of the two-dimensional medical images to obtain a coarsely aligned image;

[0016] The displacement fields of each of the coarsely aligned images and the natural image are calculated respectively; and the coarsely aligned images are elastically transformed based on the displacement fields to obtain target medical images corresponding to each of the two-dimensional medical images.

[0017] In one embodiment, performing a coarse alignment process based on contour matching on each of the two-dimensional medical images to obtain a coarsely aligned image includes:

[0018] Performing edge detection on the two-dimensional medical image to obtain a feature descriptor;

[0019] Performing feature matching calculation on the feature descriptor to obtain a target similar contour;

[0020] Based on the target similarity contour, the two-dimensional medical image is rigidly transformed to obtain a coarse alignment image.

[0021] In one embodiment, the target medical image and the natural image are superimposed to obtain a reference image, and the reference image is output for display, including:

[0022] Based on augmented reality, the target medical image is superimposed on the natural image to obtain a first superimposed reference image, and the first superimposed reference image is output and displayed.

[0023] In one embodiment, the target medical image and the natural image are superimposed to obtain a reference image, and the reference image is output for display, including:

[0024] Acquire a second text prompt, input the second text prompt, the target medical image and the natural image into a prompt-based segmentation model to obtain a second superimposed reference image, and output and display the second superimposed reference image;

[0025] The second superimposed reference image includes the segmented anatomical structure.

[0026] In one embodiment, when the content of the first text prompt includes a prompt for generating an incision, the operation effect simulation diagram includes an incision internal simulation diagram;

[0027] When the content of the first text prompt includes a prompt for incision suturing, the operation effect simulation diagram includes an incision suturing simulation diagram.

[0028] In one of the embodiments, the pre-trained image generation network model includes an image prompt adapter;

[0029] The image prompt adapter is used to process image prompts, and the image prompts include the operation position image, the target medical image and the natural image.

[0030] In a second aspect, the present application further provides an operation effect generation device based on a generative model, the device comprising:

[0031] An image acquisition module, used to acquire a target medical image and a natural image of a target object; the target medical image is a multimodal two-dimensional image that has been registered with the natural image;

[0032] An image superposition module, used for superimposing the target medical image and the natural image to obtain a reference image, and outputting and displaying the reference image;

[0033] A simulation effect module is used to obtain a first text prompt and an operation position image corresponding to the reference image; input the operation position image, the first text prompt, the target medical image and the natural image into a pre-trained image generation network to obtain an operation effect simulation image, and output the operation effect simulation image for display.

[0034] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for generating an operation effect based on a generative model as described in the first aspect is implemented.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating an operation effect based on a generative model as described in the first aspect is implemented.

[0036] The above-mentioned operation effect generation method and device based on the generative model obtain a target medical image and a natural image of the target object; the target medical image is a multimodal two-dimensional image that has been aligned with the natural image; the target medical image and the natural image are superimposed to obtain a reference image, and the reference image is output and displayed; a first text prompt and an operation position image corresponding to the reference image are obtained; the operation position image, the first text prompt, the target medical image and the natural image are input into a conditional image generation network to obtain an operation effect simulation image, and the operation effect simulation image is output and displayed; the problem of being unable to intuitively simulate the effect of the operation on the target object is solved, and the effect of the operation effect image can be accurately predicted based on information such as the text prompt, the operation position and the image of the target object, thereby assisting the operator to judge whether the current operation is compatible with the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 A hardware structure block diagram of an operation effect generation method based on a generative model in one embodiment;

[0039] Figure 2 A schematic diagram of a flow chart of a method for generating an operation effect based on a generative model in one embodiment;

[0040] Figure 3 A schematic flow chart of a method for processing raw medical imaging data and natural images in one embodiment;

[0041] Figure 4 It is a flowchart of a method for generating an operation effect based on a generative model in a preferred embodiment;

[0042] Figure 5It is a structural block diagram of an operation effect generating device based on a generative model in another embodiment.

[0043] Figure numerals: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 51, image acquisition module; 52, image overlay module; 53, simulation effect module. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0045] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as the content disclosed in this application is insufficient.

[0046] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0047] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0048] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of the operation effect generation method based on the generative model of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0049] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the operation effect generation method based on the generative model in the present embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0050] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0051] In this embodiment, a method for generating an operation effect based on a generative model is provided. Figure 2 is a flow chart of the method for generating operation effects based on a generative model in this embodiment. Figure 2 As shown, the process includes the following steps:

[0052] Step S210, acquiring a target medical image and a natural image of a target object; the target medical image is a multimodal two-dimensional image that has been registered with the natural image.

[0053] Specifically, the target medical image is processed medical image data, including but not limited to CT image data, MRI image data and X-ray image data, wherein three-dimensional medical image data (such as MRI image data) needs to be projected into a two-dimensional image first. A natural image is a two-dimensional image formed by natural light imaging on the surface of a target object, such as by photographing the target object on the operating table with a camera set on the operating table to obtain a natural image. In order to obtain a better superposition effect in subsequent steps, the target medical images here are all images that have been registered to the natural image.

[0054] Step S220, superimposing the target medical image and the natural image to obtain a reference image, and outputting the reference image for display.

[0055] Specifically, since the target medical image includes medical image data of multiple modalities, in one specific implementation, the image selection information provided by the operator is received, and the target medical image selected in the image selection information is superimposed on the natural image, that is, when performing the superposition processing, the operator can superimpose the target medical image selected as needed, for example, only select one image data of CT image data, or select CT image data, MRI image data and X-ray image data to be superimposed on the natural image. The superposition processing can be performed by means of augmented reality or artificial intelligence models. In one specific implementation, the operator selectively displays one or more reference images on the display device, the operator performs operation planning by observing the reference image, and provides the operation position by drawing an operation line segment on the reference image of the display device. The computer generates an operation position image based on the data of the operation position, and the operation position image is a binary image, the area value of the operation position is 1, and the other area values ​​are 0. In another specific implementation, the reference image is input into the artificial intelligence model for surgical planning, and the operation position image is output. In this embodiment, the operation position image can be obtained based on the reference image in a variety of ways, and the generation method of the operation position image is not limited in this embodiment.

[0056] Step S230, obtaining a first text prompt and an operation position image corresponding to a reference image; inputting the operation position image, the first text prompt, the target medical image, and the natural image into a pre-trained image generation network to obtain an operation effect simulation image, and outputting the operation effect simulation image for display.

[0057] Specifically, the first text prompt is text data input by the operator, or text data obtained based on real-time speech provided by the operator. In this embodiment, the operation position image, the target medical image and the natural image are used as image prompts, and the first text prompt is used as a text prompt. The above image prompts and text prompts are combined through a pre-trained image generation network to obtain an operation effect simulation diagram, such as an incision internal simulation diagram or an incision suture simulation diagram. The pre-trained image generation network can adopt a diffusion model (Stable Diffusion, SD). The SD model can generate a changed image based on the existing image according to the text prompt, such as adding specific content to the image.

[0058] Further, based on the operation effect simulation diagram, effect information is obtained, including but not limited to the distance between the operation position and the lesion position, the difficulty of the operation, the probability of successful operation, etc. For example, the distance between the incision and the lesion position and whether the incision is easy to suture can be analyzed. The operator can also directly observe the operation effect simulation diagram or refer to the effect information to re-provide the operation line segment to update the operation position image, and re-execute step S230 for the updated operation position image to perform effect prediction.

[0059] It should be noted that this embodiment can be applied to the medical field, the operator is a doctor, the target object is a patient, and the natural image is a human body image. However, the method provided in this embodiment is not a disease diagnosis and treatment method, and this embodiment does not involve a process of identifying, determining or eliminating the cause or lesion with a living human or animal body as the direct implementation object. The image of the target object used in this example (i.e., the target medical image and the natural image or the original medical image data used to generate the target medical image) includes but is not limited to being downloaded from a database or a network platform pre-stored, and can also be generated according to demand. The corresponding image data, etc., are not limited to the method of obtaining the image of the target object in this embodiment. The operation effect simulation diagram obtained in this embodiment is used as intermediate data for reference by the operator, and is not used as a conclusion for identifying, determining or eliminating the cause or lesion.

[0060] This embodiment obtains a target medical image and a natural image of a target object; the target medical image is a multimodal two-dimensional image that has been registered with the natural image; the target medical image and the natural image are superimposed to obtain a reference image, and the reference image is output for display; a first text prompt and an operation position image corresponding to the reference image are obtained; the operation position image, the first text prompt, the target medical image, and the natural image are input into a conditional image generation network to obtain an operation effect simulation image; this solves the problem of being unable to intuitively simulate the effect of the operation on the target object, and achieves accurate prediction of the effect of the operation effect image based on information such as the text prompt, the operation position, and the image of the target object, thereby assisting the operator in judging whether the current operation is compatible with the target object.

[0061] In one embodiment, if Figure 3 As shown, a target medical image and a natural image of a target object are obtained, including:

[0062] Step S310, obtaining original medical image data and natural images of the target object, the original medical image data including original two-dimensional image data and original three-dimensional image data; projecting the original three-dimensional image data onto a two-dimensional plane to obtain a projected two-dimensional image; and obtaining multiple two-dimensional medical images based on the original two-dimensional image data and the projected two-dimensional image.

[0063] Specifically, the original medical image data includes original CT image data, original MRI image data and original X-ray image data, wherein the original CT image data and the original MRI image data belong to original three-dimensional image data, and the original X-ray image data belongs to original two-dimensional image data. Before projection, the original medical image data can be preprocessed, such as denoising, contrast enhancement, artifact correction, etc., to improve image quality and help improve alignment accuracy. The preprocessed three-dimensional image data is reconstructed and projected. For example, a region of interest in CT and MRI is selected, and the points in the region of interest are converted from the world coordinate system to the camera coordinate system. The transformation formula is as follows: X c =RX w +t, where X w Represents a point in the world coordinate system, X c Represents a point in the camera coordinate system. R is a 3×3 rotation matrix, representing the rotation of the camera, and t is a 3×1 translation vector. After perspective projection of all three-dimensional points in the camera coordinate system, an image is generated on a two-dimensional plane to obtain a two-dimensional medical image corresponding to CT and MRI.

[0064] Step S320 , registering each two-dimensional medical image with the natural image to obtain a target medical image.

[0065] In this embodiment, a multimodal medical image is provided, and the three-dimensional image is converted into an image on a two-dimensional plane. Compared with the multimodal registration of three-dimensional images, the registration using two-dimensional multimodal images has a greater speed advantage and higher accuracy.

[0066] In one embodiment, registering each two-dimensional medical image with a natural image to obtain a target medical image includes:

[0067] Step S321 , performing rough alignment processing based on contour matching on each two-dimensional medical image to obtain a rough aligned image.

[0068] Step S321 , respectively calculating the displacement fields of each coarsely aligned image and the natural image; elastically transforming the coarsely aligned image based on the displacement field to obtain a target medical image corresponding to each two-dimensional medical image.

[0069] Specifically, the position deviation between images is preliminarily reduced through coarse alignment processing. Based on the coarse alignment, a deep neural network is used for elastic registration to achieve higher precision alignment.

[0070] Specifically, the input of the deep neural network is two images A and B to be registered, and the output of the deep neural network is the displacement field d(x,y)=(u(x,y),v(x,y)) of each pixel of B in the x and y directions relative to A, where u(x,y) and v(x,y) represent the displacement of each pixel in the image in the x and y directions respectively. The displacement field is a two-dimensional vector corresponding to each pixel, indicating the displacement of the point in two directions. The displacement field is added to the original image pixels (i.e., each coarsely aligned image) to obtain the image after elastic transformation. This enables the registration of CT, MRI, and X-ray images to natural images of the human body.

[0071] In one embodiment, performing a coarse alignment process based on contour matching on each two-dimensional medical image to obtain a coarsely aligned image includes:

[0072] Step S321a, performing edge detection on the two-dimensional medical image to obtain a feature descriptor.

[0073] Step S321b, performing feature matching calculation on the feature descriptor to obtain a target similar contour.

[0074] Step S321c, based on the target similarity contour, rigid body transformation is performed on the two-dimensional medical image to obtain a coarse alignment image.

[0075] Specifically, contour matching is used to automatically perform rough alignment. For two-dimensional images, edge detection is used to extract contours, and then feature descriptors (such as Fourier descriptors and shape contexts) are generated for each contour. Feature matching algorithms (such as nearest neighbor matching) are then used to find the most similar contour parts between different images. A rigid body transformation model is used to calculate transformation parameters, and then a rigid body transformation is performed to obtain coarsely aligned CT, MRI, and X-ray images.

[0076] In one embodiment, the target medical image and the natural image are superimposed to obtain a reference image, and the reference image is output and displayed, including:

[0077] Step S410: Based on augmented reality, the target medical image is superimposed on the natural image to obtain a first superimposed reference image, and the first superimposed reference image is output for display.

[0078] Specifically, images can be selectively superimposed in a display device. Images registered using multiple modalities of CT, X-ray, MRI, and natural images can display more reliable information, wherein the number of imaging modalities can be increased or decreased.

[0079] In one embodiment, the target medical image and the natural image are superimposed to obtain a reference image, and the reference image is output and displayed, including:

[0080] Step S420, obtain a second text prompt, input the second text prompt, the target medical image and the natural image into the prompt-based segmentation model, obtain a second overlay reference image, and output the second overlay reference image for display; wherein the second overlay reference image includes the segmented anatomical structure.

[0081] Specifically, the operator can obtain a second text prompt through real-time voice, for example, the voice is "such as segmenting out blood vessels and bones". The prompt-based segmentation model can adopt the medical industry SAM segmentation model based on supporting multimodal prompts, or its model that can segment anatomical structures. The specific implementation means are not limited in this embodiment. At this time, the segmentation map of blood vessels, bones, etc. superimposed on the surface of the human body can be seen on the display device. Compared with the method in which the operator can only look at the original image to determine the operation position, this embodiment adds a medical segmentation model based on text prompts, which can display the bones, blood vessels, lesion areas and other information that need to be paid attention to according to the prompts input by the operator, so as to fully display the anatomical structure of the target object.

[0082] In one embodiment, when the content of the first text prompt includes a prompt for generating an incision, the operation effect simulation diagram includes an incision internal simulation diagram; when the content of the first text prompt includes a prompt for incision suturing, the operation effect simulation diagram includes an incision suturing simulation diagram.

[0083] Specifically, a pre-trained generative network model is used to generate a simulated image based on the doctor's voice prompt information (prompt for generating an incision or prompt for incision suturing) and the simulated drawn incision line segments and the registered image information and other conditional information, and the preview effect of the incision and the preview effect of the incision suturing are viewed in real time. The doctor can adjust the position of the drawn line segment according to the display effect to adjust the generated effect to assist the doctor in the incision positioning selection. It should be noted that this embodiment is not a disease diagnosis and treatment method. This embodiment does not involve the process of identifying, determining or eliminating the cause or lesion with a living human or animal body as the direct implementation object, but obtains an incision internal simulation diagram or an incision suturing simulation diagram as intermediate data for the doctor's reference. In addition, this embodiment can also be used in teaching to show students the surgical effects under different operations.

[0084] In one embodiment, the pre-trained image generation network model includes an image prompt adapter; the image prompt adapter is used to process image prompts, and the image prompts include operation position images, target medical images, and natural images.

[0085] Specifically, the difference between the pre-trained image generation network and the ordinary stable diffusion network is that the pre-trained image generation network can use image prompts as generation conditions. On the one hand, it relies on the image prompt adapter. On the other hand, when training the pre-trained image generation network model, the trainable text data and the trainable image data are used as training data sets and collected into the image generation network to be trained. The image generation network model that can view the operation preview effect in real time is trained.

[0086] The present embodiment is described and illustrated below through preferred embodiments.

[0087] Figure 4 Flowchart of the method for generating operation effects based on a generative model in this preferred embodiment. Figure 4 As shown, the process steps of this preferred embodiment are as follows:

[0088] S1: Acquire original medical image data of the part to be incised, the original medical image data including three-dimensional CT images and MRI images, and two-dimensional X-ray images, and perform common preprocessing such as denoising, contrast enhancement, and artifact correction on the original medical image data.

[0089] S2: Reconstruct and project the 3D data, and perform multimodal registration of the projected 2D image with the 2D X-ray image with the natural image as the target: Reconstruct and project the preprocessed CT image and MRI image respectively. Select the region of interest in CT and MRI, and transform the points in the region of interest from the world coordinate system to the camera coordinate system. This transformation is achieved by defining a camera model, which includes the camera position, orientation and internal parameters (focal length, principal point and other camera parameters). After perspective projection of all 3D points in the camera coordinate system, a 2D image of the CT image and the MRI image is generated on the 2D plane. Roughly align the processed 2D images. For 2D images, edge detection is used to extract contours, and then feature descriptors (such as Fourier descriptors and shape contexts) are generated for each contour, and then feature matching algorithms (such as nearest neighbor matching) are used to find the most similar contour parts between different 2D images. A rigid body transformation model is used to calculate the transformation parameters, and then a rigid body transformation is performed to obtain the coarsely aligned CT, MRI, and X-ray images. The coarsely aligned CT, MRI, and X-ray images are registered to the natural image of the human body. Use deep neural networks for elastic registration. The input of the deep neural network is two images to be registered, A and B. The output of the network is the displacement field of B relative to each pixel of A in the x and y directions. The displacement field is a two-dimensional vector corresponding to each pixel, indicating the displacement of the point in two directions. Add the displacement field to the original image pixels (CT, MRI, and X-ray images after rough alignment) to obtain the CT, MRI, and X-ray images after elastic transformation, that is, the registered medical imaging data.

[0090] S3: The registered image is superimposed on the natural image of the human body through augmented reality (AR), and the image can be selectively superimposed on the display device.

[0091] S4: Doctors can use voice to get text prompts (such as segmenting blood vessels and bones), and input a large segmentation model based on the prompts (such as the medical industry SAM segmentation model that supports multimodal prompts, etc.). At this time, the segmentation map of blood vessels, bones, etc. superimposed on the surface of the human body can be seen on the display device.

[0092] S5: The doctor draws a line segment on the display device to simulate the incision position, and converts the drawn line segment into a line segment simulation mask image (0-1 image, the drawn line segment area value is 1). A text prompt is obtained through voice (generating an internal incision simulation image). The converted line segment simulation mask image, the natural image of the human body, and the registered medical imaging data are input into the stable diffusion image generation network to generate an internal incision simulation image of the drawn line segment position, and the internal incision simulation image is output. Through the internal incision simulation image, it can be seen whether the incision is closest to the lesion position.

[0093] S6: Get text prompts through voice (generate incision suture diagram), and replace the text prompts to generate an incision suture simulation diagram of the drawn line segment position, and output the incision suture simulation diagram. Through the incision internal simulation diagram, it can be seen whether the incision is easy to suture.

[0094] S7: The doctor can adjust the position of the drawn line segment according to the display effect (inside incision simulation image / incision suture simulation image), and repeat the above steps S5 or S6 to adjust the generated effect to assist the doctor in incision positioning selection.

[0095] In this preferred embodiment, during the operation, it is possible to use generative network models such as Stablediffusion to generate simulated images in real time according to the doctor's voice prompt information, simulated incision line segments drawn, and registered image information, and other conditional information, so as to view the preview effect of the incision and the preview effect of the incision suture in real time. This embodiment adds a large medical segmentation model based on text prompts, which can display information such as bones, blood vessels, and lesion areas that need to be paid attention to according to the doctor's voice. Compared with the multimodal registration of three-dimensional images, the registration using two-dimensional multimodal images has a speed advantage and higher accuracy. Compared with many solutions that only use a single modality image to determine the incision position, this solution uses CT, X-ray, MRI, and natural image multimodal registration methods to display more reliable information. The number of image modalities can be increased or decreased.

[0096] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0097] Based on the same inventive concept, in this embodiment, a device for generating an operation effect based on a generative model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0098] In one embodiment, if Figure 5 As shown, an operation effect generation device based on a generative model is provided, comprising: an image acquisition module 51, an image overlay module 52 and a simulation effect module 53, wherein:

[0099] The image acquisition module 51 is used to acquire a target medical image and a natural image of a target object; the target medical image is a multimodal two-dimensional image that has been registered with the natural image.

[0100] The image superposition module 52 is used to superimpose the target medical image and the natural image to obtain a reference image, and output the reference image for display.

[0101] The simulation effect module 53 is used to obtain the first text prompt and the operation position image corresponding to the reference image; input the operation position image, the first text prompt, the target medical image and the natural image into the pre-trained image generation network to obtain the operation effect simulation image, and output the operation effect simulation image for display.

[0102] In one embodiment, the image acquisition module 51 is also used to acquire original medical image data and natural images of the target object, the original medical image data including original two-dimensional image data and original three-dimensional image data; project the original three-dimensional image data onto a two-dimensional plane to obtain a projected two-dimensional image; obtain multiple two-dimensional medical images based on the original two-dimensional image data and the projected two-dimensional image; and align each two-dimensional medical image with the natural image to obtain a target medical image.

[0103] In one of the embodiments, the image acquisition module 51 is also used to perform coarse alignment processing based on contour matching on each two-dimensional medical image to obtain a coarsely aligned image; calculate the displacement field between each coarsely aligned image and the natural image respectively; and perform elastic transformation on the coarsely aligned image based on the displacement field to obtain a target medical image corresponding to each two-dimensional medical image.

[0104] In one embodiment, the image acquisition module 51 is also used to perform edge detection on the two-dimensional medical image to obtain a feature descriptor; perform feature matching calculation on the feature descriptor to obtain a target similar contour; and perform a rigid body transformation on the two-dimensional medical image based on the target similar contour to obtain a coarsely aligned image.

[0105] In one of the embodiments, the image overlay module 52 is further used to overlay the target medical image onto the natural image based on augmented reality to obtain a first overlay reference image, and output the first overlay reference image for display.

[0106] In one embodiment, the image overlay module 52 is used to obtain a second text prompt, input the second text prompt, the target medical image and the natural image into a prompt-based segmentation model, obtain a second overlay reference image, and output and display the second overlay reference image; wherein the second overlay reference image includes the segmented anatomical structure.

[0107] In one embodiment, when the content of the first text prompt includes a prompt for generating an incision, the operation effect simulation diagram includes an incision internal simulation diagram; when the content of the first text prompt includes a prompt for incision suturing, the operation effect simulation diagram includes an incision suturing simulation diagram.

[0108] In one embodiment, the pre-trained image generation network model includes an image prompt adapter; the image prompt adapter is used to process image prompts, and the image prompts include operation position images, target medical images, and natural images.

[0109] Each module in the above-mentioned operation effect generation device based on the generative model can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0110] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method for generating an operation effect based on a generative model in any of the above embodiments are implemented.

[0111] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating an operation effect based on a generative model in any of the above embodiments are implemented.

[0112] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the method for generating an operation effect based on a generative model in any of the above embodiments.

[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0115] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating operation effects based on a generative model, characterized in that: The method comprises: Acquire a target medical image and a natural image of a target object; the target medical image is a multimodal two-dimensional image that has been registered with the natural image; Performing superposition processing on the target medical image and the natural image to obtain a reference image, and outputting and displaying the reference image; Obtain a first text prompt and an operation position image corresponding to the reference image; input the operation position image, the first text prompt, the target medical image and the natural image into a pre-trained image generation network to obtain an operation effect simulation image, and output the operation effect simulation image for display.

2. The method for generating operation effects based on a generative model according to claim 1, characterized in that: Obtain target medical images and natural images of target objects, including: Acquire original medical image data and natural images of a target object, wherein the original medical image data includes original two-dimensional image data and original three-dimensional image data; project the original three-dimensional image data onto a two-dimensional plane to obtain a projected two-dimensional image; and obtain a plurality of two-dimensional medical images based on the original two-dimensional image data and the projected two-dimensional image; Each of the two-dimensional medical images is registered with the natural image to obtain a target medical image.

3. The method for generating operation effects based on a generative model according to claim 2, characterized in that: Registering each of the two-dimensional medical images with the natural image to obtain a target medical image includes: Performing a coarse alignment process based on contour matching on each of the two-dimensional medical images to obtain a coarsely aligned image; The displacement fields of each of the coarsely aligned images and the natural image are calculated respectively; and the coarsely aligned images are elastically transformed based on the displacement fields to obtain target medical images corresponding to each of the two-dimensional medical images.

4. The method for generating operation effects based on a generative model according to claim 3, characterized in that: Performing a coarse alignment process based on contour matching on each of the two-dimensional medical images to obtain a coarsely aligned image includes: Performing edge detection on the two-dimensional medical image to obtain a feature descriptor; Performing feature matching calculation on the feature descriptor to obtain a target similar contour; Based on the target similarity contour, the two-dimensional medical image is rigidly transformed to obtain a coarse alignment image.

5. The method for generating operation effects based on a generative model according to claim 1, characterized in that: The target medical image and the natural image are superimposed to obtain a reference image, and the reference image is output and displayed, including: Based on augmented reality, the target medical image is superimposed on the natural image to obtain a first superimposed reference image, and the first superimposed reference image is output and displayed.

6. The method for generating operation effects based on a generative model according to claim 1, characterized in that: The target medical image and the natural image are superimposed to obtain a reference image, and the reference image is output and displayed, including: Acquire a second text prompt, input the second text prompt, the target medical image and the natural image into a prompt-based segmentation model to obtain a second superimposed reference image, and output and display the second superimposed reference image; The second superimposed reference image includes the segmented anatomical structure.

7. The method for generating operation effects based on a generative model according to claim 1, characterized in that: When the content of the first text prompt includes a prompt for generating an incision, the operation effect simulation diagram includes an incision internal simulation diagram; When the content of the first text prompt includes a prompt for incision suturing, the operation effect simulation diagram includes an incision suturing simulation diagram.

8. The method for generating operation effects based on a generative model according to claim 1, characterized in that: The pre-trained image generation network model includes an image prompt adapter; The image prompt adapter is used to process image prompts, and the image prompts include the operation position image, the target medical image and the natural image.

9. An operation effect generation device based on a generative model, characterized in that: The device comprises: An image acquisition module, used to acquire a target medical image and a natural image of a target object; the target medical image is a multimodal two-dimensional image that has been registered with the natural image; An image superposition module, used for superimposing the target medical image and the natural image to obtain a reference image, and outputting and displaying the reference image; A simulation effect module is used to obtain a first text prompt and an operation position image corresponding to the reference image; input the operation position image, the first text prompt, the target medical image and the natural image into a pre-trained image generation network to obtain an operation effect simulation image, and output the operation effect simulation image for display.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images