Image registration method, apparatus, device, medium, and program product

Abstract

The application discloses an image registration method, device, equipment, medium and program product, and belongs to the field of image processing. The method comprises the following steps: acquiring a first medical image of a lesion tissue at a first time, wherein the first medical image has a lesion contour of the lesion tissue; displaying the first medical image with a first biopsy grid, wherein the first biopsy grid comprises at least two sub-grids covering the lesion contour; acquiring a second medical image of the lesion tissue at a second time, wherein the second medical image is different from at least part of the lesion contour in the first medical image; and displaying the second medical image with a second biopsy grid, wherein the second biopsy grid has at least two sub-grids corresponding to the first biopsy grid, and the same sub-grid covers the same pathological biopsy area in the two medical images. The above scheme can realize positioning of a lesion residue, and can also assist doctors in secondary resection and secondary pathological biopsy, thereby assisting in improving the accuracy of lesion resection and the efficiency of surgery.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to an image registration method, apparatus, device, medium, and program product. Background Technology

[0002] Mohs micrographic surgery (NMS) is a common surgical resection method. In Mohs surgery, the visible tumor is first removed. Then, a Mohs grid is drawn at the base and boundaries of the tumor, and tissue samples are taken. These samples are prepared into pathological sections and examined under a microscope for any remaining tumor. If tumor remnants are found, the corresponding sampling location is located, and a second resection is performed. This process is repeated until no tumor remnants remain.

[0003] However, the process of preparing tissue samples into pathological sections and performing rapid pathology is lengthy, and the tumor area inevitably undergoes some changes. When residual tumor is detected in a pathological section, it is difficult to relocate the sampling area of ​​that pathological section within the changed tumor area. Summary of the Invention

[0004] This application provides an image registration method, apparatus, device, medium, and program product. The technical solution is as follows:

[0005] According to one aspect of this application, an image registration method is provided, the method comprising:

[0006] Acquire a first medical image of the lesion tissue at a first moment, the first medical image having the lesion outline of the lesion tissue;

[0007] The first medical image is displayed having a first sampling grid, the first sampling grid including at least two sub-grids covering the outline of the lesion;

[0008] A second medical image of the lesion tissue is acquired at a second time point, wherein at least a portion of the lesion contour in the second medical image is different from that in the first medical image;

[0009] The second medical image is displayed having a second sampling grid, the second sampling grid having at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covering the same pathological sampling area in both medical images.

[0010] According to another aspect of this application, an image registration apparatus is provided, the apparatus comprising:

[0011] The acquisition module is used to acquire a first medical image of the lesion tissue at a first moment, wherein the first medical image has the lesion outline of the lesion tissue;

[0012] A display module is configured to display the first medical image having a first sampling grid, the first sampling grid including at least two sub-grids covering the outline of the lesion;

[0013] The acquisition module is used to acquire a second medical image of the lesion tissue at a second time, wherein the second medical image is different from at least a portion of the lesion outline in the first medical image;

[0014] The display module is used to display the second medical image having a second sampling grid, the second sampling grid having at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covering the same pathological sampling area in the two medical images.

[0015] According to another aspect of this application, a computer device is provided, the computer device comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the image registration method as described above.

[0016] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement the image registration method as described above.

[0017] According to another aspect of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, wherein a processor retrieves the computer instructions from the computer-readable storage medium, causing the processor to load and execute them to implement the image registration method as described above.

[0018] The beneficial effects of the technical solutions provided in this application include at least the following:

[0019] By acquiring a first medical image of the lesion tissue at a first time, the first medical image shows the lesion outline of the lesion tissue; and displaying a first medical image with a first sampling grid, the first sampling grid including at least two sub-grids covering the lesion outline. Accordingly, a first sampling grid can be automatically generated and displayed based on the first medical image to assist the doctor in pathological sampling. Then, by acquiring a second medical image of the lesion tissue at a second time, the second medical image shows at least a portion of the lesion outline different from that in the first medical image; and displaying a second medical image with a second sampling grid, the second sampling grid having at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covering the same pathological sampling area in both medical images. Accordingly, by repositioning the first sampling grid to determine the second sampling grid, the doctor can accurately reposition a sampling location in the corresponding position of the second medical image, achieving the location of lesion residues, and can also assist the doctor in secondary resection and secondary pathological sampling, thus improving the accuracy of lesion resection and surgical efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A block diagram of a computer system provided in an exemplary embodiment is shown;

[0022] Figure 2 This diagram illustrates an application scenario of an image registration method provided by an exemplary embodiment.

[0023] Figure 3 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0024] Figure 4 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0025] Figure 5 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0026] Figure 6 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0027] Figure 7 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0028] Figure 8 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0029] Figure 9 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0030] Figure 10 A schematic diagram of an image registration method provided by an exemplary embodiment is shown;

[0031] Figure 11 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0032] Figure 12 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0033] Figure 13 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0034] Figure 14 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0035] Figure 15 A schematic diagram of an image registration method provided by an exemplary embodiment is shown;

[0036] Figure 16 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0037] Figure 17 A schematic diagram of an image registration method provided by an exemplary embodiment is shown;

[0038] Figure 18 A flowchart of an image registration method provided by an exemplary embodiment is shown;

[0039] Figure 19 A structural block diagram of an image registration apparatus provided in an exemplary embodiment is shown;

[0040] Figure 20 A structural block diagram of a computer device provided in an exemplary embodiment is shown. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0042] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0043] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0044] It should be understood that although the terms first, second, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0045] Figure 1 A structural block diagram of a computer system provided in an exemplary embodiment of this application is shown. This computer system can implement a system architecture for an image registration method. The computer system includes a terminal 120 and a server 140.

[0046] Terminal 120 can be an electronic device such as a mobile phone, tablet computer, vehicle-mounted terminal (vehicle system), wearable device, PC (Personal Computer), or unmanned reservation terminal. A client application for the target application can be installed and run on terminal 120. This target application can be a specialized application for image processing of medical images, or other applications that provide image processing functions for medical images; this application does not limit the specific application. Furthermore, this application does not limit the form of the target application, including but not limited to Apps (Applications), applets, etc., installed on terminal 120, and can also be in web page form.

[0047] Server 140 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud servers, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Server 140 can be a backend server for the aforementioned target application, used to provide backend services to the clients of the target application.

[0048] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0049] In some embodiments, the server 140 can also be implemented as a node in a blockchain system. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0050] Terminal 120 and server 140 can communicate via a network, such as a wired or wireless network.

[0051] The image registration method provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 1Taking the implementation environment of the scheme shown as an example, the image registration method can be executed by the terminal 120, such as by the client of the target application installed and running in the terminal 120, or by the server 140, or by the interaction and cooperation between the terminal 120 and the server 140. This application does not limit this.

[0052] Those skilled in the art will understand that the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more. This application does not limit the number of terminals or the type of device.

[0053] The embodiments of this application relate to artificial intelligence and computer vision technologies.

[0054] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0055] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0056] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0057] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0058] The image registration method of this application embodiment can serve as a medical aid, assisting doctors in taking pathological samples from lesions. This includes, but is not limited to, applications in various medical and healthcare-related fields such as Mohs Micrographic Surgery (NMS), biopsy, aspiration biopsy, and puncture biopsy pathological testing. It can be used to display a first medical image with a first sampling grid, the first sampling grid including at least two sub-grids covering the lesion outline, and to display a second medical image with a second sampling grid, the second sampling grid having at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covering the same pathological sampling area in both medical images. This assists doctors in accurately locating the location requiring a second pathological sampling after a first sampling, thus assisting in performing a second pathological sampling.

[0059] For example, Figure 2 The illustration shows an application scenario diagram of the image registration method provided in an exemplary embodiment of this application. The application scenario mainly involves a doctor 200, a patient 201, a shooting component 202, a projection component 203, a terminal 206, and a (back-end) server 205.

[0060] Specifically, during the Morse code procedure performed on patient 201 by doctor 200, the imaging component 202 can capture images of the lesion tissue of patient 201 in the first instance, obtaining a first medical image 204-1, which shows the lesion outline of the lesion tissue. Server 205 acquires the first medical image 204-1 and generates a first medical image 207-1 with a first sampling grid, wherein the first sampling grid includes at least two sub-grids covering the lesion outline. Then, server 205 sends the first medical image 207-1 with the first sampling grid to terminal 206, where it is displayed. Alternatively, the first sampling grid can be projected onto the pathological sampling area of ​​patient 201 by projection component 203, where it is displayed. Therefore, the physician 200 can refer to the first medical image 207-1 with the first sampling grid to perform pathological sampling on the basal and boundary portions of the lesion tissue, quickly prepare paraffin slides from the sampled tissue, and examine the sampled tissue under a microscope to see if there are any residual lesions. If residual lesions are present, a second excision and a second pathological sampling are performed at the corresponding location.

[0061] Next, the imaging component 202 captures images of the lesion tissue of the patient 201 at a second time, obtaining a second medical image 204-2. Due to possible differences in the position of the imaging component 202, changes in the lesion tissue over time, or movement of the patient 201, the outline of at least a portion of the lesion in the second medical image 204-2 differs from that in the first medical image 204-1. The server 205 acquires the second medical image 204-2 and generates a second medical image 207-2 with a second sampling grid. The second sampling grid has at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covers the same pathological sampling area in both medical images. Then, the server 205 sends the second medical image 207-2 with the second sampling grid to the terminal 206, where it is displayed. Alternatively, the second sampling grid can be projected onto the pathological sampling area of ​​the patient 201 using the projection component 203, displaying the second sampling grid on the pathological sampling area. Thus, the physician 200 can refer to the second medical image 207-2 with the second sampling grid to accurately relocate the location of residual lesions in the pathological sampling area, so as to perform secondary resection and secondary pathological sampling.

[0062] Optionally, the shooting component 202 is a device with image acquisition function, which mainly includes a lens and an autofocus component, and has an autofocus function, including at least one of the following: the shooting component 202 has its own autofocus function (such as a mechanical component inside the camera moving the image sensor up and down to focus) and is equipped with a normal lens without autofocus function; the shooting component 202 uses an adapter to drive an autofocus lens with an autofocus component; the shooting component 202 has its own focusing lens; the shooting component 202 is a normal camera and is equipped with a normal lens, and focuses by adding an external liquid lens.

[0063] Optionally, the projection component 203 is a device with screen projection function, used to project the first sampling grid and / or the second sampling grid onto the pathological sampling area of ​​the patient 201, and can be at least one of the following: a digital projector, an LCD projector, or a laser projector.

[0064] Common surgical resection methods include Wide Local Excision (WLE) and Mohs Micrographic Surgery (NMS). WLE involves removing an additional portion of the surrounding tissue during tumor removal to prevent tumor residue. However, for skin tumors with significant infiltration and spread, NMS offers a significant advantage because the tumor's boundaries are difficult to precisely define. The NMS procedure can be summarized as follows: After the initial tumor resection, samples are taken from the basal and border areas of the tumor. These samples are rapidly paraffin-embedded and prepared into pathological sections. Microscopic pathological examination is used to determine if any tumor residue remains. If residue is found, a secondary resection is performed at the corresponding sampling location, and sampling and pathological examination are repeated until no tumor residue remains. Compared to WLE, NMS minimizes damage to healthy tissue and reduces the tumor recurrence rate. Generally, during sampling, the lesion tissue after tumor resection is divided into different sampling grids, and each sub-grid within the sampling grid is numbered. For rapid pathological testing, the detection and processing time still requires 4 to 6 hours, during which time the lesion tissue will inevitably undergo certain changes.

[0065] Based on this, taking its application in the Mozart procedure as an example, Figure 3 A flowchart illustrating an exemplary embodiment of the image registration method provided in this application is shown, showing how the method is applied to... Figure 1 The method, illustrated by an example of a terminal 120 or a client or server 140 installed on the terminal 120 that supports image registration, includes:

[0066] Step 320: Obtain a first medical image of the lesion tissue at the first moment. The first medical image has the lesion outline of the lesion tissue.

[0067] Lesion tissue refers to tissue containing pathogenic microorganisms. Any tissue or organ can become a lesion, and lesion tissue can be tumor tissue. For example, a lesion outline refers to a continuous contour curve of the lesion tissue. The lesion outline can be a regular shape or an irregular shape. The first medical image refers to the image taken immediately after the lesion tissue has been removed using the Mohr's procedure.

[0068] For example, after removing the lesion tissue in a Mozart procedure, samples can be taken from the basal and boundary portions of the lesion tissue. These samples are then rapidly paraffin-embedded and prepared into pathological sections for microscopic pathological examination to determine if any lesions remain. This process is also known as rapid pathology. "First time" refers to any point in time after the lesion tissue is removed but before rapid pathology is performed.

[0069] In some embodiments, the imaging component captures images of the lesion tissue at a specific time to obtain a first medical image of the lesion tissue at that specific time.

[0070] Optionally, the imaging component can be a first imaging component, including at least one of an industrial camera, a color (RGB) camera, and a two-dimensional (2D) camera. The imaging component then captures an image of the lesion tissue in a timely manner, resulting in a first medical image that is a two-dimensional planar image containing two-dimensional information about the lesion tissue.

[0071] Optionally, the imaging component can be a second imaging component, including at least one of a depth camera, a stereo color (RGBD) camera, a point cloud camera, and a three-dimensional (3D) camera. The imaging component then captures images of the lesion tissue in a first-time event, resulting in a first medical image that is a two-dimensional planar image containing two-dimensional information of the lesion tissue and / or a three-dimensional point cloud image containing three-dimensional information of the lesion tissue.

[0072] Optionally, the type of imaging component can be determined based on the location of the lesion tissue.

[0073] For example, when the lesion is located in a relatively flat area or with minimal curvature, such as on the limbs or trunk, a first imaging component that is geared towards acquiring two-dimensional planar images can be selected. When the lesion is located in an uneven area or with significant curvature, such as on the nose, head, or joints, a second imaging component that is geared towards acquiring three-dimensional point cloud images can be selected to further incorporate the three-dimensional information of the lesion and improve the accuracy of subsequent medical image processing.

[0074] Step 340: Display a first medical image with a first sampling grid, the first sampling grid including at least two sub-grids covering the outline of the lesion.

[0075] The first sampling grid, determined based on the first medical image, is used to divide the pathological sampling area corresponding to the lesion outline in the first medical image into a grid. This first sampling grid can divide the pathological sampling area into multiple smaller sampling areas. At least two sub-grids in the first sampling grid can be used to assist in locating the specific sampling position of the pathological sampling area, assisting the doctor in subsequent pathological sampling. The pathological sampling area is the area where tissue is sampled after lesion resection, and the sampled tissue is quickly paraffin-embedded to make pathological sections for pathological examination.

[0076] Optionally, the pathological sampling area can be the region corresponding to the lesion outline of the lesion tissue in the first medical image. In this case, the pathological sampling area is the region where the lesion tissue is located, and the pathological sampling area only contains the lesion tissue. Alternatively, the pathological sampling area can be the region defined by extending the lesion outline of the lesion tissue in the first medical image. In this case, the pathological sampling area includes the lesion tissue and at least a portion of healthy tissue.

[0077] Optionally, the first sampling grid includes at least two sub-grids covering the outline of the lesion, that is, the first sampling grid includes at least two sub-grids covering the pathological sampling area. The at least two sub-grids are arranged in an array, meaning they are arranged regularly in rows and columns.

[0078] Optionally, each subgrid in the first sampling grid covers a pathological sampling area containing lesion tissue. That is, even if the lesion tissue is irregular, the first sampling grid is also irregular. Accordingly, in subsequent pathological sampling based on the first sampling grid, lesion tissue can be sampled from each subgrid, which can help improve the efficiency of pathological sampling.

[0079] Optionally, at least two subgrids in the first sampling grid have the same subgrid parameters, which include at least one of the following: subgrid size, shape, area, and interior angle size.

[0080] For example, at least two subgrids in the first sampling grid can be one of the following: triangle, square, matrix, parallelogram, rhombus, or sector. The specific shape can be automatically determined based on the lesion type of the tissue, or it can be specified by a doctor based on clinical experience. For example, when the subgrid is a square, the side length of the square can be set to 20mm-30mm. When the subgrid is not a square or rectangle, the size of the interior angle of each interior angle of the subgrid must be specified.

[0081] In some embodiments, the subgrid parameters of at least a portion of the subgrids in the first sampling grid are different from the subgrid parameters of other subgrids, that is, the subgrid parameters of each subgrid in the first sampling grid are not completely identical. For example, if there is a portion of subgrids in the first sampling grid that covers very little lesion tissue, then this portion of subgrids can be merged with other subgrids. After merging, the subgrid parameters of at least a portion of the subgrids in the first sampling grid are different from the subgrid parameters of other subgrids.

[0082] In some embodiments, the method further includes: identifying a pathological sampling region corresponding to the lesion contour in the first medical image, and generating a first sampling grid for dividing the pathological sampling region into grids.

[0083] Optionally, a first sampling grid is generated based on at least one reference point in the first medical image for meshing the pathological sampling area.

[0084] For example, a reference point is a pixel used as a reference during the generation of the first sampling mesh, and the reference point includes at least one. Optionally, the at least one reference point may be predetermined before generating the first sampling mesh, or it may be determined iteratively in real time during the generation of the first sampling mesh.

[0085] Optionally, if there is only one reference point, the reference point can be the center of the entire first sampling grid, and an expansion grid can be generated centered on the reference point to divide the pathological sampling area into grids.

[0086] For example, when there is only one reference point, based on this reference point in the first medical image, the first, second, and nth rings of grids are sequentially expanded around the reference point, where n is greater than 0 and is an integer. The i-th ring of grids does not overlap with the (i+1)-th ring of grids, where i is greater than 0 and less than or equal to n and is an integer.

[0087] Optionally, when there are multiple reference points, each reference point can serve as a vertex of a sub-grid, the midpoint of a sub-grid edge, or a point that divides a sub-grid edge into thirds or quarters. Based on the reference points in each iteration, at least two sub-grids are iteratively expanded and arranged in an array, ultimately generating a first sampling grid for dividing the pathological sampling area. The expansion and generation of at least two sub-grids can be either generating a ring of grids or generating a grid-like sub-grid surrounding the reference points.

[0088] For example, when there are multiple reference points, the first reference point is used as the sub-mesh vertex of the first iteration, and multiple sub-mesh arrays are generated in a grid pattern. Then, based on the sub-mesh generated in the first iteration, the second reference point for the next iteration is determined, and multiple sub-mesh arrays in a grid pattern around each second reference point are generated.

[0089] Optionally, if there are multiple reference points, all reference points in the first medical image can be predetermined, and the reference points can be connected sequentially in a certain order to generate a first sampling grid for dividing the pathological sampling area.

[0090] In some embodiments, a first sampling grid for dividing a pathological sampling area is generated based on at least one reference point in a first medical image, including: determining the center point of the lesion contour; using the center point as a reference point, expanding to generate at least two sub-grids arranged in an array; if at least two sub-grids intersect with the entire contour of the lesion contour, ending the expansion to generate at least two sub-grids to obtain the first sampling grid for dividing the pathological sampling area; if at least two sub-grids do not intersect with at least a portion of the contour of the lesion contour, using the newly added sub-grid vertices of the at least two sub-grids generated in this round of expansion as the reference point for the next round, and continuing to expand to generate at least two sub-grids arranged in an array using the reference point of the next round.

[0091] Optionally, the center point of the lesion outline can be automatically identified using an image detection algorithm, or it can be manually drawn by the doctor. The image detection algorithm is implemented using the open-source computer vision library OpenCV.

[0092] For example, using the center point of the lesion contour as a reference point, at least two sub-grids are expanded to generate an array. Optionally, the reference point can be the center of the entire first sampling grid, or a vertex of a sub-grid, or the midpoint of a sub-grid edge, or a point that divides a sub-grid edge into thirds or quarters.

[0093] For example, with the reference point as the center of the entire first sampling grid and the subgrids being squares, the center point can be used as the reference point to expand and generate a ring of grids arranged in an array around the reference point, with each ring of grids including at least two subgrids.

[0094] For example, when the reference point is used as the vertex of the sub-mesh and the sub-mesh is a square, the center point can be used as the reference point to expand and generate four sub-mesh arrays. The common vertex of the four sub-mesh arrays is the center point.

[0095] For example, if the reference point is the midpoint of the sub-grid edge and the sub-grid is a square, the center point can be used as the reference point to expand and generate two sub-grids arranged in an array. The midpoint of the common edge of the two sub-grids is the center point.

[0096] Optionally, the method further includes: using the center point as a reference point, and generating a coordinate system for the first medical image based on the reference point. Accordingly, the coordinate positions of the reference point and at least two sub-grids of the expanded array can be determined in the coordinate system of the first medical image, improving the accuracy of the expansion process.

[0097] For example, if at least two sub-grids intersect with the entire contour of the lesion, it indicates that the first sampling grid has completely exceeded the entire contour of the lesion. At this point, the expansion is terminated to generate at least two sub-grids, resulting in the first sampling grid used for meshing the pathological sampling area.

[0098] For example, a newly added submesh vertex refers to each submesh vertex of the at least two submesh cells generated in this round of expansion. If a reference point is used as a submesh vertex during the generation of the first sampling mesh, then a newly added submesh vertex is any submesh vertex other than the reference point used in this round among the submesh vertices of the at least two submesh cells generated during expansion.

[0099] For example, if at least two sub-grids do not intersect with at least a portion of the lesion contour, it indicates that the first sampling grid has not completely exceeded the entire lesion contour, and at least a portion of the lesion contour is still outside the first sampling grid. In this case, if there are multiple reference points, the newly added sub-grid vertices of the at least two sub-grids generated in this round of expansion are used as reference points for the next round, and the expansion continues with the next round's reference points as expansion points to generate at least two sub-grids arranged in an array. If there is only one reference point, the center point is used as the reference point to continue expanding and generating at least two sub-grids arranged in an array.

[0100] Optionally, the next reference point is used as the expansion point to expand and generate at least two sub-grids in the array arrangement. This can be to generate a ring of grids or to generate sub-grids in a grid pattern around the expansion point.

[0101] In some embodiments, if at least two sub-mesh do not intersect with at least a portion of the lesion contour, the newly added sub-mesh vertices of the at least two sub-mesh generated in this round of expansion are used as reference points for the next round, and the next round of reference points are used as expansion points to generate at least two sub-mesh arranged in a grid pattern around the expansion points.

[0102] In some embodiments, a first sampling grid for dividing a pathological sampling area is generated based on at least one reference point in a first medical image, including: determining the center point of the lesion contour; determining the center point as a reference point and determining a coordinate system based on the reference point; calculating the subgrid vertex coordinates of at least two subgrids arranged in an array in the coordinate system; and generating the first sampling grid for dividing the pathological sampling area based on the vertex coordinates of each subgrid.

[0103] Optionally, the center point of the lesion contour can be automatically identified using an image detection algorithm, or it can be manually drawn by the doctor. The image detection algorithm can be any image detection algorithm from the open-source computer vision library OpenCV.

[0104] For example, the center point is determined as the reference point, and a coordinate system is determined based on the reference point. Optionally, the coordinate system is a planar coordinate system, with the horizontal axis of the coordinate system parallel to the length of the first medical image and the vertical axis of the coordinate system parallel to the width of the first medical image.

[0105] For example, based on the subgrid parameters of at least two subgrids, the subgrid vertex coordinates of at least two subgrids arranged in an array are calculated in a coordinate system, so that the subgrid vertex coordinates can be connected to generate a first sampling grid.

[0106] In some embodiments, the submesh parameters of at least two submesh include at least one of the following: submesh size, shape, area, and interior angle size; wherein the submesh parameters are fixed by default; or, the submesh parameters are preset; or, the submesh parameters are dynamically changed.

[0107] Optionally, if the submesh shape is not rectangular or square, the submesh parameters include the interior angle sizes of the submesh. For example, if the submesh is a triangle, the interior angle size refers to the size of each interior angle of the triangle. If the submesh is a rhombus, the interior angle size refers to the size of each interior angle of the rhombus.

[0108] Optionally, based on the vertex coordinates of each sub-grid, the vertex coordinates of each sub-grid are connected sequentially in a certain order to generate a first sampling grid for dividing the pathological sampling area.

[0109] In some embodiments, a first sampling grid for dividing the pathological sampling area is generated based on the vertex coordinates of each sub-grid, including: generating a sub-grid based on the vertex coordinates of at least three sub-grids; at least one edge of the sub-grid is parallel to the target axis in the coordinate axis; and determining the target sub-grid in at least two sub-grids as a sub-grid belonging to the first sampling grid, wherein at least one sub-grid vertex of the target sub-grid is located within the lesion contour.

[0110] For example, at least two sub-grids in the first sampling grid can be one of triangles, rectangles, squares, rhombuses, or parallelograms. Based on the vertex coordinates of at least three sub-grids, a corresponding sub-grid is generated, and at least one edge of the sub-grid is parallel to the target axis in the coordinate system, which can be the horizontal axis or the vertical axis.

[0111] For example, a target subgrid refers to a subgrid belonging to the first sampling grid among at least two subgrids. Optionally, the target subgrid among the at least two subgrids is determined to be a subgrid belonging to the first sampling grid, and at least one vertex of the target subgrid is located within the lesion contour. This ensures that the first sampling grid can completely extend beyond the entire lesion contour.

[0112] In an optional embodiment of this application, the method further includes: removing duplicate subgrids from at least two subgrids; or, removing subgrids from at least two subgrids that do not intersect with the lesion outline and do not contain lesion tissue.

[0113] Optionally, at least two subgrids that do not intersect with the lesion outline and do not contain lesion tissue are removed, that is, subgrids that only cover healthy tissue are removed, to ensure that the pathological sampling area covered by each subgrid in the first sampling grid contains lesion tissue.

[0114] In some embodiments, the method further includes: merging at least two subgrids in the first sampling grid that meet the merging conditions.

[0115] In some embodiments, merging at least two subgrids in a first sampling grid that meet the merging criteria includes: determining a first subgrid from the at least two subgrids based on the size of the lesion tissue contained in the at least two subgrids; determining a second subgrid that is closest to the first subgrid from the other subgrids in the at least two subgrids besides the first subgrid; and merging the first subgrid and the second subgrid.

[0116] For example, the first sub-mesh is the sub-mesh that needs to be merged. Generally, the first sub-mesh is the sub-mesh located at the edge region of the lesion outline, and it contains both lesion tissue and healthy tissue.

[0117] Optionally, the first subgrid is automatically identified. In this case, the first subgrid is determined from at least two subgrids based on the size of the lesion tissue contained in at least two subgrids. For example, a subgrid containing lesion tissue smaller than one-quarter of the total subgrid size is determined as the first subgrid. Alternatively, a subgrid containing lesion tissue smaller than the size of the healthy tissue it contains is determined as the first subgrid.

[0118] Optionally, the first subgrid is manually drawn by the physician. In this case, in response to the subgrid selection operation, the first subgrid is determined from at least two subgrids. The first subgrid can be determined by the physician based on actual technical needs.

[0119] For example, the second subgrid is a subgrid that can be merged with the first subgrid. The second subgrid and the first subgrid share a common edge. The second subgrid includes one...

[0120] Optionally, from at least two subgrids other than the first subgrid, determine the second subgrid that is closest to the first subgrid. Merge the first and second subgrids to optimize the first sampling grid.

[0121] In some embodiments, merging at least two subgrids in the first sampling grid that meet the merging criteria includes: determining a first subgrid from the at least two subgrids based on the size of the lesion tissue contained in the at least two subgrids; determining a third subgrid from the other subgrids in the at least two subgrids that is closest to the first subgrid and contains the largest lesion tissue; and merging the first subgrid and the third subgrid.

[0122] For example, the third subgrid is a subgrid that can be merged with the first subgrid. The third subgrid shares a common edge with the first subgrid. The third subgrid comprises one. If there is more than one second subgrid, i.e., more than one second subgrid is closest to the first subgrid, then the third subgrid can also be the subgrid containing the largest lesion tissue determined from the second subgrids.

[0123] Optionally, from at least two subgrids other than the first subgrid, a third subgrid is determined that is closest to the first subgrid and contains the largest amount of lesion tissue. The first and third subgrids are then merged to optimize the first sampling grid.

[0124] In some embodiments, merging at least two subgrids in the first sampling grid that meet the merging conditions includes: determining a first subgrid from the at least two subgrids based on the size of the lesion tissue contained in the at least two subgrids; determining a fourth subgrid from the other subgrids in the at least two subgrids that is closest to the first subgrid and closest in the target direction to the center point of the lesion contour; and merging the first subgrid and the fourth subgrid.

[0125] For example, the fourth subgrid is a subgrid that can be merged with the first subgrid. The fourth subgrid shares a common edge with the first subgrid. The fourth subgrid includes one. If there is more than one third subgrid, that is, if there is more than one third subgrid that is closest to the first subgrid and contains the largest amount of lesion tissue, then the fourth subgrid can also be the subgrid that is closest in the target direction to the center point of the lesion outline determined from the third subgrid.

[0126] Optionally, the target direction distance refers to the horizontal or vertical component of the distance, and the target direction distance to the center point of the lesion contour refers to the horizontal or vertical component of the distance between the center point of the sub-grid and the center point of the lesion contour.

[0127] Optionally, from at least two sub-grids other than the first sub-grid, a fourth sub-grid is determined that is closest to the first sub-grid and also closest in the target direction to the center point of the lesion contour. The first sub-grid and the fourth sub-grid are then merged to optimize the first sampling grid.

[0128] In some embodiments, the method further includes: numbering at least two subgrids in the first sampling grid to identify at least two subgrids in the first sampling grid.

[0129] Optionally, at least two subgrids in the first sampling grid may be numbered using one or more combinations of special characters, letters, and numbers to identify at least two subgrids in the first sampling grid.

[0130] Optionally, the numbering method can be clockwise or counterclockwise, sequentially numbering from the outermost circle of the first sampling grid to the innermost circle, or sequentially numbering from the innermost circle of the first sampling grid to the outermost circle. Alternatively, it can be S-shaped, sequentially numbering from the topmost sub-grid to the bottommost sub-grid, or from the bottommost sub-grid to the topmost sub-grid, or from the leftmost sub-grid to the rightmost sub-grid, or from the rightmost sub-grid to the leftmost sub-grid. In this embodiment, the numbering method is not limited, as long as it uniquely identifies at least two sub-grids in the first sampling grid.

[0131] In some embodiments, the method further includes: displaying the numbers of at least two subgrids in the first sampling grid to assist the physician in knowing the specific sampling location.

[0132] Alternatively, when applying the first sampling grid during the Mohs procedure, the first sampling grid may also be referred to as the first Mohs grid.

[0133] Step 360: Obtain a second medical image of the lesion tissue at a second time point, wherein at least a portion of the lesion outline in the second medical image differs from that in the first medical image.

[0134] The second medical image refers to an image taken at a second time after the lesion tissue has been removed using the Mohr's procedure. For example, if, after rapid pathology, microscopic pathological examination determines that residual lesions still exist, a second resection and a second pathological sample are required. The second time refers to any point in time following the rapid pathology.

[0135] For example, the second medical image differs from at least a portion of the lesion contour in the first medical image. The reasons for the difference in at least a portion of the lesion contour include at least one of the following: a change in the position of the imaging component, a change in the camera parameters of the imaging component, a change in the lesion tissue itself, or a change in the patient's position.

[0136] In some embodiments, the lesion tissue is photographed at a second time using the same imaging component as the first medical image, to obtain a second medical image of the lesion tissue at a second time.

[0137] Step 380: Display a second medical image with a second sampling grid, the second sampling grid having at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covering the same pathological sampling area in both medical images.

[0138] The second sampling grid is obtained by transforming the first sampling grid based on image registration parameters, and is not a re-division based on the second medical image. The image registration parameters are used to transform the first sampling grid, and the type of the image registration parameters also indicates the method of transformation of the first sampling grid. Different types of image registration parameters result in different methods of transformation of the first sampling grid.

[0139] For example, the second sampling grid has at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covers the same pathological sampling area in both medical images. The sub-grid parameters of the at least two sub-grids in the second sampling grid are the same as or different from the sub-grid parameters of the at least two sub-grids in the first sampling grid. The sub-grid parameters include at least one of the following: sub-grid size, shape, area, and interior angle size.

[0140] Optionally, a second medical image with a second sampling grid is displayed. The second sampling grid has at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covers the same pathological sampling area in both medical images.

[0141] For example, if all subgrids in the first sampling grid are squares and the first subgrid in the first sampling grid covers the central region of the lesion tissue in the first medical image, then after transforming the first sampling grid, a second sampling grid is obtained. The subgrids in the second sampling grid can all be rhombuses, and the second subgrid corresponding to the first subgrid covers the central region of the lesion tissue in the second medical image.

[0142] In some embodiments, the second sampling grid has at least two sub-grids numbered corresponding to the first sampling grid, and the same sub-grid has the same number in both medical images. Thus, the physician can refer to the number of the first sampling grid to determine the same pathological sampling area represented by the same number in both medical images, facilitating secondary resection and secondary pathological sampling of the pathological sampling area when residual lesions exist there.

[0143] In some embodiments, the method further includes: displaying the numbers of at least two subgrids in the second sampling grid to assist the physician in accurately relocating the specific sampling location.

[0144] In some embodiments, the method further includes: determining a first subgrid in a first sampling grid corresponding to a pathological sampling area with residual lesions, displaying a second subgrid in a second sampling grid, the second subgrid corresponding to the first subgrid, and the second subgrid covering the pathological sampling area with residual lesions.

[0145] Optionally, after rapid pathology, the pathological sampling area with residual lesions can be identified. In this case, it is not necessary to transform and reposition the entire first sampling grid. Instead, the pathological sampling area with residual lesions can be transformed and repositioned to its corresponding position in the lesion tissue after rapid pathology, in order to assist the doctor in performing secondary excision and secondary pathological sampling at that location.

[0146] Alternatively, when applying the second sampling grid during the Mohs procedure, the second sampling grid may also be referred to as the second Mohs grid.

[0147] In summary, the method provided in this application involves acquiring a first medical image of the lesion tissue at a first time, the first medical image showing the lesion outline of the lesion tissue; displaying the first medical image with a first sampling grid, the first sampling grid including at least two sub-grids covering the lesion outline. Accordingly, a first sampling grid can be automatically generated and displayed based on the first medical image to assist the doctor in pathological sampling. Then, by acquiring a second medical image of the lesion tissue at a second time, the second medical image showing at least a different portion of the lesion outline from the first medical image; displaying the second medical image with a second sampling grid, the second sampling grid having at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covering the same pathological sampling area in both medical images. Accordingly, by repositioning the first sampling grid to determine the second sampling grid, the doctor can accurately reposition a sampling location in the corresponding position of the second medical image, achieving the location of residual lesions, and can also assist the doctor in secondary resection and secondary pathological sampling, thus improving the accuracy and efficiency of lesion resection.

[0148] For example, Figure 5 A flowchart illustrating an exemplary embodiment of the image registration method provided in this application is shown, the method further comprising:

[0149] Step 420: Obtain image registration parameters for transforming the first sampling grid.

[0150] Registration refers to the process of matching two-dimensional information of lesion tissue from different time periods, or matching three-dimensional information of lesion tissue from different time periods. Image registration parameters are used to transform the first sampling grid, and can be determined based on the two-dimensional information of the lesion tissue or based on the three-dimensional information of the lesion tissue.

[0151] Optionally, when the image registration parameters are determined based on two-dimensional information of the lesion tissue, the image registration parameters are parameters used to transform the pixels corresponding to the first sampling grid. When the image registration parameters are determined based on three-dimensional information of the lesion tissue, the image registration parameters are parameters used to transform the point cloud corresponding to the first sampling grid.

[0152] Step 440: Based on the image registration parameters, transform the first sampling grid to obtain the second sampling grid.

[0153] Optionally, when the image registration parameters are determined based on two-dimensional information of the lesion tissue, the pixels corresponding to the first sampling grid are transformed based on the image registration parameters to obtain the second sampling grid. When the image registration parameters are determined based on three-dimensional information of the lesion tissue, the point cloud corresponding to the first sampling grid is transformed based on the image registration parameters to obtain the second sampling grid.

[0154] In this embodiment, the second sampling grid can be obtained by transforming the first sampling grid using image registration parameters, without having to re-execute the corresponding generation process of the first sampling grid for the second medical image, thus effectively improving the processing efficiency of medical images. Furthermore, by distinguishing between the two-dimensional and three-dimensional information of the lesion tissue and transforming the pixels or point cloud, the accuracy of the obtained second sampling grid can be improved.

[0155] In one optional embodiment of this application, Figure 5 A flowchart illustrating an exemplary embodiment of the image registration method provided in this application is shown. Step 440 can be implemented as follows:

[0156] Step 520: Based on the image registration parameters, perform a spatial transformation on the first sampling grid to obtain the second sampling grid.

[0157] For example, image registration parameters are determined based on two-dimensional information of the lesion tissue. Image registration parameters include at least one of a displacement vector field and an affine transformation matrix. The displacement vector field, also called a displacement vector field, is used to characterize the displacement of the same pixel in two medical images. Affine transformations include at least one of translation, rotation, scaling, shearing, and reflection.

[0158] Optionally, based on the displacement vector field and / or affine transformation matrix, a spatial transformation is performed on the pixels corresponding to the first sampling grid to obtain the second sampling grid.

[0159] In this embodiment, when the image registration parameters are determined based on the two-dimensional information of the lesion tissue, a spatial transformation is specifically performed on the first sampling grid, resulting in a second sampling grid with higher accuracy and better fit to the second medical image.

[0160] In one optional embodiment of this application, Figure 6 A flowchart illustrating an exemplary embodiment of the image registration method provided in this application is shown. Step 440 can be implemented as follows:

[0161] Step 540: Obtain a first three-dimensional model of the lesion tissue with a first sampling mesh drawn on it.

[0162] For example, the image registration parameters are determined based on the three-dimensional information of the lesion tissue. The first three-dimensional model is a three-dimensional model (3D mesh) determined based on the first three-dimensional point cloud image of the lesion tissue at the first time.

[0163] Optionally, a first three-dimensional model is generated based on the first three-dimensional point cloud image, and the first sampling mesh is drawn onto the first three-dimensional model to obtain a first three-dimensional model of the lesion tissue with the first sampling mesh drawn on it.

[0164] Optionally, a rolling sphere method is used to generate a first three-dimensional model based on a first three-dimensional point cloud image. There is a one-to-one correspondence between the point cloud of the first three-dimensional model and the pixels in the first medical image. Based on this correspondence, a first sampling mesh is drawn onto the first three-dimensional model.

[0165] Step 560: Based on the image registration parameters, transform the first 3D model with the first sampling grid to obtain the second 3D model with the second sampling grid.

[0166] Optionally, the image registration parameters include a point cloud transformation matrix. Point cloud transformation includes at least one of translation, rotation, and scaling of the point cloud.

[0167] Optionally, based on the point cloud transformation matrix, a point cloud transformation is performed on the first 3D model with the first sampling mesh to obtain a second 3D model with the second sampling mesh.

[0168] Step 580: Based on the second 3D model with the second sampling mesh drawn, obtain the second sampling mesh.

[0169] Optionally, based on the second 3D model with the second sampling grid drawn, the point cloud corresponding to the second sampling grid is extracted, and the point cloud corresponding to the second sampling grid is rendered to obtain the second sampling grid.

[0170] In other embodiments, the second 3D model with the second sampling mesh drawn on it can be rendered directly to obtain a second medical image with the second sampling mesh, without the need to extract the second sampling mesh separately or obtain a second point cloud image.

[0171] Optionally, a 3D rendering method in computer vision technology is used to render the second three-dimensional model with the second sampling grid to obtain a second medical image with the second sampling grid.

[0172] Optionally, the 3D rendering method includes at least one of scanline rendering, light transfer, and projection.

[0173] In this embodiment, when the image registration parameters are determined based on the three-dimensional information of the lesion tissue, the first sampling grid is specifically transformed into a point cloud, resulting in a second sampling grid with higher accuracy, better fit to the second medical image, and more suitable for the lesion tissue.

[0174] For example, Figure 7 A flowchart illustrating an exemplary embodiment of the image registration method provided in this application is shown. Step 420 can be implemented as follows:

[0175] Step 620: Register the first medical image with the second medical image to obtain image registration parameters; or, register the lesion tissue in the first medical image with the lesion tissue in the second medical image to obtain image registration parameters.

[0176] For example, when the image registration parameters are determined based on the two-dimensional information of the lesion tissue, the first medical image and the second medical image are registered to obtain the image registration parameters.

[0177] Optionally, both the first and second medical images may include lesion tissue and at least a portion of healthy tissue. Therefore, the lesion tissue in the first medical image may be registered with the lesion tissue in the second medical image to obtain image registration parameters.

[0178] Optionally, at least one of a rigid registration algorithm or an elastic registration algorithm is used to register the first medical image with the second medical image to obtain image registration parameters, which include at least one of a displacement vector field and an affine transformation matrix.

[0179] Optionally, at least one of a rigid registration algorithm or an elastic registration algorithm is used to register the lesion tissue in the first medical image with the lesion tissue in the second medical image to obtain image registration parameters, which include at least one of a displacement vector field and an affine transformation matrix.

[0180] Optionally, depending on the number of pixels used in the image registration process, rigid registration algorithms include four-point rigid registration algorithms and multi-point (more than four points) rigid registration algorithms.

[0181] In some embodiments, when the lesion contour of the lesion tissue does not change significantly between the first and second time points, a rigid registration algorithm can be used. When the lesion contour of the lesion tissue changes significantly between the first and second time points, an elastic registration algorithm can be used.

[0182] In some embodiments, the doctor may manually register the first medical image with the second medical image. In this case, in response to the registration operation, the first medical image and the second medical image are registered to obtain image registration parameters. Alternatively, in response to the registration operation, the lesion tissue in the first medical image is registered with the lesion tissue in the second medical image to obtain image registration parameters.

[0183] In this embodiment, by employing various registration algorithms, the first medical image is registered with the second medical image, or the lesion tissue in the first medical image is registered with the lesion tissue in the second medical image, to obtain image registration parameters. This can improve the accuracy of the obtained image registration parameters, thereby improving the accuracy of the obtained second sampling grid.

[0184] The following embodiment uses the registration of the first medical image and the second medical image in step 620 to obtain image registration parameters as an example. The processing of registering the lesion tissue in the first medical image and the lesion tissue in the second medical image in step 620 to obtain image registration parameters is the same as in the following embodiment.

[0185] In one optional embodiment of this application, an elastic registration algorithm is used as an example. Optical flow (or optical flow) can be used to register the first medical image with the second medical image to obtain image registration parameters, which are displacement vector fields.

[0186] In one optional embodiment of this application, a rigid registration algorithm is used as an example. Figure 8 A flowchart illustrating an exemplary embodiment of the image registration method provided in this application is shown. Step 620 can be implemented as follows:

[0187] Step 640: Identify at least two key points from the first medical image.

[0188] For example, key points are pixels determined from the first medical image for image registration, and key points include at least two.

[0189] Optionally, at least two key points are identified from the first medical image. Depending on practical technical requirements, at least two key points may refer to at least four key points.

[0190] Step 660: Based on the feature information of each key point, perform feature matching on at least two pixels of the second medical image to determine at least two pairs of matching key points.

[0191] For example, matching keypoints refer to pairs of keypoints that match in a first medical image and a second medical image. If there are at least two keypoints, then there are at least two pairs of matching keypoints.

[0192] Optionally, each pair of matching keypoints represents the same location in the first and second medical images. For example, if a keypoint is the center point of the lesion contour in the first medical image, then a pair of matching keypoints determined based on that keypoint represents the center point of the lesion contour in the first and second medical images, respectively.

[0193] Optionally, based on the feature information of each key point, feature matching is performed on at least two pixels of the second medical image to determine at least two pairs of matching key points. Depending on practical technical needs, at least two pairs of matching key points refers to at least four pairs of matching key points.

[0194] Optionally, a scale-invariant feature transform (SIFT) algorithm is used to generate scale-invariant feature vectors for the first medical image and the second medical image, respectively; based on the scale-invariant feature vectors of each key point, the Euclidean distance between them and the scale-invariant feature vectors in the second medical image is calculated; based on the Euclidean distance, matching points for each key point are determined from each pixel of the second medical image, and at least two pairs of matching key points are determined.

[0195] Step 680: Based on at least two pairs of matching key points, register the first medical image with the second medical image to obtain image registration parameters.

[0196] Optionally, based on at least two pairs of matching key points, the first medical image and the second medical image are registered to obtain image registration parameters, which are affine transformation matrices.

[0197] Optionally, when there are four pairs of matching key points, perspective projection transformation is used to register the first medical image with the second medical image to obtain image registration parameters.

[0198] Optionally, if there are more than four pairs of matching key points, the Iterative Closest Point (ICP) algorithm is used to register the first medical image with the second medical image to obtain image registration parameters.

[0199] In this embodiment, by determining multiple pairs of matching key points, the first medical image and the second medical image are registered to obtain image registration parameters, which can improve the accuracy of the obtained image registration parameters.

[0200] For example, Figure 9 A flowchart of an image registration method provided in an exemplary embodiment of this application is shown. Step 640 determines at least two key points from a first medical image, including:

[0201] Step 641: Extract features from the first medical image to obtain at least two feature points.

[0202] For example, a feature point refers to a pixel in the first medical image whose image grayscale value changes drastically or whose curvature is large at the edge of the image. Feature points include at least one of corner points, contour points, two points in a darker area, and dark points in a brighter area.

[0203] Optionally, an image feature extraction algorithm is used to extract features from the first medical image to obtain at least two feature points. The image feature extraction algorithm includes at least one of the following: Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), and Features From Accelerated Segment Test (FAST).

[0204] Step 642: Using the first AI model, identify feature points located at the edge of the lesion contour from at least two feature points, and determine at least two key points.

[0205] For example, the first AI model is a pre-trained neural network model used to determine the pixel locations at the edges of the lesion contour. The first AI model can be at least one of the following types: the Unet image segmentation model, the Deeplab V3+ semantic segmentation model.

[0206] Optionally, the first AI model identifies at least two feature points located at the edge of the lesion contour, thus determining at least two key points.

[0207] In some embodiments, at least two key points may also be manually selected by the physician. That is, in response to the selection operation, the feature point located at the edge of the lesion contour is determined from at least two feature points to obtain at least two key points. In other examples, at least two key points may also be determined jointly based on automatic identification by a first AI model and manual selection by the physician to obtain more accurate key points.

[0208] In this embodiment, the first AI model identifies at least two key points located at the edge of the lesion contour for subsequent image registration. It can identify representative location points of the lesion contour that has changed to a certain extent, thereby achieving accurate registration between the first medical image and the second medical image.

[0209] As an example, Figure 10 A schematic diagram of an image registration method provided in an exemplary embodiment of this application is shown. Figure 10 Taking the four-point rigid registration algorithm as an example, Figure 10-1 For first medical images, Figure 10-2The first medical image is the second medical image. Features are extracted from the first medical image using the SIFT algorithm to obtain at least two feature points. Using a first AI model, feature points located at the edge of the lesion contour are identified from these two feature points, determining four key points A, B, C, and D. Feature matching is then performed on at least two pixels in the second medical image using the SIFT algorithm to determine the matching point A' for key point A, B' for key point B, C' for key point C, and D' for key point D. Next, based on the four pairs of matching key points, perspective projection is used to register the first and second medical images, resulting in an affine transformation matrix.

[0210] In one optional embodiment of this application, Figure 11 A flowchart illustrating an exemplary embodiment of the image registration method provided in this application is shown. Step 420 can be implemented as follows:

[0211] Step 720: Obtain a first three-dimensional point cloud image of the lesion tissue at a first time and a second three-dimensional point cloud image of the lesion tissue at a second time; the second three-dimensional point cloud image is different from at least a portion of the point cloud in the first three-dimensional point cloud image.

[0212] For example, when the image registration parameters are determined based on the three-dimensional information of the lesion tissue, a first three-dimensional point cloud image of the lesion tissue at a first time and a second three-dimensional point cloud image of the lesion tissue at a second time are acquired. Due to changes in the lesion contour, at least a portion of the point cloud in the second three-dimensional point cloud image differs from that in the first three-dimensional point cloud image.

[0213] Step 740: Register the first 3D point cloud image with the second 3D point cloud image to obtain image registration parameters.

[0214] Optionally, a rigid point cloud registration algorithm or a non-rigid point cloud registration algorithm is used to perform point cloud registration between the first 3D point cloud image and the second 3D point cloud image to obtain image registration parameters. The image registration parameters include the point cloud transformation matrix.

[0215] Optionally, rigid point cloud registration algorithms can be implemented based on Iterative ClosestPoint (ICP) or Normal Distribution Transform (NDT) algorithms. Non-rigid point cloud registration algorithms can be implemented based on Robust Non-Rigid Registration (RNRR) algorithms.

[0216] In some embodiments, when the lesion contour of the lesion tissue does not change significantly between the first and second time points, a rigid point cloud registration algorithm can be used. When the lesion contour of the lesion tissue changes significantly between the first and second time points, a non-rigid point cloud registration algorithm can be used.

[0217] In some embodiments, the doctor may manually perform point cloud registration between the first three-dimensional point cloud image and the second three-dimensional point cloud image. In this case, in response to the point cloud registration operation, the first three-dimensional point cloud image and the second three-dimensional point cloud image are registered to obtain image registration parameters.

[0218] In this embodiment, by employing various point cloud registration algorithms, the first 3D point cloud image and the second 3D point cloud image are registered to obtain image registration parameters. This can improve the accuracy of the obtained image registration parameters, thereby improving the accuracy of the obtained second sampling mesh.

[0219] For example, Figure 12 A flowchart illustrating an exemplary embodiment of the image registration method provided in this application is shown. Step 740 above may include the following steps:

[0220] Step 760: Determine at least two key point clouds from the first 3D point cloud image.

[0221] For example, the key point cloud is a set of point cloud points determined from a first 3D point cloud image for point cloud registration, and the key point cloud includes at least two points.

[0222] Optionally, at least two key point clouds are determined from the first 3D point cloud image.

[0223] Step 770: Based on the feature information of each key point cloud, perform feature matching on at least two point clouds of the second three-dimensional point cloud image to obtain at least two pairs of matching point clouds.

[0224] For example, a matching point cloud refers to a pair of matching point clouds in a first 3D point cloud image and a second 3D point cloud image. If the key point cloud includes at least two points, then the matching point cloud includes at least two pairs.

[0225] Optionally, a rigid point cloud registration algorithm or a non-rigid point cloud registration algorithm is used to perform feature matching on at least two point clouds of the second three-dimensional point cloud image based on the feature information of each key point cloud, so as to obtain at least two pairs of matching point clouds.

[0226] Step 780: Based on at least two pairs of matching point clouds, register the first 3D point cloud image with the second 3D point cloud image to obtain image registration parameters.

[0227] Optionally, based on at least two pairs of matching point clouds, the first 3D point cloud image and the second 3D point cloud image are registered to obtain image registration parameters. The image registration parameters are point cloud transformation matrices.

[0228] In this embodiment, by determining multiple pairs of matching point clouds, the first three-dimensional point cloud image and the second three-dimensional point cloud image are registered to obtain image registration parameters, which can improve the accuracy of the obtained image registration parameters.

[0229] For example, Figure 13 A flowchart of an image registration method provided in an exemplary embodiment of this application is shown. Step 760 determines at least two key point clouds from a first 3D point cloud image, including:

[0230] Step 761: Using the second AI model, identify at least two point clouds in the first three-dimensional point cloud image that match healthy tissue located outside the lesion outline, and determine at least two key point clouds.

[0231] For example, the second AI model is a pre-trained neural network model for point cloud data processing, used to identify point clouds of healthy tissue located within the contours of lesions. The second AI model can be at least one of the following types: PointNet model for point cloud processing, PointNet++ model for point cloud processing, and F-PointNet model for point cloud detection.

[0232] In this embodiment, the second AI model identifies at least two point clouds of healthy tissue located outside the lesion outline for subsequent image registration. Since the changes in the point cloud of healthy tissue are extremely small, combining the point cloud of healthy tissue for registration can improve the accuracy of point cloud registration, thereby achieving accurate registration between the first three-dimensional point cloud image and the second three-dimensional point cloud image.

[0233] In one optional embodiment of this application, Figure 14 A flowchart illustrating an exemplary embodiment of the image registration method provided in this application is shown, the method further comprising:

[0234] Step 570: Render the second 3D model with the second sampling mesh to obtain a second medical image with the second sampling mesh.

[0235] For example, after transforming the first 3D model with the first sampling grid based on the image registration parameters to obtain the second 3D model with the second sampling grid, the second 3D model with the second sampling grid can be rendered to obtain the second medical image with the second sampling grid. There is no need to extract the second sampling grid separately, nor is there a need to obtain the second point cloud image.

[0236] Optionally, a 3D rendering method in computer vision technology is used to render the second three-dimensional model with the second sampling grid to obtain a second medical image with the second sampling grid.

[0237] Optionally, the 3D rendering method includes at least one of scanline rendering and radiosity.

[0238] In this embodiment, a second medical image with the second sampling grid is obtained by rendering the second three-dimensional model with the second sampling grid. There is no need to extract the second sampling grid separately or obtain the second point cloud image, which can improve the processing efficiency of medical images.

[0239] As an example, the overall flow of the image registration method in the embodiments of this application will be described below.

[0240] Please see Figure 15 and Figure 16 This embodiment provides an image registration method based on two-dimensional information. Please refer to [link / reference]. Figure 17 and Figure 18 This embodiment also provides an image registration method based on two-dimensional and three-dimensional information.

[0241] Optionally, when the lesion is located in a relatively flat area or with minimal curvature, such as on the limbs or trunk, a two-dimensional information-based image registration method can be used for medical image processing. When the lesion is located in an uneven area or with significant curvature, such as on the nose, head, or joints, three-dimensional information can be introduced, and a two-dimensional and three-dimensional information-based image registration method can be used for medical image processing.

[0242] 1. Image registration method based on two-dimensional information

[0243] Step 11: Obtain a first medical image of the lesion tissue at the first moment. The first medical image has the lesion outline of the lesion tissue.

[0244] Optionally, the lesion tissue is tumor tissue. "First time point" refers to any point in time after the lesion tissue is removed but before rapid pathological examination. The lesion tissue is photographed using an RGB camera to obtain a first medical image of the lesion tissue at the first time point. The first medical image is shown below. Figure 15-1 As shown.

[0245] Optionally, after step 11, the method may further include: identifying the pathological sampling area corresponding to the lesion outline in the first medical image; generating a first sampling grid for dividing the pathological sampling area into grids; numbering at least two sub-grids in the first sampling grid; and displaying the first medical image having the first sampling grid.

[0246] For example, a first medical image having a first sampling grid, such as Figure 15-3 As shown, each subgrid in the first sampling grid is a square, and Q1, Q2, Q3, and Q4 represent the first quadrant, the second quadrant, the third quadrant, and the fourth quadrant, respectively.

[0247] Optionally, a first medical image with a first sampling grid can be displayed on a terminal display screen, virtual reality (VR) glasses, or the first sampling grid can be projected onto the pathological sampling area corresponding to the lesion outline before rapid pathology using a projection component.

[0248] Step 12: Obtain a second medical image of the lesion tissue at a second time point. The second medical image has a different lesion outline from at least a portion of the lesion outline in the first medical image.

[0249] Optionally, the second time point can be any point after rapid pathology. The lesion tissue is photographed using the same RGB camera to obtain a second medical image of the lesion tissue at the second time point, as shown in the image below. Figure 15-2 As shown, due to changes in the position of the RGB camera or changes in the lesion tissue itself, the second medical image differs from at least a portion of the lesion outline in the first medical image.

[0250] Step 13: Extract features from the first medical image to obtain at least two feature points.

[0251] Optionally, the Scale Invariant Feature Transform (SIFT) algorithm is used to extract features from the first medical image to obtain at least two feature points.

[0252] Step 14: Using the first AI model, identify feature points located at the edge of the lesion contour from at least two feature points, and determine at least two key points.

[0253] Optionally, the first AI model identifies feature points located at the edge of the lesion contour from at least two feature points, thereby determining at least two key points. The first AI model can be at least one of the following types: the Unet image segmentation model, or the Deeplab V3+ semantic segmentation model.

[0254] Optionally, depending on actual technical needs, at least two key points refer to at least four key points.

[0255] Step 15: Based on the feature information of each key point, perform feature matching on at least two pixels of the second medical image to determine at least two pairs of matching key points.

[0256] Optionally, the Scale Invariant Feature Transform (SIFT) algorithm is used to perform feature matching on at least two pixels of the second medical image based on the feature information of each key point, thereby determining at least two pairs of matching key points.

[0257] Step 16: Based on at least two pairs of matching key points, register the first medical image with the second medical image to obtain image registration parameters.

[0258] Optionally, the optical flow method in the elastic registration algorithm is used to register the first medical image and the second medical image to obtain image registration parameters, which include a displacement vector field.

[0259] Optionally, when there are four pairs of matching key points, the perspective projection transformation in the four-point rigid registration algorithm is used to register the first medical image with the second medical image to obtain image registration parameters, which include the affine transformation matrix.

[0260] Optionally, when there are more than four pairs of matching key points, the Iterative Closest Point (ICP) algorithm is used to register the first medical image with the second medical image to obtain image registration parameters, which include the affine transformation matrix.

[0261] Step 17: Based on the image registration parameters, perform a spatial transformation on the first sampling grid to obtain the second sampling grid.

[0262] Optionally, based on the displacement vector field and / or affine transformation matrix, a spatial transformation is performed on the pixels corresponding to the first sampling grid to obtain the second sampling grid.

[0263] Step 18: Display a second medical image with a second sampling grid, the second sampling grid having at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covering the same pathological sampling area in both medical images.

[0264] Optionally, a second medical image with a second sampling grid can be displayed on a terminal's screen, virtual reality (VR) glasses, or the second sampling grid can be projected onto the pathological sampling area corresponding to the lesion outline after rapid pathology using a projection component. For example, a second medical image with a second sampling grid might be shown below. Figure 15-4 As shown.

[0265] 2. Image registration method based on two-dimensional and three-dimensional information

[0266] Step 21: Obtain a first medical image of the lesion tissue at the first moment. The first medical image has the lesion outline of the lesion tissue.

[0267] Optionally, the lesion tissue is tumor tissue. "First time point" refers to any point in time after the lesion tissue has been removed but before rapid pathological examination. The lesion tissue is photographed using an RGB camera to obtain the first medical image of the lesion tissue at the first time point, such as... Figure 17-1 As shown.

[0268] Optionally, after step 21, the method may further include: identifying the pathological sampling region corresponding to the lesion contour in the first medical image; generating a first sampling grid for dividing the pathological sampling region into grids; numbering at least two sub-grids in the first sampling grid; and displaying the first medical image having the first sampling grid, as shown in the image below. Figure 17-4 As shown.

[0269] Optionally, a first medical image with a first sampling grid can be displayed on a terminal display screen, virtual reality (VR) glasses, or the first sampling grid can be projected onto the pathological sampling area corresponding to the lesion outline before rapid pathology using a projection component.

[0270] Step 22: Obtain the first three-dimensional point cloud image of the lesion tissue at the first moment.

[0271] Optionally, the lesion tissue is photographed using an RGBD camera to obtain a first three-dimensional point cloud image of the lesion tissue at a specific time point. The first three-dimensional point cloud image is shown below. Figure 17-2 As shown.

[0272] Optionally, steps 22 and 21 above can be performed in parallel.

[0273] Step 23: Obtain a second three-dimensional point cloud image of the lesion tissue at the second time point.

[0274] Optionally, the lesion tissue is photographed using the same RGBD camera to obtain a second three-dimensional point cloud image of the lesion tissue at a second time point. The second three-dimensional point cloud image is shown below. Figure 17-3 As shown.

[0275] Step 24: Using the second AI model, identify at least two point clouds in the first 3D point cloud image that match healthy tissue located outside the lesion outline, and determine at least two key point clouds.

[0276] Optionally, the second AI model identifies at least two point clouds in the first 3D point cloud image that match healthy tissue located outside the lesion contour, thus determining at least two key point clouds. The second AI model can be at least one of the following types: PointNet model for point cloud processing, PointNet++ model for point cloud processing, and F-PointNet model for point cloud detection.

[0277] Step 25: Based on the feature information of each key point cloud, perform feature matching on at least two point clouds of the second three-dimensional point cloud image to obtain at least two pairs of matching point clouds.

[0278] Optionally, the ICP algorithm in the rigid point cloud registration algorithm is adopted to perform feature matching on at least two point clouds of the second three-dimensional point cloud image based on the feature information of each key point cloud, so as to obtain at least two pairs of matching point clouds.

[0279] Optionally, a robust non-rigid registration algorithm is used in non-rigid point cloud registration algorithms. Based on the feature information of each key point cloud, feature matching is performed on at least two point clouds of the second three-dimensional point cloud image to obtain at least two pairs of matching point clouds.

[0280] Step 26: Based on at least two pairs of matching point clouds, register the first 3D point cloud image with the second 3D point cloud image to obtain image registration parameters.

[0281] Optionally, based on at least two pairs of matching point clouds, the first three-dimensional point cloud image and the second three-dimensional point cloud image are registered to obtain image registration parameters, which include the point cloud transformation matrix.

[0282] Step 27: Obtain a first 3D model of the lesion tissue with a first sampling mesh.

[0283] Optionally, the rolling ball method is used to generate a first three-dimensional model based on the first three-dimensional point cloud image. The first sampling mesh is then drawn onto the first three-dimensional model to obtain a first three-dimensional model of the lesion tissue with the first sampling mesh drawn on it, such as... Figure 17-5 As shown.

[0284] Step 28: Based on the image registration parameters, transform the first 3D model with the first sampling grid to obtain the second 3D model with the second sampling grid.

[0285] Optionally, based on the point cloud transformation matrix, a point cloud transformation is performed on the first 3D model with the first sampling mesh to obtain a second 3D model with the second sampling mesh, such as... Figure 17-6 As shown.

[0286] Step 29: Render the second 3D model with the second sampling mesh to obtain a second medical image with the second sampling mesh.

[0287] Optionally, a 3D rendering method in computer vision technology is used to render the second three-dimensional model with the second sampling grid to obtain a second medical image with the second sampling grid.

[0288] Optionally, the 3D rendering method includes at least one of scanline rendering and radiosity.

[0289] Step 30: Display a second medical image with a second sampling grid, the second sampling grid having at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covering the same pathological sampling area in both medical images.

[0290] For example, a second medical image with a second sampling grid is displayed as follows: Figure 17-7 As shown. Optionally, a second medical image with a second sampling grid can be displayed on a terminal display screen, virtual reality (VR) glasses, or the second sampling grid can be projected onto the pathological sampling area corresponding to the lesion contour after rapid pathology using a projection component.

[0291] Figure 19 This application shows a structural block diagram of an image registration apparatus provided in an exemplary embodiment. The image registration apparatus includes:

[0292] The acquisition module 810 is used to acquire a first medical image of the lesion tissue at a first moment, the first medical image having the lesion outline of the lesion tissue.

[0293] Display module 820 is used to display the first medical image having a first sampling grid, the first sampling grid including at least two sub-grids covering the outline of the lesion.

[0294] The acquisition module 810 is used to acquire a second medical image of the lesion tissue at a second time, wherein the second medical image is different from at least a portion of the lesion outline in the first medical image.

[0295] The display module 820 is used to display the second medical image having a second sampling grid, the second sampling grid having at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covering the same pathological sampling area in the two medical images.

[0296] In one embodiment, the apparatus further includes: a mesh transformation module;

[0297] In one embodiment, the mesh transformation module is used for:

[0298] Obtain image registration parameters for transforming the first sampling grid;

[0299] Based on the image registration parameters, the first sampling grid is transformed to obtain the second sampling grid.

[0300] In one embodiment, the mesh transformation module is used for:

[0301] Based on the image registration parameters, the first sampling grid is spatially transformed to obtain the second sampling grid.

[0302] In one embodiment, the mesh transformation module is used for:

[0303] Obtain a first three-dimensional model of the lesion tissue, which is drawn with the first sampling mesh;

[0304] Based on the image registration parameters, the first three-dimensional model with the first sampling grid is transformed to obtain a second three-dimensional model with the second sampling grid.

[0305] The second sampling mesh is obtained based on the second three-dimensional model on which the second sampling mesh is drawn.

[0306] In one embodiment, the apparatus further includes: an image registration module;

[0307] In one embodiment, the image registration module is used for:

[0308] The first medical image is registered with the second medical image to obtain the image registration parameters; or, the lesion tissue in the first medical image is registered with the lesion tissue in the second medical image to obtain the image registration parameters.

[0309] In one embodiment, the image registration module is used for:

[0310] Identify at least two key points from the first medical image;

[0311] Based on the feature information of each key point, feature matching is performed on at least two pixels of the second medical image to determine at least two pairs of matching key points;

[0312] Based on the at least two pairs of matching key points, the first medical image and the second medical image are registered to obtain the image registration parameters.

[0313] In one embodiment, the image registration module is used for:

[0314] Feature extraction is performed on the first medical image to obtain at least two feature points;

[0315] The first AI model identifies the feature point located at the edge of the lesion contour among the at least two feature points, and determines the at least two key points.

[0316] In one embodiment, the image registration module is used for:

[0317] A first three-dimensional point cloud image of the lesion tissue at the first time point and a second three-dimensional point cloud image of the lesion tissue at the second time point are obtained; at least a portion of the point cloud in the second three-dimensional point cloud image is different from that in the first three-dimensional point cloud image.

[0318] The first 3D point cloud image and the second 3D point cloud image are registered to obtain the image registration parameters.

[0319] In one embodiment, the image registration module is used for:

[0320] Identify at least two key point clouds from the first 3D point cloud image;

[0321] Based on the feature information of each key point cloud, feature matching is performed on at least two point clouds of the second three-dimensional point cloud image to obtain at least two pairs of matching point clouds.

[0322] Based on the at least two pairs of matching point clouds, the first three-dimensional point cloud image and the second three-dimensional point cloud image are registered to obtain the image registration parameters.

[0323] In one embodiment, the image registration module is used for:

[0324] Using a second AI model, at least two point clouds matching healthy tissue located outside the lesion contour in the first three-dimensional point cloud image are identified, and the at least two key point clouds are determined.

[0325] In one embodiment, the display module 820 is configured to:

[0326] The second three-dimensional model with the second sampling grid is rendered to obtain the second medical image with the second sampling grid.

[0327] This application also provides a computer device, which includes: a processor and a memory, wherein the memory stores a computer program; the processor is used to execute the computer program in the memory to implement the image registration method provided in the above-described method embodiments.

[0328] Alternatively, the computer device is a server. For example, Figure 20 This is a structural block diagram of a server provided in an exemplary embodiment of this application.

[0329] Typically, server 1000 includes a processor 1001 and memory 1002.

[0330] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1001 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0331] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one instruction, which is executed by the processor 1001 to implement the method and / or usage method of the image processing model provided in the method embodiments of this application.

[0332] In some embodiments, the server 1000 may optionally include an input interface 1003 and an output interface 1004. The processor 1001, memory 1002, and input interfaces 1003 and 1004 can be connected via a bus or signal lines. Various peripheral devices can be connected to the input interfaces 1003 and 1004 via a bus, signal lines, or a circuit board. The input interfaces 1003 and 1004 can be used to connect at least one input / output (I / O) related peripheral device to the processor 1001 and memory 1002. In some embodiments, the processor 1001, memory 1002, and input interfaces 1003 and 1004 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, memory 1002, and input interfaces 1003 and 1004 can be implemented on separate chips or circuit boards, and this application does not limit this.

[0333] Those skilled in the art will understand that Figure 20 The structure shown does not constitute a limitation on the computer device 1000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0334] In an exemplary embodiment, this application provides a chip including programmable logic circuitry and / or program instructions, which, when the chip is run on a computer device, are used to implement the image registration method described above.

[0335] This application provides a computer-readable storage medium storing a computer program, including at least one instruction, at least one program segment, a code set, or an instruction set, wherein the at least one instruction, the at least one program segment, the code set, or the instruction set is loaded and executed by a processor to implement the image registration method provided in the above-described method embodiments.

[0336] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image registration method provided in the above-described method embodiments.

[0337] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0338] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0339] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0340] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image registration method, characterized in that, The method includes: Acquire a first medical image of the lesion tissue at a first moment, the first medical image having the lesion outline of the lesion tissue; The first medical image is displayed having a first sampling grid, the first sampling grid including at least two sub-grids covering the outline of the lesion; A second medical image of the lesion tissue is acquired at a second time point, wherein at least a portion of the lesion contour in the second medical image is different from that in the first medical image; Obtain image registration parameters for transforming the first sampling grid; transform the first sampling grid based on the image registration parameters to obtain a second sampling grid; display a second medical image having the second sampling grid, wherein the second sampling grid has at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covers the same pathological sampling area in the two medical images, and the image registration parameters are determined based on the two-dimensional information of the lesion tissue or based on the three-dimensional information of the lesion tissue; Wherein, when the image registration parameters are determined based on the three-dimensional information of the lesion tissue, the step of transforming the first sampling grid based on the image registration parameters to obtain the second sampling grid includes: acquiring a first three-dimensional model of the lesion tissue with the first sampling grid drawn on it; transforming the first three-dimensional model with the first sampling grid drawn on it based on the image registration parameters to obtain a second three-dimensional model with the second sampling grid drawn on it; obtaining the second sampling grid based on the second three-dimensional model with the second sampling grid drawn on it, wherein the first three-dimensional model is a three-dimensional model determined based on a first three-dimensional point cloud image of the lesion tissue at the first time, and the first three-dimensional point cloud image is a three-dimensional point cloud image of the first medical image containing the three-dimensional information of the lesion tissue.

2. The method according to claim 1, characterized in that, When the image registration parameters are determined based on the two-dimensional information of the lesion tissue, the step of transforming the first sampling grid based on the image registration parameters to obtain the second sampling grid includes: Based on the image registration parameters, the first sampling grid is spatially transformed to obtain the second sampling grid.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining image registration parameters for transforming the first sampling grid includes: The first medical image and the second medical image are registered to obtain the image registration parameters; Alternatively, the lesion tissue in the first medical image can be registered with the lesion tissue in the second medical image to obtain the image registration parameters.

4. The method according to claim 3, characterized in that, The process of registering the first medical image with the second medical image to obtain the image registration parameters includes: Identify at least two key points from the first medical image; Based on the feature information of each key point, feature matching is performed on at least two pixels of the second medical image to determine at least two pairs of matching key points; Based on the at least two pairs of matching key points, the first medical image and the second medical image are registered to obtain the image registration parameters.

5. The method according to claim 4, characterized in that, The determination of at least two key points from the first medical image includes: Feature extraction is performed on the first medical image to obtain at least two feature points; The first artificial intelligence (AI) model is used to identify the feature point located at the edge of the lesion contour among the at least two feature points, and to determine the at least two key points.

6. The method according to claim 1, characterized in that, The step of obtaining image registration parameters for transforming the first sampling grid includes: A first three-dimensional point cloud image of the lesion tissue at the first time point and a second three-dimensional point cloud image of the lesion tissue at the second time point are obtained; at least a portion of the point cloud in the second three-dimensional point cloud image is different from that in the first three-dimensional point cloud image. The first 3D point cloud image and the second 3D point cloud image are registered to obtain the image registration parameters.

7. The method according to claim 6, characterized in that, The step of registering the first 3D point cloud image with the second 3D point cloud image to obtain the image registration parameters includes: Identify at least two key point clouds from the first 3D point cloud image; Based on the feature information of each key point cloud, feature matching is performed on at least two point clouds of the second three-dimensional point cloud image to obtain at least two pairs of matching point clouds. Based on the at least two pairs of matching point clouds, the first three-dimensional point cloud image and the second three-dimensional point cloud image are registered to obtain the image registration parameters.

8. The method according to claim 7, characterized in that, Determining at least two key point clouds from the first 3D point cloud image includes: Using a second AI model, at least two point clouds matching healthy tissue located outside the lesion contour in the first three-dimensional point cloud image are identified, and the at least two key point clouds are determined.

9. The method according to claim 1, characterized in that, The method further includes: The second three-dimensional model with the second sampling grid is rendered to obtain the second medical image with the second sampling grid.

10. An image registration device, characterized in that, The device includes: The acquisition module is used to acquire a first medical image of the lesion tissue at a first moment, wherein the first medical image has the lesion outline of the lesion tissue; A display module is configured to display the first medical image having a first sampling grid, the first sampling grid including at least two sub-grids covering the outline of the lesion; The acquisition module is used to acquire a second medical image of the lesion tissue at a second time, wherein the second medical image is different from at least a portion of the lesion outline in the first medical image; The grid transformation module is used to obtain image registration parameters for transforming the first sampling grid; based on the image registration parameters, the first sampling grid is transformed to obtain a second sampling grid; the display module is used to display the second medical image having the second sampling grid, the second sampling grid having at least two sub-grids corresponding to the first sampling grid, and the same sub-grid covering the same pathological sampling area in the two medical images, the image registration parameters being determined based on the two-dimensional information of the lesion tissue or based on the three-dimensional information of the lesion tissue; Wherein, when the image registration parameters are determined based on the three-dimensional information of the lesion tissue, the step of transforming the first sampling grid based on the image registration parameters to obtain the second sampling grid includes: acquiring a first three-dimensional model of the lesion tissue with the first sampling grid drawn on it; transforming the first three-dimensional model with the first sampling grid drawn on it based on the image registration parameters to obtain a second three-dimensional model with the second sampling grid drawn on it; obtaining the second sampling grid based on the second three-dimensional model with the second sampling grid drawn on it, wherein the first three-dimensional model is a three-dimensional model determined based on a first three-dimensional point cloud image of the lesion tissue at the first time, and the first three-dimensional point cloud image is a three-dimensional point cloud image of the first medical image containing the three-dimensional information of the lesion tissue.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the image registration method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the image registration method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, from which a processor retrieves the computer instructions, causing the processor to load and execute them to implement the image registration method as described in any one of claims 1 to 9.

Images