Surgical sample image analysis method and device, computing equipment and storage medium

Abstract

The invention relates to the field of medical image processing and analysis, and discloses a surgical sample image analysis method and device, computing equipment and a storage medium, and the method comprises the steps: obtaining an endoscope image of a target user before tissue sampling; obtaining a macroscopic image of the sampled tissue after color marking; preprocessing the endoscope image and the macroscopic image and extracting common feature points; performing spatial transformation on the endoscope image so as to align the endoscope image with the macroscopic image; identifying and positioning a lesion area; segmenting a lesion area to generate an endoscope lesion image and a macroscopic lesion image; performing pathological analysis according to the section specimen corresponding to the lesion area to obtain a pathological analysis result; and constructing a three-dimensional sample reconstruction model according to the endoscope lesion image and the macroscopic lesion image. According to the method, the three-dimensional model of the surgical sample is reconstructed according to the endoscope lesion image and the macroscopic lesion image, the three-dimensional model with higher image quality is obtained, and subsequent combined comparison with a pathological analysis report is facilitated.

Description

Technical Field

[0001] The present application relates to the field of deployment and installation environment integration, and more specifically to a surgical sample image analysis method, apparatus, computing device and storage medium. Background Art

[0002] Various skin and mucosal specimens removed during various surgeries, including endoscopic mucosal resection, subtotal gastrectomy, skin lesion resection, hysterectomy, etc., all need to be sent to the pathology department for pathological diagnosis. It is reported that the difference rate between the histological results obtained by endoscopic forceps biopsy and resection specimens (including endoscopic submucosal dissection (ESD) and endoscopic mucosal resection (EMR)) is 2.7%-49%. In clinical practice, there is sometimes a discrepancy between the endoscopic evaluation and pathological results of early gastric cancer (EGC), especially when the area of ​​the suspected lesion is small, it is easy to be underestimated. In addition, for flat lesions that lack obvious morphological features such as depression (including ulcers) or raised areas, it is more challenging to reconstruct accurate histopathological reconstruction on endoscopic images, and only a rough pathological map can be inferred. The stretching and deformation of the specimen during fixation on the splint and the horizontal flipping of endoscopic photography will also affect the accuracy of pathological reconstruction on endoscopic images. In addition, it is also very time-consuming to determine the discrepancy between endoscopic evaluation and histopathological results, as endoscopists and pathologists need to jointly review and re-examine the preoperative endoscopic and histopathological images. Therefore, the three-dimensional model obtained after the surgical sample image analysis is convenient for the diagnosis of the resected specimen. Summary of the invention

[0003] The embodiments of the present application provide a surgical sample image analysis method, apparatus, computing device and storage medium, which implement intelligent surgical sample image analysis according to different component packages and installation information, match personalized component packages and implement integrated installation, simplify the installation process and improve reliability.

[0004] On the one hand, an embodiment of the present application provides a surgical sample image analysis method, including: obtaining an endoscopic image of a target user before tissue sampling; after sampling the target user's tissue and color-marking the sampled tissue, obtaining a macroscopic image of the sampled tissue after color marking; preprocessing the endoscopic image and the macroscopic image to obtain a standardized endoscopic image and a macroscopic image; extracting common feature points from the standardized endoscopic image and the macroscopic image; based on the extracted common feature points, spatially transforming the standardized endoscopic image to align the standardized endoscopic image and the standardized macroscopic image; identifying and locating the lesion area in the standardized endoscopic image and the macroscopic image; segmenting the lesion area to generate an endoscopic lesion image and a macroscopic lesion image; after obtaining a slice specimen corresponding to the lesion area, performing a pathological analysis on the slice specimen image corresponding to the lesion area to obtain a pathological analysis result; and constructing a three-dimensional sample reconstruction model based on the endoscopic lesion image and the macroscopic lesion image using a neural network model.

[0005] In one embodiment, the surgical sample image analysis method is applied to a Windows system, and the macroscopic image of the color-marked sampled tissue is obtained by processing the sampled tissues with different lesion structures marked with different colors using a magnifying endoscopic narrow-band imaging technique.

[0006] In one embodiment, the preprocessing includes contrast enhancement, noise removal and size unification, and the preprocessing of the endoscopic image and the macroscopic image to obtain a standardized endoscopic image and macroscopic image includes:

[0007] Calling an image enhancement algorithm to perform image enhancement processing on the endoscopic image and the macroscopic image to obtain enhanced endoscopic image and macroscopic image;

[0008] Calling a Gaussian filter to perform noise removal processing on the enhanced endoscopic image and macroscopic image to obtain a denoised endoscopic image and macroscopic image;

[0009] The denoised endoscopic image and macroscopic image are cropped to make the sizes of the denoised endoscopic image and macroscopic image uniform, thereby obtaining standardized endoscopic image and macroscopic image.

[0010] In one embodiment, extracting common feature points from the standardized endoscopic image and the macroscopic image includes:

[0011] Calling a feature detector to detect a first feature point in the standardized endoscopic image and a second feature point in the macroscopic image using a feature point detection algorithm, wherein there are a plurality of the first feature point and a plurality of the second feature point;

[0012] Calculate the descriptors of the first feature point and the second feature point by using a descriptor calculation algorithm, wherein the descriptors of the first feature point and the second feature point are respectively data sets containing image information of the first feature point and the second feature point;

[0013] Common feature points in the standardized endoscopic image and macroscopic image are determined according to the descriptors of the first feature points and the second feature points.

[0014] In one embodiment, the step of spatially transforming the standardized endoscopic image according to the extracted common feature points so as to align the standardized endoscopic image with the standardized macroscopic image comprises:

[0015] Performing feature point matching on the standardized endoscopic image and the macroscopic image using a matching algorithm to obtain matching points between the standardized endoscopic image and the macroscopic image;

[0016] Obtaining the coordinates of the matching points;

[0017] Calculate the homography matrix between the matching points according to the coordinates of the matching points;

[0018] The standardized endoscopic image is perspective transformed and aligned with the standardized macroscopic image according to the coordinates of the matching points.

[0019] In one embodiment, segmenting the lesion area to generate an endoscopic lesion image and a macroscopic lesion image includes:

[0020] Invoking an edge detection algorithm to extract edges of the lesion areas of the standardized endoscopic image and the macroscopic image, respectively, to determine edge positions of the sample lesion areas;

[0021] The coordinates of a pixel point at the edge position of the sample lesion area are obtained, and the pixel point is used as the starting point. The pixel points at the edge position are connected by calling the region growing algorithm to generate an endoscopic lesion image and a macroscopic lesion image respectively.

[0022] In one embodiment, the neural network model is a convolutional neural network model including multiple convolutional layers, and the three-dimensional sample reconstruction model is constructed using the neural network model according to the endoscopic lesion image and the macroscopic lesion image, including:

[0023] Inputting the endoscopic lesion image, the macroscopic lesion image and the pathological analysis result into the neural network model, and obtaining a multi-layer two-dimensional image of the lesion tissue according to the endoscopic lesion image and the macroscopic lesion image;

[0024] Analytically transforming and calculating the multi-layer two-dimensional images to obtain the coronal image and sagittal image of the diseased tissue;

[0025] A three-dimensional sample reconstruction model is formed according to the tube-pile position map and the sagittal position map of the diseased tissue.

[0026] In a second aspect, an embodiment of the present application provides a surgical sample image analysis device having a function of implementing the surgical sample image analysis method provided in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. In one embodiment, the surgical sample image analysis device includes:

[0027] The first acquisition module is used to acquire the endoscopic image of the target user before tissue sampling;

[0028] The second acquisition module is used for sampling the tissue of the target user and color-marking the sampled tissue, and then acquiring a macroscopic image of the sampled tissue after color marking;

[0029] Preprocessing module: used for preprocessing the endoscopic image and macroscopic image to obtain standardized endoscopic image and macroscopic image;

[0030] Extraction module: used to extract common feature points from standardized endoscopic images and macroscopic images;

[0031] A registration module: used for performing spatial transformation on the standardized endoscopic image according to the extracted common feature points so as to align the standardized endoscopic image with the standardized macroscopic image;

[0032] Identification module: used for identifying and locating the lesion area in the standardized endoscopic image and macroscopic image;

[0033] Segmentation module: used to segment the lesion area and generate endoscopic lesion images and macroscopic lesion images

[0034] Analysis module: used for performing pathological analysis on the slice specimen image corresponding to the lesion area after acquiring the slice specimen corresponding to the lesion area, so as to obtain the pathological analysis result;

[0035] Reconstruction module: used to construct a three-dimensional sample reconstruction model using a neural network model based on the endoscopic lesion image and the macroscopic lesion image.

[0036] In one embodiment, in the second acquisition module, the macroscopic image of the color-marked sampled tissue is obtained by processing the sampled tissues with different lesion structures marked with different colors using a magnifying endoscopic narrow-band imaging technique.

[0037] In one embodiment, the preprocessing includes contrast enhancement, noise removal and size unification, and the preprocessing module is specifically used for:

[0038] Calling an image enhancement algorithm to perform image enhancement processing on the endoscopic image and the macroscopic image to obtain enhanced endoscopic image and macroscopic image;

[0039] Calling a Gaussian filter to perform noise removal processing on the enhanced endoscopic image and macroscopic image to obtain a denoised endoscopic image and macroscopic image;

[0040] The denoised endoscopic image and macroscopic image are cropped to make the sizes of the denoised endoscopic image and macroscopic image uniform, thereby obtaining standardized endoscopic image and macroscopic image.

[0041] In one embodiment, the extraction module is specifically used to:

[0042] Calling a feature detector to detect a first feature point in the standardized endoscopic image and a second feature point in the macroscopic image using a feature point detection algorithm, wherein there are a plurality of the first feature point and a plurality of the second feature point;

[0043] Calculate the descriptors of the first feature point and the second feature point by using a descriptor calculation algorithm, wherein the descriptors of the first feature point and the second feature point are respectively data sets containing image information of the first feature point and the second feature point;

[0044] Common feature points in the standardized endoscopic image and macroscopic image are determined according to the descriptors of the first feature points and the second feature points.

[0045] In one embodiment, the registration module is specifically used to:

[0046] Performing feature point matching on the standardized endoscopic image and the macroscopic image using a matching algorithm to obtain matching points between the standardized endoscopic image and the macroscopic image;

[0047] Obtaining the coordinates of the matching points;

[0048] Calculate the homography matrix between the matching points according to the coordinates of the matching points;

[0049] The standardized endoscopic image is perspective transformed and aligned with the standardized macroscopic image according to the coordinates of the matching points.

[0050] In one embodiment, the segmentation module is specifically used to:

[0051] Invoking an edge detection algorithm to extract edges of the lesion areas of the standardized endoscopic image and the macroscopic image, respectively, to determine edge positions of the sample lesion areas;

[0052] The coordinates of a pixel point at the edge position of the sample lesion area are obtained, and the pixel point is used as the starting point. The pixel points at the edge position are connected by calling the region growing algorithm to generate an endoscopic lesion image and a macroscopic lesion image respectively.

[0053] In one embodiment, the modeling module is specifically used to:

[0054] Inputting the endoscopic lesion image, the macroscopic lesion image and the pathological analysis result into the neural network model, and obtaining a multi-layer two-dimensional image of the lesion tissue according to the endoscopic lesion image and the macroscopic lesion image;

[0055] Analytically transforming and calculating the multi-layer two-dimensional images to obtain the coronal image and sagittal image of the diseased tissue;

[0056] A three-dimensional sample reconstruction model is formed according to the tube-pile position map and the sagittal position map of the diseased tissue.

[0057] In a third aspect, an embodiment of the present application provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the surgical sample image analysis method described in the first aspect when executing the computer program.

[0058] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions, and when the instructions are executed on a computer, the computer executes the surgical sample image analysis method as described in the first aspect.

[0059] The present application reconstructs a three-dimensional model of a surgical sample based on endoscopic lesion images and macroscopic lesion images to obtain a three-dimensional model with better image quality, which is convenient for subsequent comparison with a pathology analysis report. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 is a schematic diagram of a scenario of a surgical sample image analysis device provided in an embodiment of the present application;

[0062] Figure 2It is a schematic diagram of an embodiment of the flow chart of the surgical sample image analysis method provided in the embodiments of the present application;

[0063] Figure 3 This is a schematic diagram of the structure of a surgical sample image analysis method and device according to an embodiment of the present application;

[0064] Figure 4 A schematic diagram of the structure of a surgical sample image analysis device according to an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the structure of a mobile phone in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of the structure of a server in an embodiment of the present application.

[0067] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0069] In the following description, the specific embodiments of the present application will be described with reference to the steps and symbols performed by one or more computers, unless otherwise specified. Therefore, these steps and operations will be mentioned several times as being performed by a computer, and the computer execution referred to herein includes the operation of a computer processing unit by an electronic signal representing data in a structured form. This operation converts the data or maintains it at a location in the memory system of the computer, which can be reconfigured or otherwise change the operation of the computer in a manner familiar to testers in the field. The data structure maintained by the data is a physical location in the memory, which has specific characteristics defined by the data format. However, the principles of the present application are described in the above text, which does not represent a limitation, and testers in the field will understand that the various steps and operations described below can also be implemented in hardware.

[0070] The terms "module" or "unit" used herein may be considered as software objects executed on the computing system. The different components, modules, engines and services described herein may be considered as implementation objects on the computing system. The devices and methods described herein are preferably implemented in software, but may also be implemented in hardware, all within the scope of protection of this application.

[0071] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0072] Embodiments of the present application provide a surgical sample image analysis method, device, and storage medium.

[0073] See also Figure 1 , Figure 1 This is a scene diagram of a surgical sample image analysis device provided in an embodiment of the present application, which may include a surgical sample image analysis system 100 and a user terminal 200. The surgical sample image analysis system 100 is connected via a network, and a surgical sample image analysis device is integrated in the surgical sample image analysis system 100. In the embodiment of the present application, the surgical sample image analysis system 100 may be a terminal device or a server, and the surgical sample image analysis system 100 may send a three-dimensional sample reconstruction model, etc. to the user terminal 200.

[0074] In the embodiment of the present application, when the surgical sample image analysis system 100 is a server, the server can be an independent server, or a server network or server cluster composed of servers. For example, the server described in the embodiment of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of a plurality of servers. Among them, the cloud server is composed of a large number of computers or network servers based on cloud computing (Cloud Computing). In the embodiment of the present application, communication between the server and the client can be achieved by any communication method, including but not limited to, mobile communication based on the third generation partnership project (3rd Generation Partnership Project, 3GPP), long term evolution (Long Term Evolution, LTE), Worldwide Interoperability for Microwave Access (Worldwide Interoperability for Microwave Access, WiMAX), or computer network communication based on TCP / IP protocol suite (TCP / IP Protocol suite, TCP / IP), User Datagram Protocol (User Datagram Protocol, UDP) protocol, etc.

[0075] It is understandable that when the surgical sample image analysis system 100 used in the embodiment of the present application is a terminal device, the terminal device can be a device that includes both receiving hardware and transmitting hardware, that is, a device with receiving and transmitting hardware that can perform two-way communication on a two-way communication link. Such terminal devices may include: cellular or other communication devices, which have a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific surgical sample image analysis system 100 can be a desktop terminal or a mobile terminal, and the surgical sample image analysis system 100 can be one of a mobile phone, a tablet computer, a laptop computer, etc.

[0076] The terminal device involved in the embodiments of the present application may also refer to a device that provides voice and / or data connectivity to a user, a handheld device with a wireless connection function, or other processing devices connected to a wireless modem. For example, a mobile phone (or "cellular" phone) and a computer with a mobile terminal, for example, a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device, which exchanges voice and / or data with a wireless access network. For example, personal communication service (English full name: Personal Communication service, English abbreviation: PCs) phone, cordless phone, session initiation protocol (sIP) phone, wireless local loop (Wireless Local Loop, English abbreviation: WLL) station, personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA) and other devices.

[0077] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer computing devices, or computing device network connection relationships, such as Figure 1 Only one computing device is shown. It can be understood that the surgical sample image analysis device can also include one or more other computing devices, or / and one or more other computing devices connected to the surgical sample image analysis system 100 network, which is not specifically limited here.

[0078] In addition, if Figure 1 As shown, the surgical sample image analysis device may further include a memory 300 for storing data, such as storing program endoscope images and pathological analysis results.

[0079] It should be noted that Figure 1 The scenario diagram of the surgical sample image analysis device shown is merely an example. The surgical sample image analysis device and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the surgical sample image analysis device and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0080] This application implements intelligent surgical sample image analysis based on different component packages and installation information, matches personalized component packages and realizes integrated installation, simplifies the installation process and improves reliability.

[0081] In this embodiment, description will be made from the perspective of a surgical sample image analysis method, which may be specifically integrated into a surgical sample image analysis system 100 .

[0082] The present application provides a surgical sample image analysis method, which includes: obtaining an endoscopic image of a target user before tissue sampling; after sampling the target user's tissue and color-marking the sampled tissue, obtaining a macroscopic image of the sampled tissue after color marking; preprocessing the endoscopic image and the macroscopic image to obtain a standardized endoscopic image and a macroscopic image; extracting common feature points from the standardized endoscopic image and the macroscopic image; according to the extracted common feature points, performing spatial transformation on the standardized endoscopic image to align the standardized endoscopic image and the standardized macroscopic image; identifying and locating the lesion area in the standardized endoscopic image and the macroscopic image; segmenting the lesion area to generate an endoscopic lesion image and a macroscopic lesion image; after obtaining a slice specimen corresponding to the lesion area, performing pathological analysis on the slice specimen image corresponding to the lesion area to obtain a pathological analysis result; and constructing a three-dimensional sample reconstruction model using a neural network model according to the endoscopic lesion image and the macroscopic lesion image.

[0083] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a surgical sample image analysis method in an embodiment of the present application. The surgical sample image analysis method includes the following steps 201 to 209:

[0084] 201. Obtain an endoscopic image of a target user before tissue sampling.

[0085] Specifically, early gastric cancer (EGC) is classified as a tumor confined to the mucosa or submucosa and is not associated with lymph node metastasis. Two types of magnifying endoscopes are widely used clinically: narrow-band imaging of magnifying endoscopy (NBI-ME) and blue laser imaging of magnifying endoscopy (BLI-ME) for accurate diagnosis and evaluation. Specifically, endoscopic images of the target user before tissue sampling are collected. Optionally, the lesion boundary result is obtained by determining whether there is a lesion with a clear demarcation line in the lesion tissue based on the endoscopic image. If the lesion boundary result indicates that there is a lesion without a clear demarcation line in the lesion tissue, NBI-ME is used for diagnosis to determine whether it is a poorly differentiated lesion or a marker cell lesion, and the NBI-ME diagnosis result is obtained. Determine whether it is necessary to biopsy the lesion tissue based on the NBI-ME diagnosis result. Specifically, for lesions without clear demarcation lines, especially lesions diagnosed by NBI-ME as poorly differentiated or marker cell lesions, a biopsy is performed before endoscopic submucosal dissection (ESD) to determine the cancer margin; for cancer lesions diagnosed by NBI-ME as well-differentiated and with clear demarcation lines, no biopsy is performed. If it is determined that a biopsy is required, after the biopsy, the resected spike specimen and the tumor margin are post-processed by NBI / BLI-ME, and the differentiation of certain areas is carefully evaluated. By comparing the preoperative and postoperative NBI / BLI-ME images, the malignant or suspected cancerous areas on the specimen are confirmed by the preoperative and postoperative NBI / BLI-ME images. Optionally, the format of the endoscopic image is a medical image standard format (Digital Imaging and Communications in Medicine, DICOM). Optionally, after the endoscopic image is acquired, a DICOM library or software package can be used to read the DICOM image, and the metadata and pixel data of the endoscopic image DICOM file can be read.

[0086] 202. After sampling tissue of the target user and color-marking the sampled tissue, a macroscopic image of the sampled tissue after color marking is obtained.

[0087] Further, after confirming the malignant or suspected cancerous areas on the specimens by preoperative and postoperative NBI / BLI-ME images, the endoscopist marked the specimens (i.e., excised tissues) with different colors (green, yellow, orange, red, and blue) to show different endoscopic manifestations, including surface microstructure (MS) and microvascular (MV) morphological changes on tissue-impregnated dried specimens. The targeted labeled specimens were then fixed in 10% buffered formalin solution for further histological processing. The specimens were histologically evaluated by hematoxylin and eosin staining, and lymphatic vessel invasion was detected by immunohistochemical staining. Macroscopic images of the sampled tissues after color marking were obtained. Optionally, the specimens were cut into 2 mm thick slices, and the differentiation, lymphatic vessel invasion, and cancer depth of the first slice were evaluated for each slice. Similarly, the format of the macroscopic images is also the medical image standard format (Digital Imaging and Communications in Medicine, DICOM).

[0088] 203. Preprocess the endoscopic image and the macroscopic image to obtain standardized endoscopic image and macroscopic image.

[0089] Specifically, after obtaining the endoscopic image before tissue sampling and the macroscopic image of the sampled tissue marked with color after tissue sampling, the endoscopic image and macroscopic image in DICOM file format are read. Optionally, the endoscopic image and macroscopic image can be preprocessed by calling some image processing open source libraries to obtain standardized endoscopic images and macroscopic images. Among them, the standardized endoscopic image and macroscopic image can be an endoscopic image and macroscopic image of a preset size, preset resolution and color space, and preset format. Optionally, the preprocessing can include contrast enhancement, noise removal, image smoothing and image enhancement, etc., to improve image quality. Optionally, images from different sources can also be unified to the same size, resolution and color space, that is, image standardization is performed to facilitate subsequent processing.

[0090] 204. Extracting common feature points from standardized endoscopic and macroscopic images.

[0091] Optionally, common feature points or structures, such as the edge of the lesion, blood vessel distribution, etc., are extracted from the standardized endoscopic image and the macroscopic image. Optionally, feature points in the standardized endoscopic image and the macroscopic image can be extracted respectively, and common feature points in the endoscopic image and the macroscopic image are obtained by comparison. Optionally, a feature detector in the image processing library can be used to detect feature points in the endoscopic image and the macroscopic image respectively using a feature point detection algorithm. Optionally, a feature detector is called to detect the first feature point in the standardized endoscopic image and the second feature point in the macroscopic image respectively using a feature point detection algorithm, and there are multiple first feature points and second feature points respectively; the descriptor calculation algorithm is used to calculate the descriptors of the first feature point and the second feature point, and the descriptors of the first feature point and the second feature point are respectively data sets containing the image information of the first feature point and the second feature point; according to the descriptors of the first feature point and the second feature point, the common feature points in the standardized endoscopic image and the macroscopic image are determined. Specifically, for example, an ORB feature detector can be called to extract common feature points from the standardized endoscopic image and the macroscopic image. Specifically, in the ORB feature detector, the FAST algorithm is called to determine the key points by comparing the brightness difference between the pixel and the surrounding pixels. The BRIEF algorithm is then used to calculate the descriptor of each key point. The descriptor is a data set of image information. The key points are then drawn according to the descriptor through the drawkeypoints function to complete the extraction of image feature points. Optionally, the ORB feature detector is used to extract feature points in the endoscopic image and the macroscopic image, respectively, to obtain the first feature point and the second feature point. It can be understood that, here, the first feature point and the second feature point are only distinguished without other limitations.

[0092] 205. Perform spatial transformation on the standardized endoscopic image according to the extracted common feature points to align the standardized endoscopic image with the standardized macroscopic image.

[0093] Specifically, common image registration methods include feature-based registration and similarity-based registration. After obtaining the extracted common feature points, the feature points can be used to realize image registration according to the registration algorithm in the image processing library, that is, to realize the standardized alignment of the endoscopic image and the macroscopic image. Specifically, according to the extracted feature points, the endoscopic image is translated, rotated, scaled and other transformations are performed to make it as consistent as possible with the macroscopic image in space and morphology. Specifically, the purpose of image registration is to make the endoscopic image achieve spatial consistency with the feature points in the macroscopic image through spatial transformation. Specifically, the feature points can be pathological structure points in the endoscopic image and the macroscopic image, and registration is to make the pathological structure points in the endoscopic image and the macroscopic image have the same spatial position.

[0094] 206. Identify and locate the diseased area in the standardized endoscopic image and macroscopic image.

[0095] Specifically, the endoscopic image and the macroscopic image are combined to identify and locate the diseased area. Optionally, the macroscopic image is a macroscopic image of the sampled tissue after color marking, and the diseased area in the macroscopic image can be confirmed by identifying the color mark. Optionally, in the endoscopic image, the diseased area in the endoscopic image can be determined by identifying the edge of the diseased area. Optionally, an edge detection algorithm can be called in the image processing library to identify the edges of the diseased area in the endoscopic image and the macroscopic image, such as the canny algorithm.

[0096] 207. Segment the lesion area to generate an endoscopic lesion image and a macroscopic lesion image.

[0097] Specifically, the lesion area is segmented from the background using image processing techniques (such as threshold segmentation, region growing, etc.). Optionally, the image processing techniques that can be used include: threshold segmentation, edge detection, and region growing, etc. Taking the region growing algorithm as an example, the edge coordinates of the lesion area are extracted using the edge detection algorithm, and the coordinates of all edge positions are connected with one edge coordinate of the lesion area as the starting point, and the region growing algorithm is called to obtain the segmented endoscopic lesion image and macroscopic lesion image.

[0098] 208. After obtaining the slice specimen corresponding to the lesion area, pathological analysis is performed on the slice specimen image corresponding to the lesion area to obtain a pathological analysis result.

[0099] Specifically, the lesion area is sliced ​​to prepare a tissue slice specimen. The slice specimen is observed under a microscope to perform pathological analysis such as cell morphology and tissue structure. Specific immunohistochemical staining is performed as needed to detect specific molecular markers. After the slice specimen is obtained, a slice specimen image is obtained, and pathological analysis is performed according to the slice specimen image corresponding to the lesion area to obtain a pathological analysis result. Optionally, the process of performing pathological analysis according to the slice specimen image can be completed by a neural network model based on image and language. Specifically, a training data set of paired pathological images and natural language descriptions is used to train the pathological analysis neural network model. The pathological analysis model generates two embedding vectors from a text encoder and an image encoder. Through contrast learning, the two embedding vectors are optimized to be similar between matching images and text vectors, and dissimilar between non-matching images and texts, thereby training to obtain the final pathological analysis model. The slice specimen image obtained according to the slice specimen, that is, the pathological image, is input into the pathological analysis model to obtain the pathological analysis result. For example: the pathological image is input into the pathological analysis model, and the model outputs a malignant tumor. The pathological analysis results output by the model are integrated into a pathological analysis report to facilitate subsequent three-dimensional reconstruction of the sample based on the pathological analysis report.

[0100] 209. Construct a three-dimensional sample reconstruction model using a neural network model based on the endoscopic lesion image and the macroscopic lesion image.

[0101] Specifically, the endoscopic image, macroscopic image and pathological analysis results are integrated into a three-dimensional space. Optionally, a neural network model can be used to construct a three-dimensional sample reconstruction model, such as a convolutional neural network. Specifically, the endoscopic image, macroscopic image and pathological analysis report are input into the convolutional neural network, and a multi-layer two-dimensional image is generated according to the convolutional neural network. According to the multi-layer two-dimensional image analysis and conversion of the lesion tissue, the coronal image and sagittal image of the lesion tissue are generated, and a three-dimensional sample reconstruction model is generated according to the coronal image and sagittal image of the lesion tissue. It can be understood that the coronal image of the lesion tissue refers to the image on the coronal axis of the lesion tissue. The coronal axis is also called the frontal axis, which refers to the axis parallel to the bottom surface in the left-right direction and perpendicular to the sagittal position. The sagittal image of the lesion tissue refers to the image on the sagittal axis of the lesion tissue. The sagittal axis refers to the axis parallel to the bottom surface in the front-back direction and perpendicular to the coronal position. The three basic axes perpendicular to each other in the human body are: the coronal axis, the sagittal axis and the vertical axis. Optionally, a three-dimensional modeling software (such as Mimics, 3D Slicer, ArtecStudio, etc.) can be used to construct a three-dimensional reconstruction model based on the morphology, structure and pathological characteristics of the lesion. Specifically, the sagittal and coronal images of the lesion tissue are input into the three-dimensional modeling software, and the software will reconstruct the vertical axis image based on the sagittal and coronal images, thereby reconstructing the sample three-dimensional model based on the sagittal, coronal and vertical axis images. Optionally, after obtaining the sample three-dimensional model, the three-dimensional reconstruction model can also be optimized and adjusted to ensure its consistency with the endoscopic image and macroscopic image, and necessary verification can be performed.

[0102] The present application reconstructs a three-dimensional model of a surgical sample based on endoscopic lesion images and macroscopic lesion images to obtain a three-dimensional model with better image quality, which is convenient for subsequent comparison with a pathology analysis report.

[0103] In one embodiment of the present application, the macroscopic image of the color-marked sampled tissue is obtained by processing the sampled tissues with different lesion structures marked with different colors using a magnifying endoscopic narrow-band imaging technique.

[0104] Specifically, the sampled tissues were marked with different colors (green, yellow, orange, red, and blue) to show different endoscopic manifestations, including surface microstructure (MS) and microvascular (MV) morphological changes on tissue-impregnated dried specimens. The targeted labeled specimens were then fixed in 10% buffered formalin solution for further histological processing. The specimens were stained with hematoxylin and eosin for histological evaluation, and immunohistochemical staining was used to detect lymphatic vessel invasion. After color marking was completed, the sampled tissues were processed using magnifying endoscopic narrow-band imaging technology to obtain macroscopic images.

[0105] The embodiment of the present application marks the structure of the sampled tissue by color marking, thereby obtaining a macroscopic image of the sampled tissue with a clear structure.

[0106] In one embodiment of the present application, the preprocessing includes contrast enhancement, noise removal and size unification, and the preprocessing of the endoscopic image and the macroscopic image to obtain a standardized endoscopic image and macroscopic image includes:

[0107] An image enhancement algorithm is called to perform image enhancement processing on the endoscopic image and the macroscopic image to obtain enhanced endoscopic image and macroscopic image; a Gaussian filter is called to perform noise removal processing on the enhanced endoscopic image and the macroscopic image to obtain denoised endoscopic image and macroscopic image; the denoised endoscopic image and the macroscopic image are cropped to make the sizes of the denoised endoscopic image and the macroscopic image uniform to obtain standardized endoscopic image and macroscopic image.

[0108] Specifically, the Retinex algorithm can be called in the image processing library to perform image enhancement processing on the endoscopic image and the macroscopic image. The Retinex algorithm can restore the reflection component of the illumination in the endoscopic image and the macroscopic image, and adjust the distribution component of the image by estimating the illumination component of the image to obtain the enhanced image. After image enhancement, the endoscopic image and the macroscopic image can also be denoised. Optionally, the Gaussian filter can be called to perform denoising processing on the enhanced endoscopic image and the macroscopic image. The Gaussian filter uses the Gaussian function as a weight to smooth the endoscopic image and the macroscopic image, thereby reducing noise. Optionally, the denoising process can also call other methods such as: mean filter, wavelet denoising algorithm and median filter. It can be understood that the endoscopic image and the macroscopic image can be enhanced and denoised at the same time, or the macroscopic image can be enhanced and denoised after the endoscopic image is enhanced and denoised, which is not limited here. After the denoising process is completed, in order to subsequently perform three-dimensional reconstruction of the sample based on the endoscopic image and the macroscopic image, it is necessary to unify the sizes of the endoscopic and macroscopic images to obtain standardized endoscopic and macroscopic images.

[0109] The embodiment of the present application pre-processes the endoscopic image and the macroscopic image to improve the image quality and unify the image size, thereby facilitating the subsequent three-dimensional reconstruction of the sample tissue using the endoscopic image and the macroscopic image.

[0110] In one embodiment of the present application, extracting common feature points from the standardized endoscopic image and the macroscopic image includes:

[0111] Call a feature detector and use a feature point detection algorithm to respectively detect a first feature point in the standardized endoscopic image and a second feature point in the macroscopic image, where there are multiple first feature points and multiple second feature points respectively; use a descriptor calculation algorithm to calculate descriptors of the first feature point and the second feature point, where the descriptors of the first feature point and the second feature point are data sets containing image information of the first feature point and the second feature point respectively; determine common feature points in the standardized endoscopic image and the macroscopic image based on the descriptors of the first feature point and the second feature point.

[0112] Optionally, the ORB feature detector can be called to extract common feature points from standardized endoscopic images and macroscopic images. Specifically, in the ORB feature detector, the FAST algorithm is called to determine the key points by comparing the brightness difference between the pixel and the surrounding pixels. The BRIEF algorithm is then used to calculate the descriptor of each key point. The descriptor is a data set of image information. The key points are then drawn according to the descriptor through the drawkeypoints function to complete the extraction of image feature points. Optionally, the feature points in the endoscopic image and the macroscopic image are extracted using the ORB feature detector respectively to obtain the first feature point and the second feature point.

[0113] The embodiment of the present application extracts feature points in the endoscopic image and the macroscopic image through a feature detector, so as to facilitate subsequent registration and alignment of the images based on the feature points.

[0114] In one embodiment of the present application, the step of spatially transforming the standardized endoscopic image according to the extracted common feature points so as to align the standardized endoscopic image with the standardized macroscopic image comprises:

[0115] The standardized endoscopic image and the macroscopic image are matched with feature points by using a matching algorithm to obtain matching points of the standardized endoscopic image and the macroscopic image; the coordinates of the matching points are obtained; the homography matrix between the matching points is calculated according to the coordinates of the matching points; the standardized endoscopic image is perspective transformed to align it with the standardized macroscopic image according to the coordinates of the matching points.

[0116] Specifically, the registered image can be obtained by aligning the feature points between different images. Specifically, the KNN matching algorithm can be used to screen the matching points of the endoscopic image and the macroscopic image. Specifically, the feature points of the endoscopic image and the macroscopic image are screened out, and the feature points are matched one by one to screen out the matching matching points. After the matching points are screened out, the coordinates of the matching points are extracted, and the RANSAC algorithm is called to calculate the homography matrix between the matching points, and the endoscopic image is perspective transformed to align the endoscopic image with the macroscopic image.

[0117] The embodiment of the present application aligns the endoscopic image and the macroscopic image through feature point matching, so as to facilitate the subsequent reconstruction of the sample three-dimensional model using the endoscopic image and the macroscopic image.

[0118] In one embodiment of the present application, segmenting the lesion area to generate an endoscopic lesion image and a macroscopic lesion image includes:

[0119] The edge detection algorithm is called to extract the edges of the lesion areas of the standardized endoscopic image and the macroscopic image respectively, and the edge position of the sample lesion area is determined; the coordinates of a pixel point in the edge position of the sample lesion area are obtained, and the pixel point is used as the starting point, and the region growing algorithm is called to connect the pixel points at the edge position to generate an endoscopic lesion image and a macroscopic lesion image respectively.

[0120] Specifically, the canny edge detection algorithm is called to extract the edges of the endoscopic image and the macroscopic image, so as to determine the edge position of the sample lesion area. Then the lesion image is segmented using the threshold segmentation or region growing algorithm. Specifically, taking the region growing algorithm as an example, a pixel point is selected as the starting point from the edge points output by the edge detection algorithm, the coordinates of the pixel point are obtained, and the region growing algorithm is called to connect all the pixel points at the edge position to generate the endoscopic lesion image and the macroscopic lesion image, thereby realizing the segmentation of the lesion area image.

[0121] The embodiment of the present application detects the edge of the lesion area through an edge detection algorithm, and segments the lesion area from the image, so as to facilitate the subsequent three-dimensional reconstruction of the lesion area based on the image.

[0122] In one embodiment of the present application, the neural network model is a convolutional neural network model including multiple convolutional layers, and the three-dimensional sample reconstruction model is constructed using the neural network model according to the endoscopic lesion image and the macroscopic lesion image, including:

[0123] The endoscopic lesion image, the macroscopic lesion image and the pathological analysis result are input into the neural network model, and a multi-layer two-dimensional image of the lesion tissue is obtained according to the endoscopic lesion image and the macroscopic lesion image; the multi-layer two-dimensional image is analyzed, transformed and calculated to obtain the coronal image and sagittal image of the lesion tissue; and a three-dimensional sample reconstruction model is formed according to the tubular and sagittal image of the lesion tissue.

[0124] Optionally, a neural network model can be used to construct a three-dimensional sample reconstruction model, such as a convolutional neural network. Specifically, the endoscopic image, macroscopic image and pathological analysis report are input into the convolutional neural network, and a multi-layer two-dimensional image is generated according to the convolutional neural network. According to the multi-layer two-dimensional image analysis and conversion of the lesion tissue, the coronal image and sagittal image of the lesion tissue are generated, and a three-dimensional sample reconstruction model is generated according to the coronal image and sagittal image of the lesion tissue. It can be understood that the coronal image of the lesion tissue refers to the image on the coronal axis of the lesion tissue, and the coronal axis is also called the frontal axis, which refers to the axis parallel to the bottom surface in the left-right direction and perpendicular to the sagittal position. The sagittal image of the lesion tissue refers to the image on the sagittal axis of the lesion tissue, and the sagittal axis refers to the axis parallel to the bottom surface in the front-back direction and perpendicular to the coronal position. The three basic axes perpendicular to each other in the human body are: coronal axis, sagittal axis and vertical axis. Optionally, a three-dimensional reconstruction model can be constructed using three-dimensional modeling software (such as Mimics, 3D Slicer, Artec Studio, etc.) according to the morphology, structure and pathological characteristics of the lesion. Specifically, the sagittal and coronal images of the lesion tissue are input into the three-dimensional modeling software, and the software will reconstruct the vertical axis image based on the sagittal and coronal images, so as to reconstruct the sample three-dimensional model based on the sagittal, coronal and vertical axis images. Optionally, after obtaining the sample three-dimensional model, the three-dimensional reconstruction model can also be optimized and adjusted to ensure its consistency with the endoscopic image and macroscopic image, and necessary verification can be performed. Optionally, after obtaining the three-dimensional reconstruction model, the three-dimensional reconstruction model is compared and analyzed with the endoscopic image and macroscopic image to evaluate the integrity, morphological changes and pathological characteristics of the lesion. According to the analysis results, a detailed histopathological reconstruction report is written, including the description of the lesion, pathological diagnosis, three-dimensional reconstruction results, etc. The report can also be submitted to clinicians to obtain clinical feedback, and necessary modifications and improvements can be made based on the feedback.

[0125] In the embodiment of the present application, a three-dimensional model of the surgical sample is reconstructed based on the endoscopic lesion image and the macroscopic lesion image to obtain a three-dimensional model with higher image quality, which is convenient for subsequent comparison with the pathological analysis report.

[0126] In order to facilitate better implementation of the surgical sample image analysis method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above surgical sample image analysis method. The meanings of the terms are the same as those in the above surgical sample image analysis method, and the specific implementation details can refer to the description in the embodiment of the surgical sample image analysis method.

[0127] The surgical sample image analysis device in the embodiment of the present application has the function of implementing the surgical sample image analysis method provided in the above embodiment. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0128] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a surgical sample image analysis device provided in an embodiment of the present application. The surgical sample image analysis device can be applied to a computing device in a scenario where content push is required. Specifically, the surgical sample image analysis method device 300 may include a first acquisition module 301, a second acquisition module 302, a preprocessing module 303, an extraction module 304, a registration module 305, a recognition module 306, a segmentation module 307, an analysis module 308 and a reconstruction module 309, as follows:

[0129] The first acquisition module is used to acquire the endoscopic image of the target user before tissue sampling;

[0130] The second acquisition module is used for sampling the tissue of the target user and color-marking the sampled tissue, and then acquiring a macroscopic image of the sampled tissue after color marking;

[0131] Preprocessing module: used for preprocessing the endoscopic image and macroscopic image to obtain standardized endoscopic image and macroscopic image;

[0132] Extraction module: used to extract common feature points from standardized endoscopic images and macroscopic images;

[0133] A registration module: used for performing spatial transformation on the standardized endoscopic image according to the extracted common feature points so as to align the standardized endoscopic image with the standardized macroscopic image;

[0134] Identification module: used for identifying and locating the lesion area in the standardized endoscopic image and macroscopic image;

[0135] Segmentation module: used to segment the lesion area and generate endoscopic lesion images and macroscopic lesion images

[0136] Analysis module: used for performing pathological analysis on the slice specimen image corresponding to the lesion area after acquiring the slice specimen corresponding to the lesion area, so as to obtain the pathological analysis result;

[0137] Reconstruction module: used to construct a three-dimensional sample reconstruction model using a neural network model based on the endoscopic lesion image and the macroscopic lesion image.

[0138] In one embodiment, in the second acquisition module, the macroscopic image of the color-marked sampled tissue is obtained by processing the sampled tissues with different lesion structures marked with different colors using a magnifying endoscopic narrow-band imaging technique.

[0139] In one embodiment, the preprocessing includes contrast enhancement, noise removal and size unification, and the preprocessing module is specifically used for:

[0140] Calling an image enhancement algorithm to perform image enhancement processing on the endoscopic image and the macroscopic image to obtain enhanced endoscopic image and macroscopic image;

[0141] Calling a Gaussian filter to perform noise removal processing on the enhanced endoscopic image and macroscopic image to obtain a denoised endoscopic image and macroscopic image;

[0142] The denoised endoscopic image and macroscopic image are cropped to make the sizes of the denoised endoscopic image and macroscopic image uniform, thereby obtaining standardized endoscopic image and macroscopic image.

[0143] In one embodiment, the extraction module is specifically used to:

[0144] Calling a feature detector to detect a first feature point in the standardized endoscopic image and a second feature point in the macroscopic image using a feature point detection algorithm, wherein there are a plurality of the first feature point and a plurality of the second feature point;

[0145] Calculate the descriptors of the first feature point and the second feature point by using a descriptor calculation algorithm, wherein the descriptors of the first feature point and the second feature point are respectively data sets containing image information of the first feature point and the second feature point;

[0146] Common feature points in the standardized endoscopic image and macroscopic image are determined according to the descriptors of the first feature points and the second feature points.

[0147] In one embodiment, the registration module is specifically used to:

[0148] Performing feature point matching on the standardized endoscopic image and the macroscopic image using a matching algorithm to obtain matching points between the standardized endoscopic image and the macroscopic image;

[0149] Obtaining the coordinates of the matching points;

[0150] Calculate the homography matrix between the matching points according to the coordinates of the matching points;

[0151] The standardized endoscopic image is perspective transformed and aligned with the standardized macroscopic image according to the coordinates of the matching points.

[0152] In one embodiment, the segmentation module is specifically used to:

[0153] Invoking an edge detection algorithm to extract edges of the lesion areas of the standardized endoscopic image and the macroscopic image, respectively, to determine edge positions of the sample lesion areas;

[0154] The coordinates of a pixel point at the edge position of the sample lesion area are obtained, and the pixel point is used as the starting point. The pixel points at the edge position are connected by calling the region growing algorithm to generate an endoscopic lesion image and a macroscopic lesion image respectively.

[0155] In one embodiment, the modeling module is specifically used to:

[0156] Inputting the endoscopic lesion image, the macroscopic lesion image and the pathological analysis result into the neural network model, and obtaining a multi-layer two-dimensional image of the lesion tissue according to the endoscopic lesion image and the macroscopic lesion image;

[0157] Analytically transforming and calculating the multi-layer two-dimensional images to obtain the coronal image and sagittal image of the diseased tissue;

[0158] A three-dimensional sample reconstruction model is formed according to the tube-pile position map and the sagittal position map of the diseased tissue.

[0159] The embodiment of the present application reconstructs a three-dimensional model of the surgical sample based on the endoscopic lesion image and the macroscopic lesion image to obtain a three-dimensional model with higher image quality, which is convenient for subsequent comparison with the pathology analysis report.

[0160] The above describes the surgical sample image analysis device in the embodiment of the present application from the perspective of modular functional entities. The following describes the surgical sample image analysis device in the embodiment of the present application from the perspective of hardware processing.

[0161] It should be noted that Figure 3 The physical devices corresponding to the first acquisition module 301 and the second acquisition module 302 shown may be transceivers, radio frequency circuits, communication modules, and input / output (I / O) interfaces, etc., and the physical device corresponding to the preprocessing module 303 may be a processor.

[0162] Figure 4 The devices shown can all have Figure 3 The structure shown, when Figure 3 The surgical sample image analysis method and device shown in the figure has the following features: Figure 4 When the structure shown is Figure 4 The processor and transceiver in the device can realize the same or similar functions as the first acquisition module 301 and the pre-processing module 303 provided in the device embodiment corresponding to the device. Figure 4The memory in the storage device stores a computer program that needs to be called when the processor executes the above-mentioned surgical sample image analysis method.

[0163] When the computing device in the embodiment of the present application is a terminal device, the embodiment of the present application also provides a terminal device, such as Figure 5 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal device can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POs), a car computer, etc., taking the terminal device as a mobile phone as an example:

[0164] Figure 5 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Figure 5 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art will understand that Figure 5 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0165] Combine the following Figure 5 A detailed introduction to the various components of the mobile phone:

[0166] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global system of mobile communication (Global system of mobile communication, GSM), General Packet Radio service (General Packet Radio service, GPRs), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, short messaging service (sMs), etc.

[0167] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0168] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0169] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 5 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0170] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0171] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0172] Wi-Fi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the Wi-Fi module 1070. It provides users with wireless broadband Internet access. Figure 5 A Wi-Fi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0173] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020, so as to monitor the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; optionally, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0174] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 1080 through a power management system, thereby managing functions such as charging, discharging, and power consumption management through the power management system.

[0175] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0176] In the embodiment of the present application, the processor 1080 included in the mobile phone also has the function of controlling and executing the surgical sample image analysis method process executed by the surgical sample image analysis method device.

[0177] The present application also provides a server. Figure 6 , Figure 6 1 is a schematic diagram of a server structure provided in an embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (full name in English: central processing units, abbreviated in English: CPU) 1122 (for example, one or more processors) and memory 1132, one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 can be short-term storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1122 can be configured to communicate with the storage medium 1130 to execute a series of instruction operations in the storage medium 1130 on the server 1100.

[0178] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows server, Mac Os X, Unix, Linux, FreeBsD, etc.

[0179] The steps in the surgical sample image analysis method in the above embodiment can be based on the Figure 6 The structure of the server 1100 is shown.

[0180] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0182] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0183] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0185] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0186] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DsL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state hard disk solid state Disk (ssD)), etc.

[0187] The technical solutions provided in the embodiments of the present application are introduced in detail above. The principles and implementation methods of the embodiments of the present application are explained by using specific examples in the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

Claims

1. A surgical sample image analysis method, characterized in that: The surgical sample image analysis method comprises: Acquiring an endoscopic image of the target user before tissue sampling; After sampling the tissue of the target user and color-marking the sampled tissue, a macroscopic image of the sampled tissue after color marking is obtained; Preprocessing the endoscopic image and the macroscopic image to obtain standardized endoscopic image and macroscopic image; Extracting common feature points from standardized endoscopic and macroscopic images; performing spatial transformation on the standardized endoscopic image according to the extracted common feature points to align the standardized endoscopic image with the standardized macroscopic image; identifying and locating lesion areas in the standardized endoscopic images and macroscopic images; Segmenting the lesion area to generate an endoscopic lesion image and a macroscopic lesion image; After obtaining the slice specimen corresponding to the lesion area, pathological analysis is performed on the slice specimen image corresponding to the lesion area to obtain a pathological analysis result; A three-dimensional sample reconstruction model is constructed using a neural network model according to the endoscopic lesion image and the macroscopic lesion image.

2. The surgical sample image analysis method according to claim 1, characterized in that: The macroscopic image of the color-marked sampled tissue is obtained by processing the sampled tissues with different pathological structures marked with different colors using a magnifying endoscope narrow-band imaging technique.

3. The surgical sample image analysis method according to claim 1, characterized in that: The preprocessing includes contrast enhancement, noise removal and size unification, and the preprocessing of the endoscopic image and the macroscopic image to obtain a standardized endoscopic image and macroscopic image includes: Calling an image enhancement algorithm to perform image enhancement processing on the endoscopic image and the macroscopic image to obtain enhanced endoscopic image and macroscopic image; Calling a Gaussian filter to perform noise removal processing on the enhanced endoscopic image and macroscopic image to obtain a denoised endoscopic image and macroscopic image; The denoised endoscopic image and macroscopic image are cropped to make the sizes of the denoised endoscopic image and macroscopic image uniform, thereby obtaining standardized endoscopic image and macroscopic image.

4. The surgical sample image analysis method according to claim 1, characterized in that: The method of extracting common feature points from the standardized endoscopic image and the macroscopic image comprises: Calling a feature detector to detect a first feature point in the standardized endoscopic image and a second feature point in the macroscopic image using a feature point detection algorithm, wherein there are a plurality of the first feature point and a plurality of the second feature point; Calculate the descriptors of the first feature point and the second feature point by using a descriptor calculation algorithm, wherein the descriptors of the first feature point and the second feature point are respectively data sets containing image information of the first feature point and the second feature point; Common feature points in the standardized endoscopic image and macroscopic image are determined according to the descriptors of the first feature points and the second feature points.

5. The surgical sample image analysis method according to claim 1, characterized in that: The step of spatially transforming the standardized endoscopic image according to the extracted common feature points so as to align the standardized endoscopic image with the standardized macroscopic image comprises: Performing feature point matching on the standardized endoscopic image and the macroscopic image using a matching algorithm to obtain matching points between the standardized endoscopic image and the macroscopic image; Obtaining the coordinates of the matching points; Calculate the homography matrix between the matching points according to the coordinates of the matching points; The standardized endoscopic image is perspective transformed and aligned with the standardized macroscopic image according to the coordinates of the matching points.

6. The surgical sample image analysis method according to claim 1, characterized in that: The step of segmenting the lesion area to generate an endoscopic lesion image and a macroscopic lesion image includes: Invoking an edge detection algorithm to extract edges of the lesion areas of the standardized endoscopic image and the macroscopic image, respectively, to determine edge positions of the sample lesion areas; The coordinates of a pixel point at the edge position of the sample lesion area are obtained, and the pixel point is used as the starting point. The pixel points at the edge position are connected by calling the region growing algorithm to generate an endoscopic lesion image and a macroscopic lesion image respectively.

7. The surgical sample image analysis method according to claim 1, characterized in that: The neural network model is a convolutional neural network model including a plurality of convolutional layers. The three-dimensional sample reconstruction model is constructed using the neural network model according to the endoscopic lesion image and the macroscopic lesion image, including: Inputting the endoscopic lesion image, the macroscopic lesion image and the pathological analysis result into the neural network model, and obtaining a multi-layer two-dimensional image of the lesion tissue according to the endoscopic lesion image and the macroscopic lesion image; Analytically transforming and calculating the multi-layer two-dimensional images to obtain the coronal image and sagittal image of the diseased tissue; A three-dimensional sample reconstruction model is formed according to the tube-pile position map and the sagittal position map of the diseased tissue.

8. A surgical sample image analysis device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the surgical sample image analysis method as claimed in any one of claims 1 to 7 when executing the computer program.

9. A computing device, characterized in that It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the surgical sample image analysis method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the surgical sample image analysis method according to any one of claims 1 to 7.

Images