Pathological image processing method, model training method and device based on deep learning

The deep learning-based pathological image processing method addresses inaccuracies in lymph node metastasis prediction by extracting and fusing features from pathological images, improving efficiency and accuracy.

CN111462036BActive Publication Date: 2024-12-03TENCENT TECHNOLOGY (SHENZHEN) CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010100257.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-02-18
Publication Date
2024-12-03
Estimated Expiration
2040-02-18

AI Technical Summary

Technical Problem

Current methods for predicting lymph node metastasis in colorectal cancer are prone to experimental and operational errors due to uncontrollable factors, leading to inaccurate results.

Method used

A pathological image processing method using deep learning that involves acquiring and processing pathological images to extract and fuse features at different levels, including block image morphology, cell nucleus morphology, and nucleus position, to predict metastasis risk.

Benefits of technology

This approach reduces the time for test results and enhances detection efficiency while avoiding errors, providing more accurate metastasis predictions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111462036B_ABST
    Figure CN111462036B_ABST
Patent Text Reader

Abstract

The present application discloses a pathological image processing method based on deep learning, including: obtaining a pathological image of a target object to be processed; performing block processing on the pathological image of the target object to be processed to obtain a block image set; performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets; generating a target fusion feature corresponding to the target object; calling a pathological image analysis model to analyze and process the target fusion feature to output a pathological analysis result of the target object to be processed. The present application also discloses a model training method. The present application can directly use pathological images to predict the risk of lesion metastasis. Compared with biological detection technology, it can reduce the time waiting for detection results, improve detection efficiency, and avoid errors caused by uncontrollable factors such as experimental errors and operational errors, thereby providing more accurate detection results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a pathological image processing method, model training method and device based on deep learning. Background Art

[0002] As a malignant tumor, colorectal cancer has a high morbidity and mortality rate. In patients with colorectal cancer, as the primary cancer cells infiltrate and break through the intestinal wall layer by layer, they may infiltrate the lymph nodes near the intestine and further metastasize to distant lymph nodes through the lymphatic system. Before colorectal cancer resection surgery, it is necessary to accurately predict lymph node metastasis to assist doctors in preoperative decision-making and determine whether to clear the intestinal lymph nodes near the cancer area.

[0003] At present, the technology for predicting lymph node metastasis in colorectal cancer can be used to predict lymph node metastasis. It is mainly based on the biomarkers screened and verified in the previous experiments, to establish an in vitro detection kit, extract the patient's blood, and detect the corresponding ribonucleic acid (RNA) in the kit to determine the lymph node metastasis.

[0004] However, there are many uncontrollable factors in the use of the above-mentioned prediction technology for colorectal cancer lymph node metastasis. The kit includes experiments for extracting and isolating exosomes, reverse transcription, and fluorescent quantitative polymerase chain reaction. This series of experiments will vary depending on the hospital operator, resulting in large experimental and operational errors in the results. Summary of the invention

[0005] The embodiments of the present application provide a pathological image processing method, model training method and device based on deep learning, which can directly use pathological images to predict the risk of lesion metastasis. Compared with biological detection technology, it can reduce the time of waiting for test results and improve detection efficiency. At the same time, it can also avoid errors caused by uncontrollable factors such as experimental errors and operational errors, thereby providing more accurate detection results.

[0006] In view of this, the first aspect of the present application provides a pathological image processing method based on deep learning, comprising:

[0007] Acquiring a pathological image of a target object to be processed;

[0008] Performing block processing on the pathological image of the target object to be processed to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1;

[0009] Performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position;

[0010] The N first feature sets, the N second feature sets, and the N third feature sets that have been subjected to classification and extraction processing are subjected to feature fusion processing according to preset rules to generate a target fusion feature corresponding to the target object, wherein the fusion feature includes P elements, where P is an integer greater than 1;

[0011] The pathological image analysis model is called to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed.

[0012] The second aspect of the present application provides a model training method, comprising:

[0013] Acquire a set of pathological images to be trained, wherein the set of pathological images to be trained includes at least one positive sample pathological image and at least one negative sample pathological image;

[0014] For each pathological image to be trained in the pathological image set to be trained, a block image set to be trained is obtained, wherein the pathological image to be trained belongs to a positive sample pathological image or a negative sample pathological image, and the block image set to be trained includes N block images to be trained, where N is an image greater than or equal to 1;

[0015] For each pathological image to be trained in the set of pathological images to be trained, N first feature sets to be trained, N second feature sets to be trained, and N third feature sets to be trained are obtained according to the set of block images to be trained, wherein the first feature set to be trained includes at least one feature associated with the morphology of the block images to be trained, the second feature set to be trained includes at least one feature associated with the morphology of the cell nucleus, and the third feature set to be trained includes at least one feature associated with the position of the cell nucleus;

[0016] For each pathological image to be trained in the set of pathological images to be trained, generating a fusion feature to be trained according to N first feature sets to be trained, N second feature sets to be trained, and N third feature sets to be trained;

[0017] For the set of pathological images to be trained, determining the target fusion features to be trained according to the fusion features to be trained corresponding to each pathological image to be trained;

[0018] Based on the fusion features of the target to be trained corresponding to each pathological image to be trained, a loss function is used to train the pathological image analysis model to be trained to obtain a pathological image analysis model, wherein the pathological image analysis model is the model adopted in the first aspect above.

[0019] The third aspect of the present application provides a pathological image processing device based on deep learning, comprising:

[0020] An acquisition module, used for acquiring a pathological image of a target object to be processed;

[0021] The acquisition module is further used to perform block processing on the pathological image of the target object to be processed to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1;

[0022] A processing module, used for performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position;

[0023] A generation module, used for performing feature fusion processing on the N first feature sets, the N second feature sets and the N third feature sets after the classification and extraction processing according to a preset rule, to generate a target fusion feature corresponding to the target object, wherein the fusion feature includes P elements, and P is an integer greater than 1;

[0024] The output module is used to call the pathological image analysis model to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed.

[0025] In a possible design, in a first implementation of the third aspect of the embodiments of the present application,

[0026] A processing module, specifically used to obtain a hematoxylin-stained image set and an eosin-stained image set according to the block image set, wherein the hematoxylin-stained image set includes N hematoxylin-stained images, the eosin-stained image set includes N eosin-stained images, the hematoxylin-stained images have a corresponding relationship with the block images, and the eosin-stained images have a corresponding relationship with the block images;

[0027] Acquire a cell nucleus image set according to the block image set, wherein the cell nucleus image set includes N cell nucleus images, and the cell nucleus images have a corresponding relationship with the block images;

[0028] According to the hematoxylin staining image set and the eosin staining image set, obtaining N first feature sets corresponding to the block image set;

[0029] According to the cell nucleus image set, N second feature sets corresponding to the block image set are obtained;

[0030] According to the cell nucleus image set, N third feature sets corresponding to the block image set are obtained.

[0031] In a possible design, in a second implementation of the third aspect of the embodiments of the present application,

[0032] A processing module, specifically used to obtain the cell nucleus centroid position corresponding to each cell nucleus image according to the cell nucleus image set;

[0033] According to the position of the cell nucleus centroid corresponding to each cell nucleus image in the cell nucleus image set, a Voronoi diagram corresponding to each cell nucleus image is generated;

[0034] According to the Voronoi diagram corresponding to each cell nucleus image in the cell nucleus image set, N third feature sets are obtained.

[0035] In a possible design, in a third implementation of the third aspect of the embodiments of the present application,

[0036] A generation module, specifically used to obtain statistical results corresponding to T feature dimensions according to N first feature sets, N second feature sets, and N third feature sets, where T is an integer greater than 1;

[0037] Generate fusion features corresponding to the target object based on the statistical results corresponding to T feature dimensions;

[0038] Obtain Q elements from the fusion features corresponding to the target object, where Q is greater than or equal to 1;

[0039] The Q elements are concatenated to obtain the target fusion features corresponding to the target object.

[0040] In a possible design, in a fourth implementation of the third aspect of the embodiments of the present application,

[0041] A generating module, specifically used to perform statistics on the distribution of N block images in T feature dimensions according to N first feature sets, N second feature sets and N third feature sets, to obtain T frequency distribution histograms, wherein the feature dimensions and the frequency distribution histograms have a corresponding relationship;

[0042] According to each frequency distribution histogram in the T frequency distribution histograms, a statistical result corresponding to each frequency distribution histogram is generated;

[0043] According to the statistical results corresponding to each frequency distribution histogram, generate statistical results corresponding to T feature dimensions;

[0044] The statistical results corresponding to T feature dimensions are concatenated to obtain the fused features corresponding to the target object.

[0045] In a possible design, in a fifth implementation of the third aspect of the embodiments of the present application,

[0046] An output module, specifically used to call the pathological image analysis model to analyze and process the target fusion features, so as to output the lesion metastasis probability and at least one importance feature of the target object to be processed;

[0047] The processing module is further used to display the pathological analysis result after the output module calls the pathological image analysis model to analyze and process the target fusion feature to output the pathological analysis result of the target object to be processed;

[0048] or,

[0049] The pathological analysis result is sent to the client so that the client can display the pathological analysis result.

[0050] In a possible design, in a sixth implementation of the third aspect of the embodiments of the present application,

[0051] The acquisition module is further used to determine at least one target block image from the N block images according to the N first feature sets, the N second feature sets and the N third feature sets after the processing module performs feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets and N third feature sets;

[0052] The processing module is also used to highlight at least one target block image in the pathological image of the target object to be processed, or to send at least one target block image to the client so that the client can highlight at least one target block image in the pathological image of the target object to be processed.

[0053] In a possible design, in a seventh implementation of the third aspect of the embodiments of the present application,

[0054] The output module is also used for the processing module to process the pathological image of the target object to be processed into blocks to obtain a set of block images, and then call the molecular typing model to process the set of block images to output the molecular typing corresponding to each block image in the block images;

[0055] The processing module is further used to count the molecular typing corresponding to each block image in the block image set to obtain H statistical counting results, where H is an integer greater than 1;

[0056] The acquisition module is also used to determine the target molecular typing corresponding to the pathological image to be processed according to the H statistical counting results;

[0057] The acquisition module is also used to acquire a pathological image analysis model from a set of pathological image analysis models to be selected according to the target molecular typing, wherein the set of pathological image analysis models to be selected includes at least one pathological image analysis model to be selected, each pathological image analysis model to be selected corresponds to a type of molecular typing, and the pathological image analysis model corresponds to the target molecular typing.

[0058] In a possible design, in an eighth implementation of the third aspect of the embodiments of the present application,

[0059] An acquisition module is specifically used to extract a region of interest ROI from a pathological image of a target object to be processed;

[0060] A set of block images is obtained from the ROI area using the sliding window method.

[0061] A fourth aspect of the present application provides a model training device, comprising:

[0062] An acquisition module, used for acquiring a set of pathological images to be trained, wherein the set of pathological images to be trained includes at least one positive sample pathological image and at least one negative sample pathological image;

[0063] The acquisition module is further used to acquire a set of block images to be trained for each pathological image to be trained in the set of pathological images to be trained, wherein the pathological image to be trained belongs to a positive sample pathological image or a negative sample pathological image, and the set of block images to be trained includes M block images to be trained, where M is an image greater than or equal to 1;

[0064] The acquisition module is further used to acquire, for each pathological image to be trained in the pathological image set to be trained, M first feature sets to be trained, M second feature sets to be trained, and M third feature sets to be trained according to the block image set to be trained, wherein the first feature set to be trained includes at least one feature associated with the morphology of the block image to be trained, the second feature set to be trained includes at least one feature associated with the morphology of the cell nucleus, and the third feature set to be trained includes at least one feature associated with the position of the cell nucleus;

[0065] A generating module, for generating, for each pathological image to be trained in the pathological image set to be trained, fusion features to be trained according to the M first feature sets to be trained, the M second feature sets to be trained, and the M third feature sets to be trained;

[0066] A determination module, for determining, for the set of pathological images to be trained, target fusion features to be trained according to the fusion features to be trained corresponding to each pathological image to be trained;

[0067] The training module is used to train the pathological image analysis model to be trained based on the fusion features of the training target corresponding to each pathological image to be trained, using a loss function to obtain a pathological image analysis model, wherein the pathological image analysis model is a model adopted by the pathological image processing method based on deep learning.

[0068] A fifth aspect of the present application provides a terminal device, including: a memory, a transceiver, a processor, and a bus system;

[0069] Wherein, the memory is used to store programs;

[0070] The processor is used to execute the program in the memory, including the implementation method of any one of the above aspects;

[0071] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0072] A sixth aspect of the present application provides a server, including: a memory, a transceiver, a processor, and a bus system;

[0073] Wherein, the memory is used to store programs;

[0074] The processor is used to execute the program in the memory, including the implementation method of any one of the above aspects;

[0075] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0076] A seventh aspect of the present application provides a medical system, comprising: a probe, a circuit, a processor, and a display;

[0077] A circuit for stimulating the probe to obtain a pathological image of a target object to be processed;

[0078] A processor, for processing a pathological image of a target object to be processed;

[0079] A display for displaying the results of pathological analysis;

[0080] The processor further performs the following steps:

[0081] Performing block processing on the pathological image of the target object to be processed to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1;

[0082] Performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position;

[0083] The N first feature sets, the N second feature sets, and the N third feature sets that have been subjected to classification and extraction processing are subjected to feature fusion processing according to preset rules to generate a target fusion feature corresponding to the target object, wherein the fusion feature includes P elements, where P is an integer greater than 1;

[0084] The pathological image analysis model is called to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed.

[0085] The eighth aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the methods of the above aspects.

[0086] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0087] In an embodiment of the present application, a pathological image processing method based on deep learning is provided. First, a set of block images in a pathological image is obtained, and then a feature classification and extraction process is performed on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets. Based on the N first feature sets, N second feature sets, and N third feature sets after classification and extraction, feature fusion processing is performed according to preset rules to generate target fusion features corresponding to the target object. Finally, a pathological image analysis model is called to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed. In the above manner, the pathological image can be directly used to predict the risk of lesion metastasis based on the pathological image analysis model. Compared with biological detection technology, it can reduce the time waiting for the test results and improve the detection efficiency. At the same time, it can also avoid errors caused by uncontrollable factors such as experimental errors and operational errors, thereby providing more accurate test results. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 A schematic diagram of the architecture of a pathological image processing system in an embodiment of the present application;

[0089] Figure 2 A schematic diagram of a process of a pathological image processing method in an embodiment of the present application;

[0090] Figure 3Another schematic diagram of the process of pathological image processing in the embodiment of the present application;

[0091] Figure 4 A schematic diagram of an embodiment of a pathological image processing method based on deep learning in an embodiment of the present application;

[0092] Figure 5 A schematic diagram of a block image set obtained by segmenting a pathological image to be processed in an embodiment of the present application;

[0093] Figure 6 A schematic diagram of generating an associated image based on a block image in an embodiment of the present application;

[0094] Figure 7 A frequency distribution histogram diagram based on different features in an embodiment of the present application;

[0095] Figure 8 A schematic diagram of a pathological analysis result display interface in an embodiment of the present application;

[0096] Fig. 9 A schematic diagram of a target block image position display interface in an embodiment of the present application;

[0097] Fig.10 A schematic diagram of a process for image recognition based on a molecular typing model in an embodiment of the present application;

[0098] Fig.11 A schematic diagram of a process for model training based on a molecular typing model in an embodiment of the present application;

[0099] Fig.12 A schematic diagram of a process of performing image recognition based on a pathological image analysis model in an embodiment of the present application;

[0100] Fig.13 A schematic diagram of a process flow of a pathological image processing method based on deep learning in an embodiment of the present application;

[0101] Fig.14 This is a schematic diagram of an embodiment of the model training method in the embodiment of the present application;

[0102] Fig.15 Schematic diagram of an embodiment of a pathological image processing device based on deep learning in an embodiment of the present application;

[0103] Fig.16 This is a schematic diagram of an embodiment of a model training device in an embodiment of the present application;

[0104] Fig.17 A schematic diagram of the structure of a terminal device in an embodiment of the present application;

[0105] Fig.18 A schematic diagram of the structure of the server in the embodiment of the present application;

[0106] Fig.19 It is a structural schematic diagram of the medical system in an embodiment of the present application. DETAILED DESCRIPTION

[0107] The embodiments of the present application provide a pathological image processing method, model training method and device based on deep learning, which can directly use pathological images to predict the risk of lesion metastasis. Compared with biological detection technology, it can reduce the time of waiting for test results and improve detection efficiency. At the same time, it can also avoid errors caused by uncontrollable factors such as experimental errors and operational errors, thereby providing more accurate detection results.

[0108] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0109] It should be understood that the pathological image processing method based on deep learning provided in this application is applied to the medical field of artificial intelligence (AI), and can be specifically applied to the field of medical image recognition based on computer vision technology (CV). AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0110] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0111] CV Computer Vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify targets, trace and measure targets, and further process images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0112] With the rapid development of science and technology, the application of AI in the medical industry is becoming more and more extensive. The most common medical images in the medical field include but are not limited to endoscopic images, angiographic images, cardiovascular angiography images, computerized tomography (CT) images, B-ultrasound images and pathological images. Specifically, the pathological images involved in this application are biopsy pathological images collected by professional pathological scanners, such as whole slide images (WSI). Taking WSI images as an example, the side length of WSI images is usually between 10,000 pixels and 100,000 pixels. For WSI images, they often need to be scaled or cut into small-size images for further processing. In the process of processing the image, it is necessary to segment the area with pathological tissue slices.

[0113] Since pathological images directly reflect the lesions inside the tissue, they are an important basis for doctors to diagnose diseases, and even the ultimate basis for diagnosing certain diseases. For example, in the diagnosis of colorectal cancer, pathological images obtained by biopsy need to be used to predict the risk of lymph node metastasis. It is understandable that colorectal cancer, as a malignant tumor, has a high incidence and mortality rate. In patients with colorectal cancer, as the primary cancer cells infiltrate and break through the intestinal wall layer by layer, they may infiltrate the lymph nodes near the intestine and further metastasize to distant lymph nodes through the lymphatic system. This type of metastasis is the main cause of treatment failure and death in patients with colorectal cancer. Therefore, before colorectal cancer resection surgery, it is necessary to accurately predict lymph node metastasis to assist doctors in preoperative decision-making and determine whether to clear the intestinal lymph nodes near the cancer area. Accurate clearance of lymph node metastasis is a key link in reducing the recurrence and metastasis of colorectal cancer and improving prognosis. Blind excessive clearance may cause side effects such as edema. It is understandable that in practical applications, lymph node metastasis prediction can also be performed on other types of cancer, including but not limited to breast cancer, gastric cancer, liver cancer, and lung cancer, etc., which are not limited here.

[0114] The pathological image processing method provided in this application can be based on existing patient biopsy pathological images and the actual lymph node metastasis found during surgery as annotations, use big data for learning, and extract hidden information from biopsy pathological images for learning. Therefore, before the patient's operation, the risk of lymph node metastasis can be predicted based on the biopsy pathological images, effectively assisting preoperative decision-making. The biopsy pathological images used in this application are the "gold standard" for cancer diagnosis. They are easy to obtain clinically and do not require additional testing or data collection, so they can be widely used.

[0115] For ease of understanding, this application proposes a medical image recognition method, which is applied to Figure 1 Pathological image processing system shown in Figure 1 , Figure 1 This is a schematic diagram of the architecture of the pathological image processing system in the embodiment of the present application. As shown in the figure, the pathological image processing system may include medical equipment, which may be a pathological scanner endoscope device or an electron microscope. After the medical equipment acquires the WSI image, it can directly segment the WSI image to obtain N patch images, and then extract features at different levels from each patch image, perform fusion processing based on the features of these patch images at different levels to obtain fusion features, and then select features with stronger expression capabilities from the fusion features as target fusion features, and finally output the pathological analysis results by calling the pathological image analysis model. In addition, after the medical equipment generates the pathological analysis results, it can also locate the lesion area on the WSI image and display the probability of lymph node metastasis.

[0116] Optionally, after acquiring the WSI image, the medical device sends the WSI image to the terminal device, which segments the WSI image, performs similar processing as described above, and finally outputs the pathological analysis result by calling the pathological image analysis model. In addition, after the terminal device generates the pathological analysis result, it can also locate the lesion area on the WSI image and display the probability of lymph node metastasis.

[0117] Optionally, after the medical device acquires the WSI image, it sends the WSI image to the server, which segments the WSI image, performs similar processing as described above, and finally outputs the pathological analysis result by calling the pathological image analysis model. In addition, after the server generates the pathological analysis result, it can also send the pathological analysis result to the medical device or terminal device, which locates the lesion area on the WSI image and displays the probability of lymph node metastasis.

[0118] It should be noted that the terminal device includes but is not limited to tablet computers, laptop computers, PDAs, mobile phones, voice interaction devices and personal computers (PCs), and is not limited here.

[0119] Based on the above introduction, the following two methods can be used for pathological image recognition. Figure 2 , Figure 2 This is a flow chart of the pathological image processing method in the embodiment of the present application. As shown in the figure, in the first recognition method, prediction can be achieved by only using the pathological image analysis model. Specifically:

[0120] In step A1, a pathological image of a target object to be processed is obtained, and the target object may be a patient or a healthy person;

[0121] In step A2, the pathological image is processed and recognized in blocks to obtain multiple groups of first feature sets, multiple groups of second feature sets and multiple groups of third feature sets, and the multiple groups of first feature sets, multiple groups of second feature sets and multiple groups of third feature sets after classification and extraction are subjected to feature fusion processing to generate target fusion features corresponding to the target object;

[0122] In step A3, the target fusion feature is input into the pathological image analysis model, that is, the pathological image analysis model is called to analyze and process the target fusion feature;

[0123] In step A4, the pathological image analysis model outputs the pathological analysis result of the target object to be processed.

[0124] See also Figure 3 , Figure 3Another flow chart of the pathological image processing method in the embodiment of the present application is shown in the figure. In the second identification method, the molecular typing model and the pathological image analysis model are used to achieve prediction. Specifically:

[0125] In step B1, a pathological image of a target object to be processed is obtained, and the target object may be a patient or a healthy person;

[0126] In step B2, the pathological image is processed into blocks to obtain a plurality of block images;

[0127] In step B3, the molecular typing model is called to classify each block image to obtain the corresponding molecular typing type, the molecular typing type corresponding to each block image is counted, and the target molecular typing is selected;

[0128] In step B4, a corresponding pathological image analysis model is selected based on the target molecular typing;

[0129] In step B5, the pathological image is processed and recognized in blocks to obtain multiple groups of first feature sets, multiple groups of second feature sets and multiple groups of third feature sets, and the multiple groups of first feature sets, multiple groups of second feature sets and multiple groups of third feature sets after classification and extraction are subjected to feature fusion processing to generate target fusion features corresponding to the target object;

[0130] In step B6, the target fusion feature is input into the pathological image analysis model, that is, the pathological image analysis model is called to analyze and process the target fusion feature;

[0131] In step B7, the pathological image analysis model outputs the pathological analysis result of the target object to be processed.

[0132] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role. Combined with the above introduction, the solution provided in the embodiment of this application involves artificial intelligence image recognition and other technologies, which are specifically explained through the following embodiments, please refer to Figure 4 In the present application, one embodiment of the pathological image processing method includes:

[0133] 101. Acquire a pathological image of a target object to be processed;

[0134] In this embodiment, the pathological image processing device obtains a pathological image to be processed. Specifically, the pathological image to be processed may be a WSI image. The WSI image is usually large in size. Therefore, it is necessary to segment such large-sized pathological image to be processed to obtain a set of segmented images. For ease of introduction, the pathological image of the target object to be processed may be referred to as a "pathological image to be processed".

[0135] It should be noted that the pathological image processing device can be deployed in a medical device, a terminal device or a server, and this application does not limit it.

[0136] 102. Perform block processing on the pathological image of the target object to be processed to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1;

[0137] In this embodiment, the pathological image processing device may perform block processing on the pathological image of the target object, thereby obtaining N block images, and the N block images constitute a block image set.

[0138] For easier understanding, see Figure 5 , Figure 5 This is a schematic diagram of obtaining a set of block images based on the segmentation of the pathological image to be processed in an embodiment of the present application. As shown in the figure, the pathological image to be processed (such as a living WSI image) can be firstly acquired by a pathological scanner, and then the pathological image to be processed is adjusted to a 20-fold magnification, and a 512×512-sized block image is cut out based on the ROI area by a sliding window method at a magnification of 20. For example, Figure 5 The block image set shown includes 16 block images. It should be noted that the pathological image to be processed can also be adjusted to 5 times, 10 times or 40 times, etc., and the size of the block image can also be 224×224, 299×299, 256×256 or 1024×1024, etc., which is not limited here.

[0139] In actual situations, the block image set may be a full sampling of the pathological image to be processed, or a random sampling of the pathological image to be processed, that is, the number N in the block image set is fixed.

[0140] 103. Perform feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position;

[0141] In this embodiment, the pathological image processing device can extract the first feature set, the second feature set and the third feature set of each block image based on the block image set. Specifically, the block image set is input into the deep learning model included in the pathological image analysis model, and the deep learning model outputs N first feature sets, N second feature sets and N third feature sets.

[0142] Among them, the first feature set includes at least one feature associated with the block image morphology, that is, the features included in the first feature set are mainly features extracted from the block image level. The second feature set includes at least one feature associated with the cell nucleus morphology, that is, the features included in the second feature set are mainly features extracted from the cell nucleus level. The third feature set includes at least one feature associated with the cell nucleus position, that is, the features included in the third feature set are mainly features extracted based on the positional relationship between multiple cell nuclei.

[0143] The pathological image processing device aggregates the first feature set, the second feature set and the third feature set corresponding to each block image in the block image set, that is, counts the various features of the N block images to obtain fusion features corresponding to the same target object, wherein the pathological image to be processed is the pathological image collected from the target object, and the target object can be a patient or other living tissue.

[0144] 104. Perform feature fusion processing on the N first feature sets, the N second feature sets, and the N third feature sets after the classification and extraction processing according to a preset rule to generate a target fusion feature corresponding to the target object, wherein the fusion feature includes P elements, and P is an integer greater than 1;

[0145] In this embodiment, in actual applications, not all elements in the fusion feature have good explanatory power, and some elements do not contribute much to the actual prediction. Therefore, it is necessary to extract Q elements from the P elements in the fusion feature according to preset rules, and then concatenate these Q elements to obtain the target fusion feature.

[0146] 105. Call the pathological image analysis model to analyze and process the target fusion features to output the pathological analysis result of the target object to be processed.

[0147] In this embodiment, the pathological image processing device inputs the target fusion feature into the classifier included in the pathological image analysis model, and the classifier outputs the corresponding pathological analysis result, wherein the pathological analysis result corresponds to the pathological image to be processed, that is, the pathological analysis result belonging to the target object.

[0148] In an embodiment of the present application, a pathological image processing method based on deep learning is provided. First, a set of block images in a pathological image is obtained, and then a feature classification and extraction process is performed on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets. Based on the N first feature sets, N second feature sets, and N third feature sets after classification and extraction, feature fusion processing is performed according to preset rules to generate target fusion features corresponding to the target object. Finally, a pathological image analysis model is called to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed. In the above manner, the pathological image can be directly used to predict the risk of lesion metastasis based on the pathological image analysis model. Compared with biological detection technology, it can reduce the time waiting for the test results and improve the detection efficiency. At the same time, it can also avoid errors caused by uncontrollable factors such as experimental errors and operational errors, thereby providing more accurate test results.

[0149] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in an optional embodiment of the pathological image processing method based on deep learning provided in the embodiments of the present application, the block image set is subjected to feature classification and extraction processing to obtain N first feature sets, N second feature sets, and N third feature sets, which may include:

[0150] Acquire a hematoxylin staining image set and an eosin staining image set according to the block image set, wherein the hematoxylin staining image set includes N hematoxylin staining images, the eosin staining image set includes N eosin staining images, the hematoxylin staining images have a corresponding relationship with the block images, and the eosin staining images have a corresponding relationship with the block images;

[0151] Acquire a cell nucleus image set according to the block image set, wherein the cell nucleus image set includes N cell nucleus images, and the cell nucleus images have a corresponding relationship with the block images;

[0152] According to the hematoxylin staining image set and the eosin staining image set, obtaining N first feature sets corresponding to the block image set;

[0153] According to the cell nucleus image set, N second feature sets corresponding to the block image set are obtained;

[0154] According to the cell nucleus image set, N third feature sets corresponding to the block image set are obtained.

[0155] In this embodiment, a method for a pathological image processing device to obtain features at different levels based on each block image is introduced. Specifically, first, for each block image in the block image set, the local maximum clustering method can be used to obtain the cell nucleus area mask, and then the cell nucleus is segmented to obtain the cell nucleus image. In addition to using the local maximum clustering method to extract the cell nucleus area mask, deep learning-based segmentation algorithms, watershed and other segmentation algorithms can also be used, which are not limited here. In addition, each block image needs to be stained, and the hematoxylin staining channel and the eosin staining channel are separated by a color deconvolution algorithm (color deconvolution), thereby obtaining a hematoxylin staining image and an eosin staining image.

[0156] It should be noted that the first feature set, the second feature set and the third feature set can be pathological genomics features, and the extracted pathological genomics features can also be integrated with clinical tumor marker test results, biochemical tests and genetic test results for multimodal learning. Optionally, the first feature set, the second feature set and the third feature set can also be imaging genomics features and features extracted by deep learning models. Deep learning models include but are not limited to Visual Geometry Group (VGG) 16, VGG19, Inception Network (InceptionNet), Dense Convolutional Network (DenseNet) and Residual Network (ResNet). Among them, the initial parameters of the deep learning model can be pre-trained based on the ImageNet dataset.

[0157] For ease of understanding, the following will take one of the block images in the block image set as an example for introduction. It can be understood that for other block images in the block image set, the following method can be used to generate corresponding cell nucleus images, hematoxylin staining images, and eosin staining images. Figure 6 , Figure 6 This is a schematic diagram of generating associated images based on block images in an embodiment of the present application. As shown in the figure, the image on the upper left is the original image, that is, the original block image, the upper right is the cell nucleus image obtained by the local maximum clustering method, the lower left is the hematoxylin-stained image obtained after hematoxylin staining, and the lower right is the eosin-stained image obtained after eosin staining.

[0158] Among them, hematoxylin-eosin staining (HE) is alkaline, which mainly makes the chromatin in the cell nucleus and the nucleic acid in the cytoplasm purple-blue. Eosin is an acidic dye, which mainly makes the components in the cytoplasm and extracellular matrix red.

[0159] Based on a block image in the block image set, the hematoxylin staining image and the eosin staining image corresponding to the block image are obtained. The hematoxylin staining image and the eosin staining image both belong to the level of the block image. Therefore, based on the hematoxylin staining image and the eosin staining image, the first feature set corresponding to the block image can be extracted, wherein the first feature set includes but is not limited to morphological features, texture features, gradient features and statistical features. Similarly, the cell nucleus image corresponding to the block image is obtained. The cell nucleus image belongs to the image at the cell nucleus level. Therefore, based on the cell nucleus image, the second feature set corresponding to the block image can be extracted, wherein the second feature set includes but is not limited to the morphological features, texture features, gradient features and statistical features of the cell nucleus. By analogy, N block image sets can obtain N first feature sets and N second feature sets.

[0160] Secondly, in an embodiment of the present application, a method is provided for obtaining features at different levels for each block image. Through the above method, features at multiple levels are utilized to more comprehensively reflect the information of the block image, thereby increasing the amount of information processed by the image and further improving the accuracy of image prediction.

[0161] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in an optional embodiment of the pathological image processing method based on deep learning provided in the embodiments of the present application, obtaining N third feature sets corresponding to the block image set according to the cell nucleus image set may include:

[0162] According to the cell nucleus image set, the cell nucleus centroid position corresponding to each cell nucleus image is obtained;

[0163] According to the position of the cell nucleus centroid corresponding to each cell nucleus image in the cell nucleus image set, a Voronoi diagram corresponding to each cell nucleus image is generated;

[0164] According to the Voronoi diagram corresponding to each cell nucleus image in the cell nucleus image set, N third feature sets are obtained.

[0165] In this embodiment, a method for obtaining a third feature set based on a Voronoi diagram is introduced for a pathological image processing device. Specifically, a segmentation process is performed on each block image to obtain a cell nucleus image. For each cell nucleus image, the position of the cell nucleus centroid of the cell nucleus is first determined. Usually, a cell nucleus image includes multiple cell nucleus centroids, so a Voronoi diagram can be constructed based on the positions of these cell nucleus centroids, and the third feature set can be extracted on the constructed Voronoi diagram. Therefore, N groups of third feature sets can be obtained from N cell nucleus images.

[0166] It should be noted that the third feature set includes but is not limited to features such as the distance between cell nuclei and the degree of aggregation of cell nuclei.

[0167] The Voronoi diagram is also called Thiessen polygon, which is a way to divide a plane area. Assume that there are 1000 nucleus centroids on a nucleus image, which are represented by P1 to P 1000 , then the corresponding Voronoi diagram is divided into 1000 regions, which are represented as S1 to S 1000 , and S i All points within the nucleus to the centroid P i The distance to the center of mass of any other cell nucleus is less than or equal to the distance to the center of mass of any other cell nucleus.

[0168] Again, in an embodiment of the present application, a method for obtaining a third feature set using a Voronoi diagram is provided. Through the above method, the Voronoi diagram is used to reflect the positional relationship between cell nuclei, thereby increasing the feature relationship between cell nuclei, further increasing the amount of information for image processing, and thereby improving the accuracy of image prediction.

[0169] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in an optional embodiment of the pathological image processing method based on deep learning provided in the embodiments of the present application, the N first feature sets, the N second feature sets and the N third feature sets after classification and extraction are subjected to feature fusion processing according to preset rules to generate target fusion features corresponding to the target object, which may include:

[0170] According to the N first feature sets, the N second feature sets, and the N third feature sets, obtain statistical results corresponding to T feature dimensions, where T is an integer greater than 1;

[0171] Generate fusion features corresponding to the target object based on the statistical results corresponding to T feature dimensions;

[0172] Obtain Q elements from the fusion features corresponding to the target object, where Q is greater than or equal to 1;

[0173] The Q elements are concatenated to obtain the target fusion features corresponding to the target object.

[0174] In this embodiment, a method for a pathological image processing device to fuse the features of each block image into the overall features of the target object is introduced. Specifically, assuming that N is 10, it means that the entire pathological image to be processed is divided into 10 block images, and each block image has a corresponding first feature set, a second feature set, and a third feature set. Assuming that the first feature set includes feature A, feature B, and feature C, the second feature set includes feature D, feature E, and feature F, and the third feature set includes feature G and feature H, it can be seen that a total of 8 feature dimensions are included, that is, T is 8.

[0175] Based on the above introduction, for ease of explanation, please refer to Table 1, which shows the scores of the 10 block images in different feature dimensions, and the score range can be set to 0 to 100.

[0176] Table 1

[0177] Feature A Feature B Feature C Feature D Feature E Feature F Feature G Feature H patch1 51 42 37 44 59 55 43 71 patch2 65 66 91 90 60 99 37 78 patch3 12 98 77 39 29 13 74 2 patch4 10 31 34 36 55 39 73 62 patch5 84 18 33 72 62 57 100 41 patch6 74 11 67 57 87 34 12 69 patch7 23 64 25 18 74 66 19 45 patch8 61 26 93 19 62 65 100 16 patch9 59 17 19 81 30 64 34 48 patch10 36 50 33 16 71 75 54 53

[0178] As can be seen from Table 1, patch1 to patch10 represent block images 1 to block images 10 respectively, and features A to H are the features extracted from each block image. Here, 8 feature dimensions are taken as an example for introduction, and then for each feature, the distribution of the block image is statistically analyzed. Taking feature A as an example, it is assumed that 0 to 100 is divided into 5 score segments, and each score segment can be called a "bin". Each block image is divided based on different score segments. Please refer to Table 2, which is a schematic result of statistical analysis of the block image based on feature A.

[0179] Table 2

[0180]

[0181] It can be seen from Table 2 that for feature A, the statistical results of different score segments can be obtained, which can be expressed as (2, 2, 2, 3, 1). Similarly, statistics are performed on other features to obtain the statistical results corresponding to T feature dimensions. The statistical results of each feature can be concatenated to obtain the fused features corresponding to the target object. For example, feature A is expressed as (2, 2, 2, 3, 1), and feature B is expressed as (5, 2, 1, 1, 1), then the fused features are expressed as (2, 2, 2, 3, 1, 5, 2, 1, 1, 1).

[0182] It should be noted that the entire pathological image to be processed can be divided into 100 block images, 10,000 block images or other numbers of block images. The number of feature dimensions can be 50, 100 or other numbers, and the feature dimensions included in the first feature set, the feature dimensions included in the second feature set and the feature dimensions included in the third feature set are not limited.

[0183] Finally, Q elements are obtained from the fusion features corresponding to the target object, and then the Q elements are concatenated to obtain the target fusion features corresponding to the target object. For example, the first, second, fifth, sixth, and seventh elements are extracted from the fusion features (2,2,2,3,1,5,2,1,1,1) to obtain the target fusion features (2,2,1,5,2), and the value of Q is 5 at this time.

[0184] Secondly, in an embodiment of the present application, a method is provided for fusing the features of each block image into the overall features of the target object. Through the above method, considering that the size of the pathological image to be processed is large and it is difficult to directly perform an overall analysis, it is necessary to cut the pathological image to be processed into multiple block images, and then fuse the features of these block images to obtain a fused feature that can express the patient as a whole, thereby improving the operability of image processing.

[0185] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in an optional embodiment of the pathological image processing method based on deep learning provided in the embodiments of the present application, obtaining statistical results corresponding to T feature dimensions according to N first feature sets, N second feature sets, and N third feature sets may include:

[0186] According to the N first feature sets, the N second feature sets and the N third feature sets, statistics are performed on the distribution of the N block images in T feature dimensions to obtain T frequency distribution histograms, wherein the feature dimensions and the frequency distribution histograms have a corresponding relationship;

[0187] According to each frequency distribution histogram in the T frequency distribution histograms, a statistical result corresponding to each frequency distribution histogram is generated;

[0188] According to the statistical results corresponding to each frequency distribution histogram, generate statistical results corresponding to T feature dimensions;

[0189] According to the statistical results corresponding to the T feature dimensions, the fusion features corresponding to the target object are generated, which may include:

[0190] The statistical results corresponding to T feature dimensions are concatenated to obtain the fused features corresponding to the target object.

[0191] In this embodiment, a method for performing feature statistics based on a frequency distribution histogram in a pathological image processing device is introduced. Specifically, assuming that N is 1000, it means that the entire pathological image to be processed is divided into 1000 block images, and each block image has a corresponding first feature set, a second feature set, and a third feature set. Assuming that the first feature set includes feature A, the second feature set includes feature B, and the third feature set includes feature C, it can be seen that a total of 3 feature dimensions are included, that is, T is 3.

[0192] For easier understanding, see Figure 7 , Figure 7 is a frequency distribution histogram diagram based on different features in the embodiment of the present application, such as Figure 7 As shown in (a), 1000 block images are grouped in different score segments based on feature A to obtain the corresponding grouping results. For the statistical results based on feature A, please refer to Table 3. Table 3 is a schematic diagram of the statistical results of 1000 block images based on feature A.

[0193] Table 3

[0194]

[0195] It can be seen that the feature vector of feature A is represented as (50, 180, 360, 310, 100).

[0196] Please refer again Figure 7 ,like Figure 7 As shown in (b), 1000 block images are grouped in different score segments based on feature B to obtain corresponding grouping results. For the statistical results based on feature B, please refer to Table 4. Table 4 is a schematic diagram of the statistical results of 1000 block images based on feature B.

[0197] Table 4

[0198]

[0199] It can be seen that the feature vector of feature B is represented as (200, 50, 100, 400, 250).

[0200] Please refer again Figure 7 ,like Figure 7 As shown in (c), 1000 block images are grouped in different score segments based on feature C to obtain corresponding grouping results. For the statistical results based on feature C, please refer to Table 5. Table 5 is a schematic diagram of the statistical results of 1000 block images based on feature C.

[0201] Table 5

[0202]

[0203] It can be seen that the feature vector of feature C is represented as (350, 220, 130, 250, 50). Therefore, the feature vectors of feature A, feature B and feature C are concatenated to obtain the fused feature corresponding to the target object, which is represented as (50, 180, 360, 310, 10, 200, 50, 100, 400, 250, 350, 220, 130, 250, 50), that is, the fused feature includes 15 elements, that is, P is 15 at this time. Next, Q elements need to be selected from the fused feature. Assume that the first element, the third element, the sixth element, the ninth element, the tenth element and the thirteenth element are selected from the fused feature and concatenated into the target fused feature, that is, (50, 360, 200, 400, 250, 130), that is, the fused feature includes 6 elements, that is, Q is 6 at this time.

[0204] It should be noted that, in the process of target object feature fusion, in addition to the frequency distribution histogram method, one-dimensional k-means clustering algorithm, average method, maximum method, P-norm method or graph convolution method can also be used.

[0205] Again, in an embodiment of the present application, a method for feature statistics based on a frequency distribution histogram is provided. Through the above method, the statistical results in different feature dimensions can be more intuitively reflected. When performing feature fusion, the statistical results can be obtained based on the frequency distribution histogram of each feature, thereby improving the feasibility and operability of the solution.

[0206] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in an optional embodiment of the pathological image processing method based on deep learning provided in the embodiments of the present application, calling the pathological image analysis model to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed may include:

[0207] Calling a pathological image analysis model to analyze and process the target fusion feature to output a lesion metastasis probability and at least one importance feature of the target object to be processed;

[0208] After calling the pathological image analysis model to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed, the following may also be included:

[0209] Presentation of pathological analysis results;

[0210] or,

[0211] The pathological analysis result is sent to the client so that the client can display the pathological analysis result.

[0212] In this embodiment, a method for displaying pathological analysis results is introduced. Specifically, after a target fusion feature is selected from the fusion features, the target fusion feature is input into a pathological image analysis model, and the pathological image analysis model outputs a pathological analysis result, wherein the pathological analysis result may specifically include a probability of lesion metastasis and at least one importance feature. The probability of lesion metastasis may represent the probability of lymph node metastasis, and the at least one importance feature represents the first several important features screened out from T feature dimensions, for example, the first 10 features are screened out from 200 feature dimensions, and a bar chart is made for display.

[0213] For easier understanding, see Figure 8 , Figure 8 This is a schematic diagram of the display interface of the pathological analysis results in the embodiment of the present application. As shown in the figure, the pathological image of patient A can be displayed in the automatic medical system. The pathological image can specifically be a WSI image. After the pathological image analysis model, the probability of lesion metastasis can be predicted and displayed on the interface, such as 15%. In addition, the importance of the features used in the model can be calculated based on the information gain. Assuming that the top 10 important features are taken, these important features are sorted according to the F-score to generate a bar chart. Figure 8 As shown, feature 2 is displayed on the interface, and the F-score of feature 2 is 102. Feature 5 is also displayed on the interface, and the F-score of feature 5 is 94. Feature 9 is also displayed on the interface, and the F-score of feature 9 is 91. Feature 4 is also displayed on the interface, and the F-score of feature 4 is 85. Feature 3 is also displayed on the interface, and the F-score of feature 3 is 81. Feature 7 is also displayed on the interface, and the F-score of feature 7 is 56. Feature 0 is also displayed on the interface, and the F-score of feature 0 is 34. Feature 6 is also displayed on the interface, and the F-score of feature 6 is 2. Feature 8 is also displayed on the interface, and the F-score of feature 8 is 1.

[0214] In addition to using F-score when evaluating important features, other scores can also be used for evaluation, such as accuracy, precision, or recall, which are not limited here.

[0215] It should be noted that if the pathological image processing device is deployed on a server, the server can send the pathological analysis result to the client, and the client will display the pathological analysis result. If the pathological image processing device is deployed on a medical device or a terminal device, the pathological analysis result will be directly displayed by the medical device or the terminal device.

[0216] Secondly, in an embodiment of the present application, a method for displaying pathological analysis results is provided. Through the above method, the pathological analysis results can be displayed to medical staff more intuitively, thereby assisting medical staff in diagnosis, thereby improving the practicality and convenience of the solution.

[0217] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in an optional embodiment of the pathological image processing method based on deep learning provided in the embodiments of the present application, after performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets and N third feature sets, it may also include:

[0218] Determine at least one target block image from the N block images according to the N first feature sets, the N second feature sets, and the N third feature sets;

[0219] At least one target block image is highlighted in the pathological image of the target object to be processed, or at least one target block image is sent to the client so that the client highlights at least one target block image in the pathological image of the target object to be processed.

[0220] In this embodiment, a method for displaying the detection results of block images is introduced. Specifically, after collecting N block images, the first feature set, the second feature set and the third feature set corresponding to each block image can be respectively obtained, and then analysis is performed based on these features. For example, based on the cell nucleus morphological features included in the second feature set in a certain block image, it is found that the cell nucleus is relatively large, then it is possible that the cell nucleus proliferates relatively quickly. This feature can be used to explain lymph node metastasis, that is, it means that this phenomenon is related to lymph node metastasis, so the block image can be marked to facilitate subsequent analysis.

[0221] For ease of introduction, see Fig. 9 , Fig. 9 This is a schematic diagram of the target block image position display interface in the embodiment of the present application. As shown in the figure, the pathological image to be processed is first analyzed and the corresponding feature interpretation is associated. For example, if the distance between cell nuclei is small, it means that the number of cell nuclei is large and there may be diffusion. For example, if the texture of the cell nucleus is large, it means that lesions may occur. After obtaining the feature interpretation, the position of these key block images (i.e., target block images) can be visualized in the thumbnail of the pathological image to be processed, such as Fig. 9As shown, it is assumed that region 1 is used to indicate target segmented image 1, region 2 is used to indicate target segmented image 2, region 3 is used to indicate target segmented image 3, and so on. Based on these annotated positions, doctors can be effectively assisted in making preoperative decisions for lesion (such as colorectal cancer) resection.

[0222] Optionally, different colors can be used to mark the target block images corresponding to different features. For example, the red frame is used to mark the area related to the cell nucleus size, and the blue area is used to mark the area related to the cell nucleus texture. This is only a schematic diagram and should not be understood as a limitation of the present application.

[0223] It should be noted that if the pathological image processing device is deployed on a server, the server can send the pathological analysis result to the client, and the client will display the pathological analysis result. If the pathological image processing device is deployed on a medical device or a terminal device, the pathological analysis result will be directly displayed by the medical device or the terminal device.

[0224] Secondly, in an embodiment of the present application, a method for displaying the detection results of block images is provided. Through the above method, the output results of lesion metastasis can be further explained, and the explanation results can be more intuitively displayed to medical staff, thereby assisting medical staff in diagnosis, thereby improving the practicality and convenience of the solution.

[0225] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in an optional embodiment of the pathological image processing method based on deep learning provided in the embodiments of the present application, after the pathological image of the target object to be processed is processed into blocks to obtain a set of block images, the method may also include:

[0226] Calling the molecular typing model to process the block image set to output the molecular typing corresponding to each block image in the block image;

[0227] Counting the molecular typing corresponding to each block image in the block image set to obtain H statistical counting results, where H is an integer greater than 1;

[0228] Determine the target molecular typing corresponding to the pathological image to be processed according to H statistical counting results;

[0229] According to the target molecular typing, a pathological image analysis model is obtained from a set of pathological image analysis models to be selected, wherein the set of pathological image analysis models to be selected includes at least one pathological image analysis model to be selected, each pathological image analysis model to be selected corresponds to a type of molecular typing, and the pathological image analysis model corresponds to the target molecular typing.

[0230] In this embodiment, a method for selecting to use a molecular typing model for prediction is introduced. Specifically, after obtaining a set of block images in a pathological image to be processed, each block image is input into a molecular typing model, thereby outputting the molecular typing corresponding to each block image. Assuming that N is equal to 1000 and H is equal to 4, that is, there are 1000 block images in total, and there are 4 molecular typings (respectively molecular typing C1, molecular typing C2, molecular typing C3 and molecular typing C4), each block image is input into a molecular typing model, and then 1000 molecular typings are obtained. Assuming that after statistics, 500 block images are predicted to be molecular typing C1, 300 block images are predicted to be molecular typing C2, 100 block images are predicted to be molecular typing C3, and 100 block images are predicted to be molecular typing C4. Therefore, it can be determined that the target molecular typing corresponding to the pathological image to be processed is molecular typing C1. Therefore, the pathological image analysis model corresponding to the molecular classification C1 is selected as the pathological image analysis model used subsequently.

[0231] Taking the molecular typing of colorectal cancer as an example, molecular typing C1 is immune type, molecular typing C2 is classical type, molecular typing C3 is metabolic type, and molecular typing C4 is mesenchymal type. It is understandable that if it is for other types of cancer, its molecular typing type can also be other types, which is not limited this time.

[0232] Specifically, the patient's pathological image to be processed is first obtained, and then based on the pathological image to be processed, the ROI area can be extracted by automatic recognition, or the doctor outlines the cancer area as the ROI area, and then the pathological image to be processed with the outlined ROI area is input into the molecular typing model, and the molecular typing model outputs the molecular typing of each block image in the pathological image to be processed, and the target molecular typing is obtained after statistics. Specifically, assuming that the target molecular typing is C1, the pathological image analysis model A is used to perform the analysis of lymph node metastasis prediction, and then the pathological analysis result 1 is obtained. Assuming that the target molecular typing is C2, the pathological image analysis model B is used to perform the analysis of lymph node metastasis prediction, and then the pathological analysis result 2 is obtained. Assuming that the target molecular typing is C3, the pathological image analysis model C is used to perform the analysis of lymph node metastasis prediction, and then the pathological analysis result 3 is obtained. Assuming that the target molecular typing is C4, the pathological image analysis model D is used to perform the analysis of lymph node metastasis prediction, and then the pathological analysis result 4 is obtained.

[0233] Based on the above introduction, in the prediction stage of molecular typing model, please refer to Fig.10 , Fig.10This is a flow chart of image recognition based on the molecular typing model in the embodiment of the present application. As shown in the figure, assuming that the patient's pathological image to be processed is a WSI image, the doctor outlines the cancer area as the ROI area, and then cuts the WSI image marked with the ROI area at 20 times by the sliding window method to obtain 512×512 block images, and predicts the classification of each block image through the trained molecular typing model (such as the InceptionV3 model) and records it. Count the classifications of all patches in the patient's WSI, and vote for the molecular typing with the largest proportion as the final predicted target molecular typing of the patient, where, Fig.10 The molecular classification C1 is the largest, so the target molecular classification for this patient is C1.

[0234] Based on the above introduction, during the training phase of the molecular typing model, please refer to Fig.11 , Fig.11 A flow chart of model training based on the molecular typing model in the embodiment of the present application is shown in the figure. A large number of pathological images to be trained are collected. The pathological images to be trained are WSI images. The doctor outlines the cancer area as the ROI area, and then cuts the WSI image marked with the ROI area at 20 times by the sliding window method to extract 512×512 block images, and uses the known patient molecular typing labels as the labels of each block image, such as the label of block image 1 is molecular typing C1, and the label of block image 2 is molecular typing C2. The block images cut out from the set of pathological images to be trained are input into the molecular typing model to be trained, and finally the prediction results of 4 molecular typings are output. The cross entropy between the labeled molecular typing and the predicted molecular typing is used as the loss function, and the training is tuned until the molecular typing model to be trained converges, so that it has a good molecular typing model accuracy.

[0235] At this point, the training of the molecular typing model is completed. Considering that lesions have four molecular typings, and different molecular typings transcribe and express different proteins, which cause different activations of cell signaling pathways, and then exercise different molecular functions, which are ultimately reflected in phenotypic differences. Therefore, the molecular typing of lesions can be predicted through WSI images, and the results of molecular typing can assist in the prediction of subsequent pathological image analysis models, targeted training optimization and prediction, that is, molecular typing can separate patients with genotype heterogeneity, and then make targeted predictions.

[0236] Secondly, in an embodiment of the present application, a method for selecting a molecular typing model for prediction is provided. Through the above method, the lesions can be subdivided into four molecular typings, and a pathological image analysis model can be trained separately for each molecular typing, so as to predict lymph node metastasis more specifically and improve the accuracy of the prediction.

[0237] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in an optional embodiment of the pathological image processing method based on deep learning provided in the embodiments of the present application, the pathological image of the target object to be processed is processed in blocks to obtain a set of block images, which may include:

[0238] Extracting a region of interest (ROI) from a pathological image of a target object to be processed;

[0239] A set of block images is obtained from the ROI area using the sliding window method.

[0240] In this embodiment, a method for image segmentation for the ROI area is introduced. Specifically, a pathological image to be processed of the patient is first obtained. The pathological image to be processed may be a WSI image. Optionally, the pathological image to be processed may also be a photograph of the field of view under a 10x microscope, or a photograph of the field of view under a 20x microscope, or a photograph of the field of view under a 40x microscope, which is not limited here.

[0241] Then the doctor manually outlines the ROI region in the pathological image to be processed. Optionally, the ROI region can also be generated using the prediction results of the adenocarcinoma segmentation model based on deep learning. Segmentation is performed based on the selected ROI region to obtain a set of block images.

[0242] For easier understanding, see Fig.12 , Fig.12 This is a flow chart of image recognition based on the pathological image analysis model in the embodiment of the present application. As shown in the figure, the patient's pathological image to be processed is first collected, and then a 512×512 block image is cut out according to the ROI area using the sliding window method, and then each block image is subjected to feature extraction to obtain a first feature set, a second feature set, and a third feature set. Statistics are performed based on each feature set of each image to obtain a fusion feature, and then the target fusion feature is extracted from the fusion feature as the final feature of the target object. Finally, the target fusion feature is input into the pathological image analysis model, and the pathological image analysis model outputs the pathological analysis result. The pathological image analysis model includes a classifier. It should be noted that the classifier includes but is not limited to machine learning and deep learning models such as XGBoost, logistic regression, support vector machine (SVM), long short-term memory network (LSTM), and multilayer perceptron (MLP).

[0243] Taking colorectal cancer prediction as an example, for the WSI images of colorectal cancer patients, a 512×512 block image at 20 times magnification is selected. This block image contains certain tissue structures and cell information. Therefore, the overall structure and cell nuclear features in the block image are extracted at the same time. The hematoxylin staining channel and the eosin staining channel can be separated by color deconvolution, and multiple features are extracted in each channel to characterize the cell's nuclear division image and nuclear hyperchromasia and other cancer cell atypicality. The relationship between cell position and tissue structure, such as tumor budding, can also be learned by extracting features at the level of the overall block image. The above features are of great significance. Taken together, they can reflect the invasiveness of cancer cells. The more invasive the cancer cells are, the more likely they are to break through the intestinal membrane and cause mesenteric lymph node metastasis. By sampling and counting all the block images of the entire WSI image, the information of the entire WSI image is reflected, and the significant difference features are extracted through statistical tests, and the probability of lymph node metastasis is predicted using the tree-based XGBoost classifier.

[0244] Again, in an embodiment of the present application, a method for image segmentation for a ROI area is provided. Through the above method, the ROI area can be first selected from the entire WSI image, and then the ROI area can be segmented. This can reduce the amount of data processing, and only the ROI area is identified and processed, thereby improving the prediction efficiency.

[0245] For easier understanding, see Fig.13 , Fig.13 This is a flow chart of the pathological image processing method based on deep learning in an embodiment of the present application. As shown in the figure, it can be specifically divided into three parts, namely, the training part of the molecular typing model, the training part of the pathological image analysis model, and the prediction part using the molecular typing model and the pathological image analysis model, which will be introduced separately below.

[0246] The training part of the pathology image analysis model includes:

[0247] In step C1, a set of pathological images to be trained is obtained, wherein the set of pathological images to be trained includes at least one positive sample pathological image and at least one negative sample pathological image;

[0248] In step C2, for each pathological image to be trained in the pathological image set to be trained, a set of block images to be trained is obtained, wherein the pathological image to be trained belongs to a positive sample pathological image or a negative sample pathological image, and the set of block images to be trained includes M block images to be trained, where M is an image greater than or equal to 1;

[0249] In step C3, for each pathological image to be trained in the pathological image set to be trained, M first feature sets to be trained, M second feature sets to be trained, and M third feature sets to be trained are obtained according to the block image set to be trained, wherein the first feature set to be trained includes at least one feature associated with the morphology of the block image to be trained, the second feature set to be trained includes at least one feature associated with the morphology of the cell nucleus, and the third feature set to be trained includes at least one feature associated with the position of the cell nucleus;

[0250] In step C4, for each pathological image to be trained in the pathological image set to be trained, a fusion feature to be trained is generated according to the M first feature sets to be trained, the M second feature sets to be trained, and the M third feature sets to be trained;

[0251] In step C5, for the set of pathological images to be trained, according to the fusion features to be trained corresponding to each pathological image to be trained, a target fusion feature to be trained is determined;

[0252] In step C6, based on the fusion features of the target to be trained corresponding to each pathological image to be trained, a loss function is used to train the pathological image analysis model to be trained, so as to obtain a pathological image analysis model;

[0253] The training part of the molecular typing model includes:

[0254] In step D1, a set of pathological images to be trained is obtained, wherein each pathological image to be trained has a labeled molecular typing;

[0255] In step D2, for each pathological image to be trained in the pathological image set to be trained, a set of block images to be trained corresponding to each pathological image to be trained is obtained, wherein the set of block images to be trained includes M block images to be trained, and M is an image greater than or equal to 1;

[0256] In step D3, for each block image set to be trained corresponding to each pathological image to be trained, the molecular typing model to be trained is called to process the block image set to be trained, so as to output the molecular typing corresponding to each block image in the block image set to be trained;

[0257] In step D4, the molecular typing corresponding to each block image in the block image set is counted to obtain H statistical counting results, where H is an integer greater than 1;

[0258] In step D5, the predicted molecular typing corresponding to the pathological image to be processed is determined according to the H statistical counting results;

[0259] In step D6, based on the predicted molecular typing corresponding to each pathological image to be trained, the molecular typing model to be trained is trained using a loss function to obtain a molecular typing model.

[0260] The prediction part using molecular typing model and pathological image analysis model includes:

[0261] In step S1, a pathological image of a target object to be processed is obtained;

[0262] In step S2, the pathological image of the target object to be processed is processed into blocks to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1;

[0263] In step S3, the molecular typing model is called to process the block image set to output the molecular typing corresponding to each block image in the block image;

[0264] In step S4, the molecular typing corresponding to each block image in the block image set is counted to obtain H statistical counting results, where H is an integer greater than 1;

[0265] In step S5, the target molecular typing corresponding to the pathological image to be processed is determined according to the H statistical counting results;

[0266] In step S6, according to the target molecular typing, a pathological image analysis model is obtained from a set of pathological image analysis models to be selected, wherein the set of pathological image analysis models to be selected includes at least one pathological image analysis model to be selected, each pathological image analysis model to be selected corresponds to a type of molecular typing, and the pathological image analysis model corresponds to the target molecular typing;

[0267] In step S7, feature classification and extraction processing is performed on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position;

[0268] In step S8, the N first feature sets, the N second feature sets, and the N third feature sets that have been subjected to the classification and extraction processing are subjected to feature fusion processing according to a preset rule to generate a target fusion feature corresponding to the target object, wherein the fusion feature includes P elements, where P is an integer greater than 1;

[0269] In step S9, the pathological image analysis model is called to analyze and process the target fusion features to output the pathological analysis result of the target object to be processed.

[0270] In conjunction with the above introduction, please refer to Fig.14 In the embodiment of the present application, one embodiment of the model training method includes:

[0271] 201. Acquire a set of pathological images to be trained, wherein the set of pathological images to be trained includes at least one positive sample pathological image and at least one negative sample pathological image;

[0272] In this embodiment, the model training device obtains a set of pathological images to be trained. Specifically, the set of pathological images to be trained includes at least one positive pathological image and at least one negative pathological image, wherein both the positive pathological image and the negative pathological image can be WSI images. WSI images usually have a large size. Therefore, it is necessary to segment such large-sized pathological images to be processed to obtain a set of segmented images. The positive pathological image represents a pathological image under a diseased condition, and the negative pathological image represents a pathological image under a normal condition.

[0273] It should be noted that the model training device can be deployed on a server or a terminal device with strong computing power, and this application does not limit it.

[0274] 202. For each pathological image to be trained in the pathological image set to be trained, obtain a set of block images to be trained, wherein the pathological image to be trained is a positive sample pathological image or a negative sample pathological image, and the set of block images to be trained includes M block images to be trained, where M is an image greater than or equal to 1;

[0275] In this embodiment, the model training device obtains a set of block images to be trained for each pathological image to be trained in the set of pathological images to be trained. That is, both positive sample pathological images and negative sample pathological images are pathological images to be trained, so each pathological image to be trained needs to be segmented to obtain a set of block images to be trained, and the set of block images to be trained includes M block images to be trained.

[0276] In actual situations, the block image set may be a full sampling of the block images to be trained, or a random sampling of the block images to be trained.

[0277] 203. For each pathological image to be trained in the pathological image set to be trained, obtain M first feature sets to be trained, M second feature sets to be trained, and M third feature sets to be trained according to the block image set to be trained, wherein the first feature set to be trained includes at least one feature associated with the morphology of the block image to be trained, the second feature set to be trained includes at least one feature associated with the morphology of the cell nucleus, and the third feature set to be trained includes at least one feature associated with the position of the cell nucleus;

[0278] In this embodiment, the model training device obtains M first feature sets to be trained, M second feature sets to be trained, and M third feature sets to be trained for each pathological image to be trained in the pathological image set to be trained, according to the set of block images to be trained corresponding to each pathological image to be trained.

[0279] Among them, the first feature set to be trained includes at least one feature associated with the block image morphology, that is, the features included in the first feature set to be trained are mainly features extracted from the block image level. The second feature set to be trained includes at least one feature associated with the cell nucleus morphology, that is, the features included in the second feature set to be trained are mainly features extracted from the cell nucleus level. The third feature set to be trained includes at least one feature associated with the cell nucleus position, that is, the features included in the third feature set to be trained are mainly features extracted based on the positional relationship between multiple cell nuclei.

[0280] 204. For each pathological image to be trained in the set of pathological images to be trained, generate a fusion feature to be trained according to the M first feature sets to be trained, the M second feature sets to be trained, and the M third feature sets to be trained;

[0281] In this embodiment, the model training device aggregates the first feature set to be trained, the second feature set to be trained, and the third feature set to be trained corresponding to each block image in the block image set for each pathological image to be trained in the pathological image set, that is, statistics are performed on the various features of M block images in a pathological image to be trained, so as to obtain fusion features corresponding to the same object, wherein the pathological image to be trained is the pathological image collected from the object, and the object can be a patient or other living tissue.

[0282] 205. For the set of pathological images to be trained, determine the target fusion feature to be trained according to the fusion feature to be trained corresponding to each pathological image to be trained;

[0283] In this embodiment, the model training device performs a statistical test on each dimensional feature of the patient in the pathological image set to be trained, that is, calculates the p-value of the Smirnov (Kolmogorov-Smirnov, KS) test between the metastatic and non-metastatic samples, and then uses the fusion features to be trained with a p-value less than a preset threshold as the final target fusion features to be trained, and then inputs them into the classifier.

[0284] Specifically, assuming that there are 100 transferred samples and 100 non-transferred samples, and assuming that the 100 transferred samples come from one distribution and the 100 non-transferred samples also come from one distribution, through hypothesis testing, verify whether the two sets of data come from the same distribution. If they are significantly different, it is considered that the two sets of data can be distinguished based on the current features, and this feature can be retained. In this way, all the fusion features to be trained are traversed to select the target fusion features to be trained.

[0285] Optionally, other statistical test methods such as Analysis of Variance (ANOVA) test, T test or Mann-Whitney U test (MannWhitney U) may also be used, which are not limited here.

[0286] 206. Based on the fusion features of the target to be trained corresponding to each pathological image to be trained, a loss function is used to train the pathological image analysis model to be trained to obtain a pathological image analysis model.

[0287] In this embodiment, the model training device inputs each target fusion feature to be trained into the pathological image analysis model to be trained for training based on the target fusion feature to be trained corresponding to each pathological image to be trained, wherein the pathological image analysis model to be trained includes a classifier. It should be noted that the classifier includes but is not limited to machine learning and deep learning models such as XGBoost, logistic regression, support vector machine (SVM), long short-term memory network (LSTM) and multilayer perceptron (MLP).

[0288] Specifically, if the tree model is selected as the base classifier, the features can be input into XGBoost, the loss function is determined as the cross entropy function of binary classification, the evaluation indicator used is the receiver operating characteristic curve (ROC)-the area under the ROC curve (Area Under Curve, AUC), the number of early stopping steps is set to 50 steps, and the optimal number of trees is determined by cross-validation to obtain a pathological image analysis model.

[0289] It should be noted that the pathological images to be trained and the pathological images to be processed used in this application are independent and identically distributed (IID). In this way, the model trained based on the training set has a similar effect on the test set.

[0290] In an embodiment of the present application, a model training method is provided. Through the above method, a pathological image analysis model can be trained to predict lymph node metastasis in the early stage and assist doctors in preoperative decision-making. The implementation can better support the prediction of colorectal cancer lymph node metastasis and preoperative decision-making, thereby providing medical solutions more accurately, ultimately reducing the risk of recurrence and metastasis and improving the quality of life of patients.

[0291] The pathological image processing device in this application is described in detail below. Fig.15 , Fig.15 The following is a schematic diagram of an embodiment of a pathological image processing device based on deep learning in an embodiment of the present application. The pathological image processing device 30 includes:

[0292] An acquisition module 301 is used to acquire a pathological image of a target object to be processed;

[0293] The acquisition module 301 is further used to perform block processing on the pathological image of the target object to be processed to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1;

[0294] The processing module 302 is used to perform feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position;

[0295] A generating module 303 is used to perform feature fusion processing on the N first feature sets, the N second feature sets and the N third feature sets after the classification and extraction processing according to a preset rule to generate a target fusion feature corresponding to the target object, wherein the fusion feature includes P elements, and P is an integer greater than 1;

[0296] The output module 304 is used to call the pathological image analysis model to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed.

[0297] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the pathological image processing device 30 provided in the embodiment of the present application,

[0298] The processing module 302 is specifically used to obtain a hematoxylin-stained image set and an eosin-stained image set according to the block image set, wherein the hematoxylin-stained image set includes N hematoxylin-stained images, the eosin-stained image set includes N eosin-stained images, the hematoxylin-stained images have a corresponding relationship with the block images, and the eosin-stained images have a corresponding relationship with the block images;

[0299] Acquire a cell nucleus image set according to the block image set, wherein the cell nucleus image set includes N cell nucleus images, and the cell nucleus images have a corresponding relationship with the block images;

[0300] According to the hematoxylin staining image set and the eosin staining image set, obtaining N first feature sets corresponding to the block image set;

[0301] According to the cell nucleus image set, N second feature sets corresponding to the block image set are obtained;

[0302] According to the cell nucleus image set, N third feature sets corresponding to the block image set are obtained.

[0303] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the pathological image processing device 30 provided in the embodiment of the present application,

[0304] The processing module 302 is specifically used to obtain the cell nucleus centroid position corresponding to each cell nucleus image according to the cell nucleus image set;

[0305] According to the position of the cell nucleus centroid corresponding to each cell nucleus image in the cell nucleus image set, a Voronoi diagram corresponding to each cell nucleus image is generated;

[0306] According to the Voronoi diagram corresponding to each cell nucleus image in the cell nucleus image set, N third feature sets are obtained.

[0307] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the pathological image processing device 30 provided in the embodiment of the present application,

[0308] The generating module 303 is specifically used to obtain statistical results corresponding to T feature dimensions according to the N first feature sets, the N second feature sets and the N third feature sets, where T is an integer greater than 1;

[0309] Generate fusion features corresponding to the target object based on the statistical results corresponding to T feature dimensions;

[0310] Obtain Q elements from the fusion features corresponding to the target object, where Q is greater than or equal to 1;

[0311] The Q elements are concatenated to obtain the target fusion features corresponding to the target object.

[0312] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the pathological image processing device 30 provided in the embodiment of the present application,

[0313] The generating module 303 is specifically used to perform statistics on the distribution of the N block images in T feature dimensions according to the N first feature sets, the N second feature sets and the N third feature sets, so as to obtain T frequency distribution histograms, wherein the feature dimensions and the frequency distribution histograms have a corresponding relationship;

[0314] According to each frequency distribution histogram in the T frequency distribution histograms, a statistical result corresponding to each frequency distribution histogram is generated;

[0315] According to the statistical results corresponding to each frequency distribution histogram, generate statistical results corresponding to T feature dimensions;

[0316] The statistical results corresponding to T feature dimensions are concatenated to obtain the fused features corresponding to the target object.

[0317] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the pathological image processing device 30 provided in the embodiment of the present application,

[0318] The output module 304 is specifically used to call the pathological image analysis model to analyze and process the target fusion feature to output the lesion metastasis probability and at least one importance feature of the target object to be processed;

[0319] The processing module is further used to display the pathological analysis result after the output module calls the pathological image analysis model to analyze and process the target fusion feature to output the pathological analysis result of the target object to be processed;

[0320] or,

[0321] The pathological analysis result is sent to the client so that the client can display the pathological analysis result.

[0322] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the pathological image processing device 30 provided in the embodiment of the present application,

[0323] The acquisition module 301 is further configured to, after the processing module performs feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets, determine at least one target block image from the N block images according to the N first feature sets, the N second feature sets, and the N third feature sets;

[0324] The processing module 302 is further used to highlight at least one target block image in the pathological image of the target object to be processed, or to send at least one target block image to the client so that the client highlights at least one target block image in the pathological image of the target object to be processed.

[0325] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the pathological image processing device 30 provided in the embodiment of the present application,

[0326] The output module 304 is also used for the processing module to process the pathological image of the target object to be processed into blocks to obtain a set of block images, and then call the molecular typing model to process the set of block images to output the molecular typing corresponding to each block image in the block images;

[0327] The processing module 302 is further used to count the molecular typing corresponding to each block image in the block image set to obtain H statistical counting results, where H is an integer greater than 1;

[0328] The acquisition module 301 is also used to determine the target molecular typing corresponding to the pathological image to be processed according to the H statistical counting results;

[0329] The acquisition module 301 is also used to acquire a pathological image analysis model from a set of pathological image analysis models to be selected according to the target molecular typing, wherein the set of pathological image analysis models to be selected includes at least one pathological image analysis model to be selected, each pathological image analysis model to be selected corresponds to a type of molecular typing, and the pathological image analysis model corresponds to the target molecular typing.

[0330] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the pathological image processing device 30 provided in the embodiment of the present application,

[0331] An acquisition module 301 is specifically used to extract a region of interest ROI from a pathological image of a target object to be processed;

[0332] A set of block images is obtained from the ROI area using the sliding window method.

[0333] The model training device in this application is described in detail below. Fig.16 , Fig.16 This is a schematic diagram of an embodiment of a model training device in an embodiment of the present application. The model training device 40 includes:

[0334] An acquisition module 401 is used to acquire a set of pathological images to be trained, wherein the set of pathological images to be trained includes at least one positive sample pathological image and at least one negative sample pathological image;

[0335] The acquisition module 401 is further used to acquire a set of block images to be trained for each pathological image to be trained in the set of pathological images to be trained, wherein the pathological image to be trained belongs to a positive sample pathological image or a negative sample pathological image, and the set of block images to be trained includes M block images to be trained, where M is an image greater than or equal to 1;

[0336] The acquisition module 401 is further used to acquire, for each pathological image to be trained in the pathological image set to be trained, M first feature sets to be trained, M second feature sets to be trained, and M third feature sets to be trained according to the block image set to be trained, wherein the first feature set to be trained includes at least one feature associated with the morphology of the block image to be trained, the second feature set to be trained includes at least one feature associated with the morphology of the cell nucleus, and the third feature set to be trained includes at least one feature associated with the position of the cell nucleus;

[0337] A generating module 402, configured to generate, for each pathological image to be trained in the pathological image set to be trained, fusion features to be trained according to the M first feature sets to be trained, the M second feature sets to be trained, and the M third feature sets to be trained;

[0338] A determination module 403 is used to determine the target fusion feature to be trained according to the fusion feature to be trained corresponding to each pathological image to be trained, for the set of pathological images to be trained;

[0339] The training module 404 is used to train the pathological image analysis model to be trained based on the target fusion features to be trained corresponding to each pathological image to be trained, using a loss function to obtain a pathological image analysis model, wherein the pathological image analysis model is a model adopted by the pathological image processing method based on deep learning.

[0340] The present application embodiment also provides another pathological image processing device, which is deployed in a terminal device, such as Fig.17For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal device can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a car computer, etc., taking the mobile phone as an example:

[0341] Fig.17 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Fig.17 The mobile phone includes: a radio frequency (RF) circuit 510, a memory 520, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (WiFi) module 570, a processor 580, and a power supply 590. Those skilled in the art will understand that Fig.17 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0342] Combine the following Fig.17 A detailed introduction to the various components of the mobile phone:

[0343] The RF circuit 510 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 580 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 510 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the RF circuit 510 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0344] The memory 520 can be used to store software programs and modules. The processor 580 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 520. The memory 520 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 520 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0345] The input unit 530 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 530 may include a touch panel 531 and other input devices 532. The touch panel 531, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 531 or near the touch panel 531 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 531 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 580, and can receive and execute commands sent by the processor 580. In addition, the touch panel 531 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 531, the input unit 530 may also include other input devices 532. Specifically, other input devices 532 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0346] The display unit 540 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 540 may include a display panel 541. Optionally, the display panel 541 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 531 may cover the display panel 541. When the touch panel 531 detects a touch operation on or near it, it is transmitted to the processor 580 to determine the type of touch event. Subsequently, the processor 580 provides a corresponding visual output on the display panel 541 according to the type of touch event. Although in Fig.17 In the embodiment, the touch panel 531 and the display panel 541 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 531 and the display panel 541 can be integrated to realize the input and output functions of the mobile phone.

[0347] The mobile phone may also include at least one sensor 550, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 541 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 541 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0348] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the mobile phone. The audio circuit 560 can transmit the received audio data to the speaker 561 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another mobile phone through the RF circuit 510, or the audio data is output to the memory 520 for further processing.

[0349] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 570. It provides users with wireless broadband Internet access. Fig.17A WiFi module 570 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0350] The processor 580 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520. Optionally, the processor 580 may include one or more processing units; optionally, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 580.

[0351] The mobile phone also includes a power source 590 (such as a battery) for supplying power to various components. Optionally, the power source can be logically connected to the processor 580 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0352] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0353] In the embodiment of the present application, the processor 580 included in the terminal device also has the following functions:

[0354] Acquiring a pathological image of a target object to be processed;

[0355] Performing block processing on the pathological image of the target object to be processed to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1;

[0356] Performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position;

[0357] The N first feature sets, the N second feature sets, and the N third feature sets that have been subjected to classification and extraction processing are subjected to feature fusion processing according to preset rules to generate a target fusion feature corresponding to the target object, wherein the fusion feature includes P elements, where P is an integer greater than 1;

[0358] The pathological image analysis model is called to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed.

[0359] The embodiment of the present application also provides another model training device, which is deployed on a server. Fig.18 6 is a schematic diagram of a server structure provided in an embodiment of the present application. The server 600 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 622 (for example, one or more processors) and a memory 632, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 may be short-term storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 622 may be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the server 600.

[0360] The server 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server 2000 or Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0361] The steps performed by the server in the above embodiment can be based on the Fig.18 The server structure shown.

[0362] Acquire a set of pathological images to be trained, wherein the set of pathological images to be trained includes at least one positive sample pathological image and at least one negative sample pathological image;

[0363] For each pathological image to be trained in the pathological image set to be trained, a block image set to be trained is obtained, wherein the pathological image to be trained belongs to a positive sample pathological image or a negative sample pathological image, and the block image set to be trained includes M block images to be trained, where M is an image greater than or equal to 1;

[0364] For each pathological image to be trained in the set of pathological images to be trained, M first feature sets to be trained, M second feature sets to be trained, and M third feature sets to be trained are obtained according to the set of block images to be trained, wherein the first feature set to be trained includes at least one feature associated with the morphology of the block images to be trained, the second feature set to be trained includes at least one feature associated with the morphology of the cell nucleus, and the third feature set to be trained includes at least one feature associated with the position of the cell nucleus;

[0365] For each pathological image to be trained in the set of pathological images to be trained, generating a fusion feature to be trained according to M first feature sets to be trained, M second feature sets to be trained, and M third feature sets to be trained;

[0366] For the set of pathological images to be trained, determining the target fusion features to be trained according to the fusion features to be trained corresponding to each pathological image to be trained;

[0367] Based on the fusion features of the target to be trained corresponding to each pathological image to be trained, the pathological image analysis model to be trained is trained using a loss function to obtain a pathological image analysis model.

[0368] Fig.19 : is a diagram showing the structure of a medical system 70 of an embodiment of the present application. The medical system 70 of this embodiment is a system for supporting endoscopy services. The medical system 70 has a probe 701, a processor 702, a display 703, a circuit 704 and an interface 705. The medical system 70 can work in conjunction with a terminal device. The probe 701 can specifically be an endoscopic probe, which can be inserted into the esophagus, gastrointestinal tract, bronchus, etc. for real-time scanning and imaging. By using an endoscopic probe, a doctor can clearly identify the tumor growth layer and infiltration depth. In addition, the endoscopic probe can also be used for imaging of organs near the intestines, and plays a role in the diagnosis of lesions of the pancreas, bile duct, and gallbladder.

[0369] The processor 702 is used to identify the endoscopic image captured by the probe 701 and generate an identification result. The display 703 displays the lesion identification result according to the image signal input by the processor 702. The lesion identification result is specifically an image result, which can display the image obtained by the probe 701 in real time. The circuit 704 is used to connect the various modules in the medical system 70 and provide electrical signals so that the medical system 70 can work normally and establish a communication connection with the terminal device.

[0370] The medical system 70 can directly identify and process the collected endoscopic images, or send the collected endoscopic images to the terminal device through the interface 705, and the terminal device can identify and process the endoscopic images. The terminal device can create electronic medical files, prescriptions, or print them out directly based on the lesion identification results sent by the medical system 70. In the embodiment of the present application, the processor 702 included in the medical system also has the following functions:

[0371] Acquiring a pathological image of a target object to be processed;

[0372] Performing block processing on the pathological image of the target object to be processed to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1;

[0373] Performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position;

[0374] The N first feature sets, the N second feature sets, and the N third feature sets that have been subjected to classification and extraction processing are subjected to feature fusion processing according to preset rules to generate a target fusion feature corresponding to the target object, wherein the fusion feature includes P elements, where P is an integer greater than 1;

[0375] The pathological image analysis model is called to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed.

[0376] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0377] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0378] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0379] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0380] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0381] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A pathological image processing method based on deep learning, It is characterized in that include: Acquiring a pathological image of a target object to be processed; Performing block processing on the pathological image of the target object to be processed to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1; Performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position; Obtain statistical results corresponding to T feature dimensions according to the N first feature sets, the N second feature sets, and the N third feature sets, where T is an integer greater than 1; Generate a fusion feature corresponding to the target object according to the statistical results corresponding to the T feature dimensions, wherein the fusion feature includes P elements, and P is an integer greater than 1; Obtain Q elements from the fusion features corresponding to the target object, where Q is greater than or equal to 1; The Q elements are concatenated to obtain a target fusion feature corresponding to the target object; The pathological image analysis model is called to analyze and process the target fusion feature to output the pathological analysis result of the target object to be processed.

2. The method according to claim 1, It is characterized in that The performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets and N third feature sets includes: Acquire a hematoxylin staining image set and an eosin staining image set according to the block image set, wherein the hematoxylin staining image set includes N hematoxylin staining images, the eosin staining image set includes N eosin staining images, the hematoxylin staining images have a corresponding relationship with the block images, and the eosin staining images have a corresponding relationship with the block images; Acquire a cell nucleus image set according to the block image set, wherein the cell nucleus image set includes N cell nucleus images, and the cell nucleus images have a corresponding relationship with the block images; According to the hematoxylin staining image set and the eosin staining image set, acquiring the N first feature sets corresponding to the block image set; According to the cell nucleus image set, acquiring the N second feature sets corresponding to the block image set; According to the cell nucleus image set, the N third feature sets corresponding to the block image set are obtained.

3. The method according to claim 2, It is characterized in that The step of obtaining the N third feature sets corresponding to the block image set according to the cell nucleus image set includes: According to the cell nucleus image set, obtaining the cell nucleus centroid position corresponding to each cell nucleus image; Generating a Voronoi diagram corresponding to each cell nuclear image according to the cell nuclear centroid position corresponding to each cell nuclear image in the cell nuclear image set; The N third feature sets are obtained according to the Voronoi diagram corresponding to each cell nucleus image in the cell nucleus image set.

4. The method according to claim 1, It is characterized in that The obtaining statistical results corresponding to T feature dimensions according to the N first feature sets, the N second feature sets, and the N third feature sets includes: According to the N first feature sets, the N second feature sets and the N third feature sets, statistics are performed on the distribution of the N block images in the T feature dimensions to obtain T frequency distribution histograms, wherein the feature dimensions have a corresponding relationship with the frequency distribution histograms; According to each frequency distribution histogram in the T frequency distribution histograms, generating a statistical result corresponding to each frequency distribution histogram; Generate statistical results corresponding to the T feature dimensions according to the statistical results corresponding to each frequency distribution histogram; Generating the fusion feature corresponding to the target object according to the statistical results corresponding to the T feature dimensions includes: The statistical results corresponding to the T feature dimensions are concatenated to obtain the fused features corresponding to the target object.

5. The method according to claim 1, It is characterized in that The calling of the pathological image analysis model to analyze and process the target fusion feature to output the pathological analysis result of the target object to be processed includes: Calling a pathological image analysis model to analyze and process the target fusion feature to output a lesion metastasis probability and at least one importance feature of the target object to be processed; After calling the pathological image analysis model to analyze and process the target fusion feature to output the pathological analysis result of the target object to be processed, the method further includes: presenting the pathological analysis results; or, The pathological analysis result is sent to a client so that the client displays the pathological analysis result.

6. The method according to claim 1, It is characterized in that After performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets, the method further includes: Determine at least one target block image from the N block images according to the N first feature sets, the N second feature sets, and the N third feature sets; The at least one target block image is highlighted in the pathological image of the target object to be processed, or the at least one target block image is sent to the client so that the client highlights the at least one target block image in the pathological image of the target object to be processed.

7. The method according to claim 1, It is characterized in that After the pathological image of the target object to be processed is processed in blocks to obtain a set of block images, the method further includes: Invoking a molecular typing model to process the block image set to output a molecular typing corresponding to each block image in the block images; Performing statistics on the molecular typing corresponding to each of the block images in the block image set to obtain H statistical counting results, wherein H is an integer greater than 1; Determine the target molecular typing corresponding to the pathological image to be processed according to the H statistical counting results; According to the target molecular typing, the pathological image analysis model is obtained from a set of pathological image analysis models to be selected, wherein the set of pathological image analysis models to be selected includes at least one pathological image analysis model to be selected, each pathological image analysis model to be selected corresponds to a type of molecular typing, and the pathological image analysis model corresponds to the target molecular typing.

8. The method according to any one of claims 1 to 7, It is characterized in that The block processing of the pathological image of the target object to be processed to obtain a set of block images includes: Extracting a region of interest (ROI) from the pathological image of the target object to be processed; The block image set is acquired from the ROI region by using a sliding window method.

9. A model training method, It is characterized in that include: Acquire a set of pathological images to be trained, wherein the set of pathological images to be trained includes at least one positive sample pathological image and at least one negative sample pathological image; For each pathological image to be trained in the set of pathological images to be trained, a set of block images to be trained is obtained, wherein the pathological image to be trained belongs to a positive sample pathological image or a negative sample pathological image, and the set of block images to be trained includes M block images to be trained, where M is an image greater than or equal to 1; For each pathological image to be trained in the set of pathological images to be trained, M first feature sets to be trained, M second feature sets to be trained, and M third feature sets to be trained are obtained according to the set of block images to be trained, wherein the first feature set to be trained includes at least one feature associated with the morphology of the block images to be trained, the second feature set to be trained includes at least one feature associated with the morphology of the cell nucleus, and the third feature set to be trained includes at least one feature associated with the position of the cell nucleus; For each pathological image to be trained in the set of pathological images to be trained, generating a fusion feature to be trained according to the M first feature sets to be trained, the M second feature sets to be trained and the M third feature sets to be trained; For the set of pathological images to be trained, determining a target fusion feature to be trained according to the fusion feature to be trained corresponding to each pathological image to be trained; Based on the fusion features of the target to be trained corresponding to each pathological image to be trained, a loss function is used to train the pathological image analysis model to be trained to obtain a pathological image analysis model, wherein the pathological image analysis model is the model adopted by any one of claims 1 to 8 above.

10. A pathological image processing device based on deep learning, It is characterized in that include: An acquisition module, used for acquiring a pathological image of a target object to be processed; The acquisition module is further used to perform block processing on the pathological image of the target object to be processed to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1; A processing module, used for performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position; A generation module is used to obtain statistical results corresponding to T feature dimensions according to N first feature sets, N second feature sets and N third feature sets, wherein T is an integer greater than 1; generate fusion features corresponding to the target object according to the statistical results corresponding to the T feature dimensions, wherein the fusion features include P elements, wherein P is an integer greater than 1; obtain Q elements from the fusion features corresponding to the target object, wherein Q is greater than or equal to 1; and concatenate the Q elements to obtain a target fusion feature corresponding to the target object; The output module is used to call the pathological image analysis model to analyze and process the target fusion feature to output the pathological analysis result of the target object to be processed.

11. The device according to claim 10, It is characterized in that The processing module is specifically used for: Acquire a hematoxylin staining image set and an eosin staining image set according to the block image set, wherein the hematoxylin staining image set includes N hematoxylin staining images, the eosin staining image set includes N eosin staining images, the hematoxylin staining images have a corresponding relationship with the block images, and the eosin staining images have a corresponding relationship with the block images; Acquire a cell nucleus image set according to the block image set, wherein the cell nucleus image set includes N cell nucleus images, and the cell nucleus images have a corresponding relationship with the block images; According to the hematoxylin staining image set and the eosin staining image set, acquiring the N first feature sets corresponding to the block image set; According to the cell nucleus image set, acquiring the N second feature sets corresponding to the block image set; According to the cell nucleus image set, the N third feature sets corresponding to the block image set are obtained.

12. The device according to claim 11, It is characterized in that The processing module is specifically used for: According to the cell nucleus image set, obtaining the cell nucleus centroid position corresponding to each cell nucleus image; Generating a Voronoi diagram corresponding to each cell nuclear image according to the cell nuclear centroid position corresponding to each cell nuclear image in the cell nuclear image set; The N third feature sets are obtained according to the Voronoi diagram corresponding to each cell nucleus image in the cell nucleus image set.

13. The device according to claim 10, It is characterized in that The generation module is specifically used for: According to the N first feature sets, the N second feature sets and the N third feature sets, statistics are performed on the distribution of the N block images in the T feature dimensions to obtain T frequency distribution histograms, wherein the feature dimensions have a corresponding relationship with the frequency distribution histograms; According to each frequency distribution histogram in the T frequency distribution histograms, generating a statistical result corresponding to each frequency distribution histogram; Generate statistical results corresponding to the T feature dimensions according to the statistical results corresponding to each frequency distribution histogram; The statistical results corresponding to the T feature dimensions are concatenated to obtain the fused features corresponding to the target object.

14. The device according to claim 10, It is characterized in that The output module is specifically used to call the pathological image analysis model to analyze and process the target fusion feature to output the lesion metastasis probability and at least one importance feature of the target object to be processed; The processing module is also used to display the pathological analysis results after the output module calls the pathological image analysis model to analyze and process the target fusion features to output the pathological analysis results of the target object to be processed; or, send the pathological analysis results to the client so that the client displays the pathological analysis results.

15. The device according to claim 10, It is characterized in that The acquisition module is further used to perform feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets and N third feature sets, and then determine at least one target block image from the N block images according to the N first feature sets, the N second feature sets and the N third feature sets; The processing module is also used to highlight the at least one target block image in the pathological image of the target object to be processed, or to send at least one target block image to the client so that the client can highlight the at least one target block image in the pathological image of the target object to be processed.

16. The device according to claim 10, It is characterized in that The output module is further used to perform block processing on the pathological image of the target object to be processed to obtain a set of block images, and then call the molecular typing model to process the set of block images to output the molecular typing corresponding to each block image in the block images; The processing module is further used to perform statistics on the molecular typing corresponding to each block image in the block image set to obtain H statistical counting results, wherein H is an integer greater than 1; The acquisition module is also used to determine the target molecular typing corresponding to the pathological image to be processed according to the H statistical counting results; The acquisition module is also used to acquire the pathological image analysis model from the set of pathological image analysis models to be selected according to the target molecular typing, wherein the set of pathological image analysis models to be selected includes at least one pathological image analysis model to be selected, each pathological image analysis model to be selected corresponds to a type of molecular typing, and the pathological image analysis model corresponds to the target molecular typing.

17. The device according to any one of claims 10 to 16, It is characterized in that The acquisition module is specifically used to extract an interesting ROI region from the pathological image of the target object to be processed; and to acquire the block image set from the ROI region by a sliding window method.

18. A model training device, It is characterized in that include: An acquisition module, used for acquiring a set of pathological images to be trained, wherein the set of pathological images to be trained includes at least one positive sample pathological image and at least one negative sample pathological image; The acquisition module is further used to acquire a set of block images to be trained for each pathological image to be trained in the set of pathological images to be trained, wherein the pathological image to be trained belongs to a positive sample pathological image or a negative sample pathological image, and the set of block images to be trained includes M block images to be trained, where M is an image greater than or equal to 1; The acquisition module is further used to acquire, for each pathological image to be trained in the pathological image set to be trained, M first feature sets to be trained, M second feature sets to be trained, and M third feature sets to be trained according to the block image set to be trained, wherein the first feature set to be trained includes at least one feature associated with the morphology of the block image to be trained, the second feature set to be trained includes at least one feature associated with the morphology of the cell nucleus, and the third feature set to be trained includes at least one feature associated with the position of the cell nucleus; A generating module, configured to generate, for each pathological image to be trained in the set of pathological images to be trained, fusion features to be trained according to the M first feature sets to be trained, the M second feature sets to be trained and the M third feature sets to be trained; A determination module, configured to determine, for the set of pathological images to be trained, target fusion features to be trained according to the fusion features to be trained corresponding to each pathological image to be trained; A training module is used to train the pathological image analysis model to be trained based on the fusion features of the target to be trained corresponding to each pathological image to be trained, using a loss function to obtain a pathological image analysis model, wherein the pathological image analysis model is the model adopted by any one of claims 1 to 8 above.

19. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 8, or the method according to claim 9.

20. A terminal device, It is characterized in that include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, including the method according to any one of claims 1 to 8, or the implementation of the method according to claim 9; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

21. A server, It is characterized in that include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, including the method according to any one of claims 1 to 8, or the implementation of the method according to claim 9; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

22. A medical system, It is characterized in that include: Probes, circuits, processors, and displays; A circuit for stimulating the probe to obtain a pathological image of a target object to be processed; A processor, for processing a pathological image of a target object to be processed; A display for displaying the results of pathological analysis; The processor further performs the following steps: Performing block processing on the pathological image of the target object to be processed to obtain a block image set, wherein the block image set includes N block images, where N is an image greater than or equal to 1; Performing feature classification and extraction processing on the block image set to obtain N first feature sets, N second feature sets, and N third feature sets, wherein the first feature set includes at least one feature associated with the block image morphology, the second feature set includes at least one feature associated with the cell nucleus morphology, and the third feature set includes at least one feature associated with the cell nucleus position; Obtain statistical results corresponding to T feature dimensions according to the N first feature sets, the N second feature sets, and the N third feature sets, where T is an integer greater than 1; Generate a fusion feature corresponding to the target object according to the statistical results corresponding to the T feature dimensions, wherein the fusion feature includes P elements, and P is an integer greater than 1; Obtain Q elements from the fusion features corresponding to the target object, where Q is greater than or equal to 1; The target fusion features are analyzed and processed using a pathological image analysis model to output the pathological analysis results of the target object to be processed.