Image processing method, device and computer equipment based on artificial intelligence

Abstract

The present application relates to an image processing method, device, computer equipment and storage medium based on artificial intelligence. The method comprises: obtaining an image to be processed; performing element region detection on the image to be processed to determine the element region in the image to be processed; using a target element region detection method based on artificial intelligence to detect the target element region in the image to be processed; finding an envelope for the detected target element region to generate a target element envelope region; fusing the element region and the target element envelope region to obtain the target element region contour. The present method can accurately detect and divide the image into regions.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer device and storage medium based on artificial intelligence. Background Art

[0002] Computer Vision (CV) is a simulation of biological vision using computers and related equipment. Its main task is to obtain three-dimensional information of the corresponding scene by processing the collected pictures or videos, just like humans and many other creatures do every day. Among them, object recognition and detection is a very basic and important research direction in computer vision. Object recognition and detection, that is, given an input picture, automatically find common objects in the picture and output their category and position. Specific examples include face detection, vehicle detection, tissue detection, etc., which have been widely used in manufacturing, inspection, document analysis, medical diagnosis and other fields.

[0003] In current object recognition and detection processing methods, only a single object is often recognized and detected, but the contour of the target object area in the image cannot be effectively recognized and divided, resulting in low accuracy of image area division. Summary of the invention

[0004] Based on this, it is necessary to provide an artificial intelligence-based image processing method, device, computer equipment and storage medium that can accurately detect and divide images into regions in order to solve the above technical problems.

[0005] An image processing method based on artificial intelligence, the method comprising:

[0006] Get the image to be processed;

[0007] Performing element region detection on the image to be processed to determine the element region in the image to be processed;

[0008] Adopting the target element area detection method based on artificial intelligence to detect the target element area in the image to be processed;

[0009] Finding the envelope of the detected target element area and generating the target element envelope area;

[0010] The element area and the target element envelope area are fused to obtain the target element area contour.

[0011] An image processing device based on artificial intelligence, the device comprising:

[0012] An image acquisition module, used for acquiring an image to be processed;

[0013] An element region detection module is used to perform element region detection on the image to be processed to determine the element region in the image to be processed;

[0014] A target element region detection module is used to detect the target element region in the image to be processed by using an artificial intelligence-based target element region detection method;

[0015] An envelope generation module is used to find the envelope of the detected target element area and generate the target element envelope area;

[0016] The regional fusion module is used to fuse the element region and the target element envelope region to obtain the target element region contour.

[0017] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Get the image to be processed;

[0019] Performing element region detection on the image to be processed to determine the element region in the image to be processed;

[0020] Adopting the target element area detection method based on artificial intelligence to detect the target element area in the image to be processed;

[0021] Finding the envelope of the detected target element area and generating the target element envelope area;

[0022] The element area and the target element envelope area are fused to obtain the target element area contour.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0024] Get the image to be processed;

[0025] Performing element region detection on the image to be processed to determine the element region in the image to be processed;

[0026] Adopting the target element area detection method based on artificial intelligence to detect the target element area in the image to be processed;

[0027] Finding the envelope of the detected target element area and generating the target element envelope area;

[0028] The element area and the target element envelope area are fused to obtain the target element area contour.

[0029] The above-mentioned artificial intelligence-based image processing method, device, computer equipment and storage medium detect the target element area in the image to be processed by the artificial intelligence-based target element area detection method, and find the envelope of the target element envelope area by comparing the element area, and then fuse the element area determined by the element area detection and the target element envelope area to obtain the target element area contour. By detecting the target element area by the artificial intelligence-based target element area detection method to find the envelope, the detection result of the artificial intelligence target element area detection method can be effectively used to obtain an accurate target element envelope area. At the same time, by fusing the element area in the image to be processed and the target element envelope area, the target element envelope area can be corrected using the result of the element area detection, thereby improving the accuracy of the detected and divided target element area contour. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A diagram showing an application environment of an image processing method based on artificial intelligence in one embodiment;

[0031] Figure 2 A schematic diagram of a flow chart of an image processing method based on artificial intelligence in one embodiment;

[0032] Figure 3 A schematic diagram of a process of element region detection in one embodiment;

[0033] Figure 4 is a schematic diagram of a tumor in a HE staining image in one embodiment;

[0034] Figure 5 It is a flowchart of the traditional IC ratio analysis method;

[0035] Figure 6 A comparison diagram of a HE-stained WSI image and a PD-L1-stained WSI image in one embodiment;

[0036] Figure 7 is a comparison diagram of a HE staining WSI image and a PD-L1 staining WSI image in another embodiment;

[0037] Figure 8 This is a comparison diagram of a HE staining WSI image and a PD-L1 staining WSI image in another embodiment;

[0038] Fig. 9 A schematic diagram of a flow chart of an image processing method based on artificial intelligence in another embodiment;

[0039] Fig.10 A schematic diagram of a result of color segmentation processing in one embodiment;

[0040] Fig.11 for Fig.10A schematic diagram of the result of the binary segmentation process in the illustrated embodiment;

[0041] Fig.12 is a dyed image in one embodiment;

[0042] Fig.13 for Fig.12 A schematic diagram of the result of detecting the slice tissue area in the embodiment shown;

[0043] Fig.14 is a network model structure in one embodiment;

[0044] Fig.15 A cancer cell region probability map output by a model in one embodiment;

[0045] Fig.16 for Fig.15 Schematic diagram of the cancer cell region determined in the embodiment;

[0046] Fig.17 for Fig.16 Schematic diagram of connected domain detection results in the embodiment;

[0047] Fig.18 for Fig.17 Schematic diagram of the removal result after removing the black hole in the embodiment;

[0048] Fig.19 for Fig.18 Schematic diagram of the result after morphological expansion in the embodiment;

[0049] Fig. 20 for Fig.16 and Fig.18 A schematic diagram of the fusion result in the illustrated embodiment;

[0050] Fig.21 for Fig. 20 Schematic diagram of the result of superimposing the dyeing image in the embodiment;

[0051] Fig. 22 is a structural block diagram of an image processing device based on artificial intelligence in one embodiment;

[0052] Fig.23 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0055] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. Among them, computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, trying to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0056] The solution provided in the embodiments of the present application involves computer vision technology of artificial intelligence, which is specifically illustrated by the following embodiments.

[0057] The artificial intelligence-based image processing method provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The terminal 102 collects the image to be processed, such as a landscape image, a scene image, a slice image, etc., and the terminal 102 sends the image to be processed to the server 104 for detection and division of the target element area contour. After receiving the image to be processed, the server 104 detects the target element area in the image to be processed by a target element area detection method based on artificial intelligence, and searches for the envelope of the element area to obtain the target element envelope area, and then fuses the element area determined by the element area detection and the target element envelope area to obtain the target element area contour. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices with shooting functions, and can also be various cameras or scanning devices and other devices that can collect images. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0058] In one embodiment, Figure 2 As shown, an image processing method based on artificial intelligence is provided, and the method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0059] Step 202: Obtain an image to be processed.

[0060] Among them, the image to be processed is an image that needs to be detected and divided into regions. The image to be processed can be an image taken or scanned in various scenes, such as landscape images taken in outdoor scenes, scene images taken in crowd gathering scenes, slice images scanned in medical scenes, and other types of images. The image to be processed includes elements captured by shooting or scanning, such as trees photographed in landscape images, people photographed in scene images, etc., and the elements have different categories, such as trees of different species, people of different genders, etc. At present, most of the image recognition only judges the elements in the image, and cannot accurately identify the regions of elements of different categories, and cannot obtain the distribution of the target element region contour in the image, which is not conducive to subsequent image analysis based on the image to be processed, such as analyzing the shapes of various poplars in the trees in the image, and the shapes of various men in the scene image.

[0061] Specifically, the image to be processed can be obtained by photographing a camera or scanning a scanning device, and sent to a server through a network. The server performs recognition and division processing on the received image to be processed to identify the contour of the target element region, thereby determining the distribution of the contour of the target element region in the image to be processed, which is convenient for subsequent image analysis. In specific implementation, the target element can be set according to actual needs. For example, when the element included in the image to be processed is a tree, the target element can be a set tree species; for example, when the element included in the image to be processed is a cell, the target element can be a specific type of cell, such as a white blood cell, a somatic cell, a cancer cell, etc.

[0062] Step 204: perform element region detection on the image to be processed to determine the element region in the image to be processed.

[0063] Among them, the element is the object captured by shooting or scanning in the image to be processed, and the regional contour of the target element to be detected is a certain type of element. For example, if the tree is an element, the target element can be different organs of the tree (such as leaves, trunks, etc.) or trees of different species, etc.; if the element is a dog, the target element can be different parts of the dog. The element area refers to the distribution range of the element in the image to be processed, such as the distribution range of trees of various species in the image to be processed. In specific applications, the element is determined according to the actual scene of image processing. For example, the element can also be a cell in a slice image, and the element area is the cell area in the slice image, that is, the slice tissue area. Element area detection is used to detect the element area in the image to be processed. Specifically, various object detection algorithms, such as R-CNN (Regions with Convolutional Neural Networks features, regions based on convolutional neural network features) algorithm, YOLO (You Only Look Once) algorithm, SSD (Single Shot Detector, single shot detector) algorithm, etc., can be used to perform element area detection on the image to be processed to determine the element area in the image to be processed. In addition, prior knowledge may be used, such as the prior knowledge of color features of elements and non-elements in the image to be processed, and the color features of each pixel in the image to be processed may be used to detect the element region in the image to be processed.

[0064] Specifically, after obtaining the image to be processed, the server determines a corresponding element region detection method according to the type characteristics of the elements in the image to be processed, and performs element region detection on the image to be processed by using the element region detection method to determine the element region in the image to be processed.

[0065] Step 206: Detect the target element region in the image to be processed by using an artificial intelligence-based target element region detection method.

[0066] Among them, the target element can be a subcategory element in the element, that is, the target element belongs to the element that needs to be detected in the element. The target element area is the distribution range of the target element in the image to be processed. The target element area detection method is based on artificial intelligence, such as the target element area detection based on a machine learning model. For example, the target element area can be detected through various image segmentation algorithm models such as FCN (Fully Convolutional Networks) model, SegNet (A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, a deep convolutional encoding-decoding architecture for image segmentation) model, Linknet (Relational Embedding for Scene Graph, scene graph relational embedding) model, etc., to obtain the target element area in the image to be processed.

[0067] Specifically, when performing target element region detection on the image to be processed, a target element region detection method based on artificial intelligence, such as using a pre-trained detection model to perform target element region detection on the image to be processed, detects the target element region in the image to be processed.

[0068] Step 208: Find the envelope of the detected target element area and generate the target element envelope area.

[0069] The target element envelope area is a preliminary target element area contour obtained by searching for an envelope based on the target element area. Specifically, after the target element area in the image to be processed is detected by the target element area detection method based on artificial intelligence, the server searches for an envelope for the target element area. Specifically, the target element areas can be connected to find an envelope line that can include each target element area, and the target element envelope area is generated according to the area surrounded by the envelope line. By searching for an envelope for the target element area, the influence of errors in the image processing process can be reduced, and an accurate target element envelope area can be obtained.

[0070] In a specific implementation, the server may connect the target element areas to obtain a connected area, and then remove the non-target element areas within the connected area, thereby ensuring accurate target element recognition results within the connected area. Finally, the server smoothes the connected area from which the non-target element areas have been removed, such as by smoothing through morphological dilation to obtain a target element envelope area with smooth edges. At the same time, morphological dilation is used to denoise smaller areas, thereby improving the accuracy of target element envelope area recognition and division.

[0071] Step 210: fuse the element region and the target element envelope region to obtain the target element region contour.

[0072] After obtaining the element area and the target element envelope area, the two are merged to obtain the target element area contour, which overall shows the distribution range of the target element in the image to be processed. Specifically, the server can obtain the intersection between the element area and the target element envelope area, and determine the intersection of the element area and the target element envelope area as the final target element area contour. The target element area contour combines the detection results of the element area detection of the image to be processed and the target element area detection results of the image to be processed based on artificial intelligence, which can effectively reduce the error impact in a single detection method, and the accuracy of the target element area contour detection and division is high.

[0073] In the above-mentioned image processing method based on artificial intelligence, the target element region in the image to be processed is detected by the target element region detection method based on artificial intelligence, and the envelope of the target element envelope region is obtained by matching the element region, and then the element region determined by the element region detection and the target element envelope region are fused to obtain the target element region contour. By detecting the target element region by the target element region detection method based on artificial intelligence and finding the envelope, the detection result of the target element region detection method based on artificial intelligence can be effectively used to obtain an accurate target element envelope region, and at the same time, by fusing the element region in the image to be processed and the target element envelope region, the result of the element region detection can be used to correct the target element envelope region, thereby improving the accuracy of the detected and divided target element region contour.

[0074] In one embodiment, Figure 3 As shown, the processing process of element region detection, i.e., performing element region detection on the image to be processed to determine the element region in the image to be processed, includes:

[0075] Step 302: determine the channel difference feature corresponding to each pixel in the image to be processed.

[0076] In this embodiment, the element region detection is performed on the image to be processed using the prior knowledge of the element color features, so as to determine the element region in the image to be processed. Specifically, the elements and non-elements in the image to be processed have obvious color differences, such as when there is an obvious color difference between snow and non-snow in the image to be processed of a snowy scene, the element region detection can be achieved based on the color features of each pixel in the image to be processed; for example, in the stained image of a physiological section, the stained element and the non-stained element have obvious color differences, and the stained element region detection can be achieved based on the color features of each pixel in the stained image.

[0077] Among them, the channel difference feature is used to describe the color feature of each pixel in the image to be processed, and can be obtained specifically according to the difference between the channel values ​​corresponding to each color channel of the pixel. For example, when the image to be processed is an RGB color image, each pixel in the image to be processed corresponds to three color channels of R, G and B respectively, then the channel difference feature can be obtained according to the difference between the channel values ​​of the three color channels of R, G and B, such as the maximum difference among the channel values ​​of the three color channels of R, G and B. In specific applications, the channel values ​​corresponding to each color channel of each pixel in the image to be processed can be traversed, and the channel difference feature of the corresponding pixel can be determined according to each channel value.

[0078] Step 304 , performing color segmentation on the image to be processed based on the channel difference feature corresponding to each pixel point to obtain a color segmented image.

[0079] Among them, the color segmentation image is obtained by performing color segmentation processing on the processed image according to the channel difference characteristics corresponding to each pixel point. For example, the pixel value of each pixel point of the processed image can be updated according to the channel difference characteristics corresponding to each pixel point, so as to realize color segmentation of the processed image using the channel difference characteristics corresponding to each pixel point, and obtain the color segmentation image. The color segmentation image reflects the color characteristics of the image to be processed. For example, if the channel difference feature is the maximum difference in the channel values ​​of each color channel, when the channel difference feature is directly replaced with the pixel value of the corresponding pixel during color segmentation, the obtained color segmentation image reflects whether each pixel of the image to be processed tends to be grayscale or color. If it tends to be grayscale, the maximum difference in the channel values ​​of each color channel is small, and the value of the channel difference feature is small, then the corresponding pixel in the color segmentation image tends to be black, otherwise the corresponding pixel in the color segmentation image tends to be white. Therefore, the colored area and non-colored area in the image to be processed can be intuitively determined based on the color segmentation image. If the image to be processed is a stained image obtained by staining an element, then the colored area in the image to be processed is the element area, thereby using prior knowledge to achieve effective detection of the element area.

[0080] Specifically, after obtaining the channel difference features corresponding to each pixel point in the image to be processed, color segmentation is performed on the image to be processed based on each channel difference feature, such as updating the pixel value of the corresponding pixel point in the image to be processed based on each channel difference feature, so as to realize color segmentation of the image to be processed using the channel difference features and obtain a color segmented image.

[0081] Step 306: perform binary segmentation on the color segmentation image to obtain element regions in the image to be processed.

[0082] Among them, the binary segmentation is to further segment the color segmentation image through binarization processing, so as to determine the element area in the image to be processed. Specifically, the pixel points in the color segmentation image can be traversed through the preset binary segmentation threshold for binarization processing, and the pixel value of each pixel point in the color segmentation image can be binary mapped, such as mapping to white or black, so that the element area in the image to be processed can be intuitively determined according to the binary segmentation result. The binary segmentation threshold can be flexibly set according to actual needs, such as being set to a certain pixel value, or being set according to the pixel value of each pixel point in the color segmentation image, so as to more accurately perform binary segmentation on the color segmentation image.

[0083] Specifically, after obtaining the color segmentation image, the server performs binarization segmentation on the color segmentation image. For example, the server can obtain a preset binarization segmentation threshold, or set the binarization segmentation threshold according to the channel difference feature corresponding to each pixel point in the image to be processed, and perform binarization segmentation on the color segmentation image through the binarization segmentation threshold, thereby obtaining the element area in the image to be processed according to the binarization segmentation result.

[0084] In specific applications, the image to be processed can be divided into regions of interest in advance, such as dividing the image to be processed into regions of interest with a size of 100*100 pixels, obtaining the regions of interest corresponding to the image to be processed, and performing element region detection on each region of interest to obtain the element region in each region of interest, and finally obtaining the element region in the image to be processed by splicing the element regions in each region of interest. By dividing the image to be processed into regions of interest and then performing element region detection, the amount of data processed by a single element region detection can be reduced, and each region of interest can be processed in parallel, which can effectively improve the processing efficiency of element region detection.

[0085] In this embodiment, the prior knowledge of element color features is used to perform color segmentation on the image to be processed, and the color segmented image is further mapped and segmented in combination with binary segmentation. This can effectively utilize the prior knowledge to perform accurate element region detection while increasing the contrast between element regions and non-element regions through binary segmentation, thereby further improving the accuracy of element region detection and accurately determining the element region in the image to be processed.

[0086] In one embodiment, determining the channel difference features corresponding to each pixel in the image to be processed includes: determining the pixel value corresponding to each pixel in the image to be processed; obtaining the color channel values ​​of the corresponding pixel according to the pixel value; determining the maximum channel difference between the color channel values, and obtaining the channel difference features according to the maximum channel difference.

[0087] In this embodiment, a channel difference feature is obtained according to the maximum channel difference between the channel values ​​corresponding to each color channel of each pixel in the image to be processed, and pixel value mapping is performed on each pixel in the image to be processed based on the maximum channel difference to achieve color segmentation of the image to be processed and obtain a color segmentation image. Among them, the color channel value of each pixel in the image to be processed corresponds to the value of the pixel in each color channel, and the value of the color channel value is related to the color space in which the image to be processed is located. For example, if the image to be processed is an RGB image, the color space of the image to be processed includes R, G and B, which correspond to 3 color channel values. For example, if the pixel value of a certain pixel is (123, 15, 255), the color channel values ​​of the pixel are 123, 15 and 255 respectively. The maximum channel difference refers to the maximum value of the difference between the color channel values ​​corresponding to the pixel. Specifically, the color channel values ​​can be subtracted from each other and the maximum value can be found based on the calculated difference. The maximum channel difference reflects the degree of difference between pixels in each color channel. The larger the maximum channel difference, the greater the difference between the pixel points corresponding to each color channel, the richer the color of the pixel point, and the pixel point tends to be colorful; if the maximum channel difference is smaller, the difference between the pixel points corresponding to each color channel is smaller, the color of the pixel point is not rich, and the pixel point tends to be grayscale.

[0088] Specifically, when determining the channel difference feature corresponding to each pixel in the image to be processed, the server determines the pixel value corresponding to each pixel in the image to be processed. Specifically, the server can traverse each pixel in the image to be processed and read the pixel value of each pixel. After obtaining the pixel value corresponding to each pixel, the server parses the pixel value corresponding to each pixel to determine the color channel value of each pixel. Generally, the pixel value of a pixel is obtained by combining the color channel values ​​of each color channel in a certain order. Based on the combination rule of the pixel value and the color channel image, the color channel value corresponding to each color channel can be determined according to the pixel value. The server calculates the difference between each color channel value and determines the maximum channel difference of the pixel according to the difference result. In addition, the server can also determine the maximum value and the minimum value from each color channel value respectively, and directly determine the maximum channel difference of the pixel according to the absolute value of the difference between the maximum value and the minimum value. The server obtains the channel difference feature according to the obtained maximum channel difference, such as directly using the maximum channel difference as the channel difference feature of the corresponding pixel.

[0089] Furthermore, the image to be processed is color segmented based on the channel difference feature corresponding to each pixel point to obtain a color segmented image, including: pixel value mapping of the image to be processed according to the maximum channel difference corresponding to each pixel point to obtain a color segmented image.

[0090] Among them, pixel value mapping refers to mapping the pixel value of each pixel in the image to be processed based on the maximum channel difference, so as to perform color segmentation on the image to be processed according to the maximum channel difference. For example, the corresponding channel difference pixel can be determined according to the maximum channel difference, and the pixel value of the corresponding pixel in the image to be processed can be updated according to the channel difference pixel, thereby realizing color segmentation of the image to be processed. After traversing all the pixels in the image to be processed, a color segmented image is obtained.

[0091] Specifically, after the server determines the maximum channel difference corresponding to each pixel in the image to be processed, the pixel value of the corresponding pixel in the image to be processed is mapped according to the maximum channel difference corresponding to each pixel, so as to update the pixel value of the corresponding pixel based on the maximum channel difference, thereby realizing color segmentation of the image to be processed. For example, the server can generate a channel difference pixel according to the maximum channel difference. For example, if the maximum channel difference of a certain pixel in the image to be processed is 200, an RGB channel difference pixel (200, 200, 200) can be generated, and the channel difference pixel is mapped to the pixel value of the pixel, so as to map the pixel value of the pixel through the maximum channel difference, thereby realizing color segmentation of the image to be processed and obtaining a color segmented image.

[0092] In this embodiment, a channel difference feature is obtained according to the maximum channel difference between the channel values ​​corresponding to each color channel of each pixel in the image to be processed, and pixel value mapping is performed on each pixel in the image to be processed based on the maximum channel difference. The color features of each pixel can be effectively used to perform accurate color segmentation on the image to be processed, thereby ensuring the accuracy of the color segmented image.

[0093] In one embodiment, a color segmentation image is binarized to obtain an element region in the image to be processed, including: denoising the color segmentation image to obtain a denoised image; obtaining a binarization segmentation threshold; based on the binarization segmentation threshold and the pixel value corresponding to each pixel point in the denoised image, binarizing the denoised image to obtain the element region in the image to be processed.

[0094] Among them, the color segmentation image is denoised, such as fuzzy denoising is performed on the color segmentation image to remove the noise in the color segmentation image, so as to further improve the credibility of the color segmentation image. The binary segmentation threshold can be preset according to actual needs, such as being preset to a certain pixel value, or being preset according to the pixel value of each pixel in the color segmentation image. Binarization segmentation is to further segment the color segmentation image through binarization processing, and specifically, each pixel can be binarized according to the size relationship between the binary segmentation threshold and the pixel value corresponding to each pixel in the denoised image, so as to obtain the element area in the image to be processed according to the binary segmentation.

[0095] Specifically, when the color segmentation image is binarized, the server performs denoising on the color segmentation image, such as the server can perform Gaussian blur denoising on the color segmentation image. Gaussian blur denoising can make the intensity of a pixel point correlated with the surrounding points, reduce the influence of mutation, and obtain a denoised image with interference removed. The server obtains the binarization segmentation threshold, and specifically can query the pre-set binarization segmentation threshold, or can determine the binarization segmentation threshold according to the pixel value of each pixel point in the color segmentation image or the denoised image. For example, the binarization segmentation threshold can be determined according to the maximum channel difference corresponding to each pixel point in the image to be processed. In addition, if the image to be processed is divided into each region of interest for element region detection, the binarization segmentation threshold can be set according to each region of interest, such as determining the binarization segmentation threshold according to the maximum channel difference corresponding to each pixel point in each region of interest, so as to flexibly set the binarization segmentation threshold of each region of interest, improve the adaptation of the binarization segmentation threshold to each region of interest, improve the effectiveness of binarization segmentation, and ensure the accuracy of element region division. After obtaining the binary segmentation threshold, the server performs binary segmentation on the denoised image based on the binary segmentation threshold and the pixel value corresponding to each pixel in the denoised image. For example, the server can perform binary mapping on the pixel value corresponding to each pixel in the denoised image based on the size relationship between the binary segmentation threshold and the pixel value corresponding to each pixel in the denoised image, thereby realizing the binary segmentation of the denoised image and obtaining the element area in the image to be processed.

[0096] In this embodiment, after the color segmentation image is denoised to remove the noise in the color segmentation image, the denoised image is binarized based on the binary segmentation threshold, so that the denoised image is segmented by the binary segmentation threshold to accurately divide the element area and non-element area in the image to be processed.

[0097] In one embodiment, based on a binary segmentation threshold and a pixel value corresponding to each pixel in the denoised image, a denoised image is binarized and segmented to obtain an element region in the image to be processed, including: determining a pixel value corresponding to each pixel in the denoised image; performing pixel binarization mapping on the pixel values ​​corresponding to each pixel in the denoised image based on the binary segmentation threshold to obtain a pixel mapping result corresponding to each pixel in the denoised image; and obtaining the element region in the image to be processed according to the pixel mapping result corresponding to each pixel in the denoised image.

[0098] In this embodiment, the pixel values ​​corresponding to each pixel point in the denoised image are mapped to a binary value through a binary segmentation threshold, and the element region in the image to be processed is obtained according to the pixel mapping results corresponding to each pixel point. Specifically, when the denoised image is binarized, the server determines the pixel values ​​corresponding to each pixel point in the denoised image, such as traversing each pixel point in the denoised image to determine the pixel values ​​corresponding to each pixel point. The server traverses each pixel point in the denoised image, and performs pixel binarization mapping on the pixel values ​​corresponding to each pixel point in the denoised image based on the binary segmentation threshold. Specifically, the binary segmentation threshold is numerically compared with the pixel values ​​corresponding to each pixel point, and the pixel points whose pixel values ​​are greater than the binary segmentation threshold and the pixel points whose pixel values ​​are not greater than the binary segmentation threshold are mapped to different pixel values, such as black or white pixel values, respectively, so as to intuitively divide the element pixel points and non-element pixel points in the image to be processed, and the element region in the image to be processed can be determined according to each element pixel point. Among them, element pixels refer to pixels belonging to elements in the image to be processed, and non-element pixels refer to pixels that do not belong to elements, that is, all pixels in the element area are element pixels, and all pixels in the non-element area are non-element pixels. After the pixel values ​​corresponding to each pixel in the denoised image are mapped to pixels based on the binary segmentation threshold, and the pixel mapping results corresponding to each pixel in the denoised image are obtained, the server obtains the element area in the image to be processed according to the pixel mapping results corresponding to each pixel. For example, after the pixel values ​​corresponding to each pixel in the denoised image are mapped to pixels, the pixels mapped to element pixel values ​​in the pixel mapping results can be determined as element pixels, and the pixels mapped to non-element pixel values ​​can be determined as non-element pixels. For example, when the element pixel value is white and the non-element pixel value is black, the pixel in the denoised image with a white pixel mapping result is determined as an element pixel, and the pixel with a black pixel mapping result is determined as a non-element pixel. The element area in the image to be processed can be determined according to all element pixels in the area with a white pixel mapping result, thereby achieving accurate segmentation of the element area.

[0099] In one embodiment, an artificial intelligence-based target element region detection method is used to detect the target element region in the image to be processed, including: querying a target element region detection model pre-trained using training image data labeled with target element regions; performing target element region detection on the image to be processed using the target element region detection model to obtain a target element region probability map; performing target element region determination on the target element region probability map, and obtaining the target element region in the image to be processed based on the determination result.

[0100] Among them, the target element region detection model is obtained by pre-training the training image data with the target element region labeled. The target element region detection model can detect the target element region of the input image and output the target element region probability map. The target element region probability map represents the probability that each pixel in the image to be processed is the target element. By judging the probability in the target element region probability map, the target element region in the image to be processed can be determined.

[0101] Specifically, when performing target element region detection on the image to be processed, the server queries a pre-trained target element region detection model. The target element region detection model is trained by annotating the training image data of the target element region. Compared with annotating the target element region contour, the training of the training image data of the target element region can be directly based on the target element with obvious features. The annotation is more accurate and can ensure the training effect of the training image data. The server performs target element region detection on the image to be processed by the target element region detection model obtained by querying. Specifically, the image to be processed can be input into the target element region detection model for target element region detection. The target element region detection model outputs the detection result, and the detection result includes a target element region probability map. The server determines the target element region probability map output by the target element region detection model to determine whether each pixel in the target element region probability map is a target element, thereby determining the target element region in the image to be processed according to the determination result.

[0102] In this embodiment, a target element region detection model obtained by pre-training training image data with labeled target element regions is used to perform target element region detection on the image to be processed, and a target element region probability map output by the target element region detection model is judged to determine the target element region in the image to be processed. By performing target element region detection with the pre-trained target element region detection model, effective detection of the target element region can be achieved, and the accurate target element region can be identified from the image to be processed.

[0103] In one embodiment, target element region detection is performed on an image to be processed through a target element region detection model to obtain a target element region probability map, including: obtaining regions of interest obtained by region division processing of the image to be processed; inputting each region of interest into the target element region detection model for target element region detection, and obtaining region of interest probability maps corresponding to each region of interest output by the target element region detection model; and splicing each region of interest probability map to obtain a target element region probability map corresponding to the image to be processed.

[0104] In this embodiment, after the image to be processed is divided into various regions of interest, the target element region detection model performs target element region detection on each region of interest respectively, and the detection results corresponding to each region of interest are spliced ​​to obtain the target element region detection result corresponding to the image to be processed. Among them, the region of interest is obtained by dividing the image to be processed, such as dividing the image to be processed according to a preset size, so that the image to be processed is divided into several regions of interest, which reduces the amount of data processed at a single time and improves the data processing efficiency. In specific implementation, the size of the region of interest can be set according to actual needs, and a fixed size can be directly set, such as 1000*1000 pixels.

[0105] Specifically, when the target element region detection is performed on the image to be processed through the target element region detection model, the server obtains each region of interest obtained by region division processing of the image to be processed. Among them, the region division of the region of interest can be realized when the element region detection is performed on the image to be processed. Then, when the target element region detection is performed on the image to be processed, the regions of interest obtained by the region division during the element region detection are directly obtained. The server inputs each region of interest into the target element region detection model to perform target element region detection, and obtains each region of interest probability map output by the target element region detection model. The region of interest probability map records the probability of the pixel point in the corresponding region of interest being detected as the target element region. The server splices the region of interest probability maps corresponding to each region of interest, and obtains the target element region probability map corresponding to the image to be processed, thereby realizing the target element region detection of the image to be processed.

[0106] In this embodiment, target element region detection is performed on each region of interest obtained by region division processing of the image to be processed through a target element region detection model, and the detection results corresponding to each region of interest are spliced ​​to obtain a target element region detection result corresponding to the image to be processed. This can reduce the amount of data processed by a single target element region detection, and can perform parallel processing on each region of interest, which can effectively improve the processing efficiency of the target element region detection.

[0107] In one embodiment, a target element region is determined on a target element region probability map, and a target element region in an image to be processed is obtained according to a determination result, including: obtaining a probability threshold; performing probability binary mapping on the target element region probability map based on the probability threshold and the probability value corresponding to each pixel point in the target element region probability map, and obtaining a probability mapping result for each pixel point in the target element region probability map; and obtaining the target element region in the image to be processed according to the probability mapping result for each pixel point in the target element region probability map.

[0108] Among them, the probability threshold is pre-set according to actual needs, such as being set to 0.5, and the probability threshold is used to determine the target element region of the target element region probability map, and determine the target element region in the image to be processed according to the target element region determination. Specifically, after obtaining the target element region probability map output by the target element region detection model, the server obtains a preset probability threshold, and the probability threshold is used to perform binary mapping on the probability value of each pixel in the target element region probability map. The server performs probability binary mapping on the target element region probability map based on the probability threshold and the probability value corresponding to each pixel in the target element region probability map. Specifically, the server can compare the probability value corresponding to each pixel in the target element region probability map and the probability threshold, and perform probability binary mapping on each pixel according to the comparison result to obtain the probability mapping result of each pixel in the target element region probability map. For example, if the probability threshold is 0.5, and the probability value of pixel A in the target element region probability map is 0.8 and the probability value of pixel B is 0.2, the pixel value of pixel A is mapped to a white pixel value and the pixel value of pixel A is mapped to a black pixel value, thereby realizing the binary mapping of pixel A and pixel B. After traversing all the pixels, the binary mapping of the target element region probability map is realized. After obtaining the probability mapping results of each pixel in the target element region probability map, the server obtains the target element region in the image to be processed based on each probability mapping result, such as the server can determine the pixel mapped to the target element pixel value in each probability mapping result as the target element pixel, and determine the target element region in the image to be processed based on all the target element pixels.

[0109] In this embodiment, the target element area probability map is probability-binarized by a preset probability threshold, so as to determine the target element area in the image to be processed according to the probability mapping result of each pixel in the target element area probability map. The target element area and non-target element area in the image to be processed can be intuitively and accurately divided through the binary mapping, thereby ensuring the accuracy of the target element area division.

[0110] In one embodiment, the envelope of the detected target element region is found to generate the target element envelope region, including: performing regional connectivity on the target element region in the processed image to generate a target element connected region; removing non-target element regions within the target element connected region to obtain an updated target element connected region; and performing morphological expansion on the updated target element connected region to obtain the target element envelope region.

[0111] In this embodiment, after the detected target element area is connected, the non-target elements in the connected area are removed, thereby filtering the non-target elements within the range of the connected area, and then the edge of the connected area is smoothed by morphological expansion, and at the same time, denoising is achieved for the smaller area on the edge of the connected area, thereby finding the envelope of the detected target element area and obtaining the target element envelope area with high recognition and division accuracy.

[0112] Specifically, after detecting the target element region in the image to be processed, the server performs regional connectivity on the target element region in the image to be processed. Specifically, the target element region and the non-target region in the image to be processed can be traversed by a filter of a preset size, so as to connect the target element region in the image to be processed by the filter, and generate a target element connected region. The server then filters out the non-target elements within the generated target element connected region, thereby ensuring that the target element connected region is inside the target element connected region. Specifically, the server removes the non-target element region in the target element connected region, such as the server can directly update the non-target element region in the target element connected region to the target element region, and obtain the updated target element connected region. The inside of the updated target element connected region is the target element region. Further, the server performs smoothing on the edge of the updated target element connected region, specifically by performing morphological expansion on the updated target element connected region, to obtain the target element envelope region. By performing morphological expansion on the updated target element connected region, the edge of the updated target element connected region can be smoothed, and at the same time, effective denoising is performed on the smaller edge area of ​​the updated target element connected region, further improving the recognition and division accuracy of the target element envelope region.

[0113] In one embodiment, target element regions in an image to be processed are connected to generate target element connected regions, including: filtering and dividing the target element region and non-target region in the image to be processed to obtain filter regions; performing region type analysis based on pixel values ​​of pixel points in each filter region to obtain analysis results; and connecting the filter regions whose analysis results are connected region types to generate target element connected regions.

[0114] In this embodiment, after the target element area and non-target area in the image to be processed are divided into filter areas, region type analysis is performed on each filter area respectively, and based on the analysis results, it is determined whether the filter area is a connected area type, and the filter areas of the connected area type are connected to generate target element connected areas.

[0115] Specifically, when connecting the target element area in the image to be processed, the server performs filtering division on the target element area and the non-target area in the image to be processed to obtain each filtering area. The size of the filtering division can be determined according to the size of the filter. For example, when the size of the filter is 10*10 pixels, the target element area and the non-target area in the image to be processed are divided according to the filter to obtain filtering areas of 10*10 pixels. After determining each filtering area, the server performs regional type analysis on the pixel values ​​of the pixels in each filtering area. Specifically, the server can traverse each filtering area, and for each pixel in each filtering area, if the target element pixel is detected, that is, the target element pixel exists in the filtering area, then the type of the filtering area is determined to be a connected area type, that is, the filtering area can be identified as the area of ​​the target element area, that is, the filtering area can be connected. After obtaining the analysis results of each filter, the server connects the filtering areas in each filtering area whose corresponding analysis results are connected area types to obtain the target element connected area. In the specific implementation, when connecting the filter areas of each connected area type, the server maps the pixel values ​​of the pixel points in the filter areas of each connected area type to the target element pixel values, and then connects the filter areas of each connected area type after the mapping process, thereby generating the target element connected area.

[0116] In a specific application, the image to be processed can be directly traversed through the set filter. For the range of each filter area processed by the filter, if there is a target element pixel point, it can be judged according to the pixel value of each pixel point in the range processed by the filter. If there is a pixel value corresponding to the target element pixel point, it is considered that the target element pixel point exists, and the filter area is determined as a connected area type, and the pixel value of each pixel point in the connected area is mapped to the pixel value corresponding to the target element pixel point. For example, if the pixel value corresponding to the target element pixel point is a white pixel value, if the filter detects a white pixel point in the filter area, all the pixels in the filter area are mapped to white pixels, that is, the filter area is determined to be an area that needs to be connected. After the filter traverses the image to be processed, the filter areas of each connected area type are connected to generate a target element connected area.

[0117] In one embodiment, the artificial intelligence-based image processing method further includes: superimposing the target element region contour onto the image to be processed, and identifying the target element region contour in the image to be processed by a predetermined identification method.

[0118] Among them, the identification method can be flexibly set in advance according to actual needs, such as a contour line, symbol, etc. of a highlight color. Specifically, after the target element area contour is detected from the image to be processed, the server superimposes the target element area contour on the image to be processed, and identifies the target element area contour in the image to be processed by a predetermined identification method, thereby intuitively identifying the target element area contour in the image to be processed, which is convenient for subsequent image analysis based on the target element area contour and the image to be processed.

[0119] The present application also provides an application scenario, which applies the above-mentioned artificial intelligence-based image processing method. Specifically, the application of the artificial intelligence-based image processing method in this application scenario is as follows:

[0120] The image to be processed is a landscape image, a scene image, etc., wherein the elements are landscape objects in the landscape, such as trees, snow, sand and gravel, etc., or the elements are people or animals in the scene image; the target elements are the number of different species or different parts of the number, or different people or animals, such as men, women, children or the elderly, etc. The target elements can also be animals of different species such as dogs and cats in the scene image. In addition, the target elements can also be different parts of a person or an animal, such as various parts of a person's hair, face, etc., and the trunk, limbs or head of an animal, etc.: the target elements are determined according to the contour of the area to be detected.

[0121] In a specific application, the server obtains a captured landscape image, performs element region detection on the landscape image, and determines the tree region in the landscape image. On the other hand, an artificial intelligence-based target element region detection method is used to detect the leaf region in the landscape image, and the detected leaf region is searched for an envelope to generate a leaf envelope region, and the tree region and the leaf envelope region are fused to obtain a leaf region contour. The leaf region contour intuitively displays the distribution of the leaf region contour in the landscape image, which is convenient for subsequent image analysis.

[0122] The present application also provides an application scenario, which applies the above-mentioned image processing method based on artificial intelligence. Specifically, the application of the image processing method based on artificial intelligence in this application scenario is as follows:

[0123] In this application scenario, the image to be processed is a WSI (Whole Slide Image) stained image, the elements are cells in the slice, and the target elements are different types of cells, such as white blood cells, T cells or cancer cells.

[0124] In a specific application, the WSI staining image is a slice image based on the breast pathology PD-L1 (programmed cell death-Ligand 1, programmed cell death factor ligand 1, which is a ligand of PD-1). In the treatment of breast cancer, the PD-L1 method can be used to obtain the number of immune cells (PD-L1 expression), and the ability to resist cancer is further evaluated based on the number of immune cells, that is, by estimating the proportion of stained immune cells (IC, immune cell) in the tumor area, it is used to select the method of cancer treatment. However, there are currently only qualitative methods for estimating the number of immune cells, and no quantitative methods, which leads to inaccurate estimation of the IC ratio value.

[0125] At present, the immunohistochemical method for evaluating the efficacy of immune therapy based on PD-1 (programmed death 1, a receptor protein on the surface of immune cells T cells) / PD-L1 has become a research focus and hotspot. Changes in PD-L1 expression are associated with tumor progression and poor prognosis, and PD-L1 is considered to be an effective biomarker for predicting prognosis. Breast cancer is the most common malignant tumor in women, and precision medicine provides an opportunity for more refined and individualized treatment of breast cancer. Currently, a diagnostic and treatment method combining paclitaxel protein binders has been proposed for the treatment of unresectable locally advanced TNBC (Triple-Negative Breast Cancer) or metastatic TNBC. Figure 4 As shown, it includes the manifestation of tumors in 5 types of HE staining images. In Figure A, it is a local observation image after the HE staining image is magnified 10 times. It can be seen that after the slice tissue is stained, it can be divided into the stroma around the tumor and the intratumor stroma in the tumor area. Figure BE shows the results of staining of different slice tissues. The distribution of the tumor can be determined through the HE staining image for subsequent diagnosis and treatment analysis. Roche proposed a guiding method for interpreting IC values ​​based on SP142 stain. Specifically, the HE staining (hematoxylin-eosin staining) WSI image is observed first, the tumor area in the HE staining image is obtained, and then the observation results are matched to the PD-L1 staining image, and the IC score is determined by further combining the estimated HE staining image and PD-L1 staining image.

[0126] like Figure 5As shown in the figure, in the traditional IC ratio analysis process, the HE staining WSI image is first observed to determine the tumor area, and then the observation results of the HE staining WSI image are matched to the PD-L1 staining WSI image, and the IC ratio is estimated by combining the two for image analysis. Specifically, when observing the HE staining WSI image, it is necessary to determine whether there is tumor and necrosis in the HE staining WSI image. At the same time, to ensure adequacy, at least 50 living tumor cells and stroma need to be detected. In addition, the area of ​​the tumor area can also be estimated. It can be seen that in the traditional processing method for estimating the IC ratio, two different WSI staining images are required, that is, two slides need to be made, one stained with HE stain and the other stained with PD-L1 stain, which significantly increases the workload of image analysis. Moreover, it is difficult to keep the images obtained by the two stains completely consistent, because to make two slides with different stains, two different cancer tissues need to be cut from the wax block, and there will be errors in the acquired WSI images, as well as position translation and rotation, which affect the estimation of the IC ratio. As shown in the figure, the HE staining WSI image is not stained with PD-L1 stain, and the image is not stained with PD-L1 stain. Figure 6-8 As shown in FIG. 1 , HE staining WSI images and PD-L1 staining WSI images in different embodiments are compared. It can be seen that Figure 6 The HE staining WSI image and PD-L1 staining WSI image have a certain degree of rotation. Figure 7 The HE staining WSI image and PD-L1 staining WSI image in the middle are flipped. Figure 8 The HE-stained WSI image and the PD-L1-stained WSI image have a certain degree of rotation and translation. Therefore, when the tumor tissue area obtained from the HE-stained WSI image is found on the PD-L1-stained WSI image, it is difficult to effectively correspond the tumor area to the PD-L1-stained WSI image for IC ratio estimation, which greatly reduces the accuracy of IC ratio estimation and leads to low accuracy of image analysis.

[0127] Based on this, the present application provides an artificial intelligence-based image processing method applied to this scenario, which can effectively and accurately divide and detect the contour of the target element area, facilitating the subsequent image analysis and processing of the IC ratio value. Specifically, Fig. 9 As shown, the server obtains a staining image, which is an image obtained by staining the WSI image with a PD-L1 stain. The server performs tissue cell detection and cancer cell area detection on the staining image respectively to obtain a slice tissue area and a cancer cell area. The server fuses the slice tissue area and the cancer cell area to determine the contour of the tumor area, and superimposes the contour of the tumor area on the staining image, so that the IC ratio value can be estimated directly based on the superposition result later without comparing the two staining images for analysis, which can effectively ensure the accuracy of the IC ratio value estimation.

[0128] Furthermore, for slice tissue region detection, prior knowledge can be used for effective detection, that is, non-tissue areas tend to be grayscale, and tissue areas are obviously stained. Therefore, by judging whether a pixel is closer to gray or color, it is possible to judge whether the pixel belongs to the background area or to stained tissue cells. Considering the large size of WSI images, the stained image can be divided into regions to obtain ROIs (Regions of Interest), and tissue region detection can be achieved by traversing and processing each ROI one by one. Specifically, in each ROI, the local threshold is calculated using the color difference of the pixel points to achieve segmentation of the background area and tissue area based on the current ROI, thereby segmenting the slice tissue area from the stained image. For each ROI, slice tissue region detection is as follows (1):

[0129] Diff roi (x,y)=Abs{Max[ROI(x,y,i)]| i＝1,2,3 -Min[ROI(x,y,i)]| i＝1,2,3} (1)

[0130] Among them, Diff roi (x,y) refers to the difference between the maximum and minimum pixel values ​​in the three different color channels (i=1,2,3) at the point (x,y) within the ROI. Among them, Max[ROI(x,y,i)]| i＝1,2,3 Used to determine the maximum value of the three color channels at point (x,y), Min[ROI(x,y,i)]| i＝1,2,3 It is used to determine the minimum value of the three color channels at the (x, y) point, and Abs{} is used to determine the absolute value of the difference between the maximum and minimum values. In this embodiment, the color channels of the image (i=1,2,3) are R, G and B channels. In addition, they can also be channels of other color spaces, such as HSV (Hue-Saturation-Value) color space. Generally speaking, if a pixel is a color pixel, the value of Abs{} is larger; if it is a grayscale pixel, the value of Abs{} is relatively small. Fig.10 As shown in FIG. 1 , in one embodiment, the color segmentation of the stained image ROI (x, y) on the left is performed to obtain the color segmentation image Diff on the right. roi (x,y), in the color segmentation image, the gray pixel area is the slice tissue area.

[0131] After obtaining the color segmentation image, the color segmentation image is further binarized to determine the element area in the image to be processed. The specific processing is as follows (2):

[0132] Mask roi(x,y)=Binary{GaussianBlur[Diff roi (x,y)]}| Thresh＝Diff(x,y) / 10 (2)

[0133] Among them, the color segmentation image Diff roi (x, y) is blurred and denoised, specifically using Gaussian blur denoising GaussianBlur[], then, for each pixel, binary calculation is performed, specifically the binary segmentation threshold Thresh = Diff(x, y) / 10 is used for binary segmentation, and the binary segmentation result Mask is obtained rol (x, y), the slice tissue in the stained image can be determined based on the binary segmentation result. Fig.11 As shown, Fig.10 In the embodiment shown, the color segmentation image is subjected to a binary segmentation process to obtain a binary segmentation result, wherein the white area is the slice tissue area.

[0134] like Figure 12-13 As shown, in one embodiment, Fig.12 For the stained image, Fig.13 For Fig.12 The slice tissue area (white area) is determined after the staining image is subjected to slice tissue detection.

[0135] On the other hand, for cancer cell region detection, the pre-trained Linknet network model (Chaurasia2017) is used to detect cancer cell regions in stained images. Fig.14In the Linknet network model structure, conv means convolution, and full-conv means transposed convolution. / 2 means downsampling by a factor of 2, which is achieved by convolution with a step size. *2 means upsampling by a factor of 2. BN (Batch Normalization) and ReLU (Rectified Linear Unit) are used between convolution layers. The left half is the Encoder Block, and the right half is the Decoder Block. There is an initial block at the beginning of the Encoder Block, and the rest is composed of residual blocks. The training network model uses 900 annotated images (832*832 pixels, 0.848 microns / pixel). In specific applications, if there is annotated data for the tumor cell area, the model can be directly trained for tumor area detection, but in actual engineering, sufficient annotations are very difficult to obtain, and the tumor area actually contains various complex pathological tissues. Even if annotations are performed, it is difficult to train a deep learning network, that is, it is difficult to obtain an actually available tumor area. Based on this, considering that the characteristics of cancer cells are very different from other pathological tissues, it is not necessary to annotate the tumor area, but only the cancer cell area, and the model training is performed through the annotated images of the cancer cell area. On the test set of more than 200 images, the training can be terminated when the model accuracy reaches F1 (F-measure, F value) = 0.89, Recall (recall rate) = 0.89, Precision (accuracy) = 0.91, and the trained model is obtained.

[0136] The trained Linknet network model is used to predict the ROT of the stained images one by one, and then spliced ​​to obtain the predicted cancer cell region probability map, whose value is 0 to 1. Based on the cancer cell region probability map, a probability threshold of 0.5 is used for binarization mapping to obtain the binary cancer cell region result. Fig.15 As shown in FIG. 1 , a probability map of cancer cell regions is obtained by predicting cancer cell regions in a stained image based on a Linknet network model in one embodiment; Fig.16 As shown in FIG. 1 , the cancer cell region result is obtained after the probability map of the cancer cell region is binarized using a probability threshold of 0.5, where the white area is the cancer cell region.

[0137] Furthermore, in the post-processing of the slice tissue area and the cancer cell area, the stroma connecting the cancer cell areas is taken into account to find the envelope of the entire cancer cell area, that is, to determine the tumor area. When looking for the envelope of the cancer cell area, it is achieved by calculating the connected area. Specifically, a 10*10 pixel filter is used to traverse the image of the cancer cell area one by one. Within the 10*10 pixel range, as long as there are pixels greater than 0 (white pixels), the entire 10*10 pixel ROI is regarded as a connected area. Among them, the 10*10 pixel filter corresponds to the use of a scaled image of about 50 microns / pixel. Furthermore, a 4*4 circular kernel is used to achieve morphological expansion, obtain smooth edges, and denoise smaller areas at the same time, further improving the accuracy of the overall contour of the cancer area.

[0138] like Fig.17 As shown, for Fig.16 The connected domain detection result obtained after the cancer cell region is connected; Fig.18 For Fig.17 The connected domain detection results in the removal result after removing the black holes; Fig.19 For Fig.18 The overall contour of the cancer area is obtained after morphological dilation of the removal result.

[0139] After obtaining the overall contour result of the cancer area, the overall contour results of the slice tissue area and the cancer area are fused. Specifically, the tumor area can be obtained by calculating the intersection of the two. The tumor area refers to the cancer cell area and part of the stroma between the cancer cell areas. The slice tissue area detection can obtain the area of ​​all tissues, which includes the tumor area and the non-tumor area. The non-tumor area can be determined by using the cancer cell area detection. The fusion of these two results can obtain the detection result of the tumor area. Fig. 20 , for fusion Fig.18 Cancer cell regions and Fig.16 The outline of the cancer cell area obtained by slicing the tissue area in the image is the result of the tumor area. Further, the tumor area is superimposed on the original staining image to mark the detected tumor area directly in the staining image. Fig.21 As shown, Fig. 20 The tumor area in the image is superimposed on the stained image and marked. It can be seen that the edge of the tumor area is relatively close, and the stained area in the non-tumor area is also well removed.

[0140] It should be understood that although Figure 2-3The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-3 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0141] In one embodiment, Fig. 22 As shown, an image processing device 2200 based on artificial intelligence is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: an image acquisition module 2202, an element region detection module 2204, a target element region detection module 2206, an envelope generation module 2208 and a region fusion module 2210, wherein:

[0142] An image acquisition module 2202 is used to acquire an image to be processed;

[0143] An element region detection module 2204 is used to perform element region detection on the image to be processed to determine the element region in the image to be processed;

[0144] The target element region detection module 2206 is used to detect the target element region in the image to be processed by using a target element region detection method based on artificial intelligence;

[0145] An envelope generation module 2208 is used to find an envelope for the detected target element region and generate a target element envelope region;

[0146] The region fusion module 2210 is used to fuse the element region and the target element envelope region to obtain the target element region contour.

[0147] In this embodiment, the target element region detection module detects the target element region in the image to be processed by using a target element region detection method based on artificial intelligence, and the envelope generation module searches for the envelope of the target element region to obtain the target element envelope region, and then the region fusion module fuses the element region determined by the element region detection module through element region detection and the target element envelope region to obtain the target element region contour. By detecting the target element region by searching for the envelope based on the target element region detection method based on artificial intelligence, the detection result of the target element region detection method based on artificial intelligence can be effectively used to obtain an accurate target element envelope region, and at the same time, by fusing the element region in the image to be processed and the target element envelope region, the target element envelope region can be corrected using the result of the element region detection, thereby improving the accuracy of the detected and divided target element region contour.

[0148] In one embodiment, the element region detection module 2204 includes a channel difference feature determination module, a color segmentation module and a binarization segmentation module; wherein: the channel difference feature determination module is used to determine the channel difference feature corresponding to each pixel in the image to be processed; the color segmentation module is used to perform color segmentation on the image to be processed based on the channel difference feature corresponding to each pixel to obtain a color segmentation image; the binarization segmentation module is used to perform binarization segmentation on the color segmentation image to obtain the element region in the image to be processed.

[0149] In one embodiment, the channel difference feature determination module includes a pixel value determination module, a channel value determination module and a maximum channel difference determination module; wherein: the pixel value determination module is used to determine the pixel value corresponding to each pixel point in the image to be processed; the channel value determination module is used to obtain the color channel values ​​of the corresponding pixel point according to the pixel value; the maximum channel difference determination module is used to determine the maximum channel difference between the color channel values, and obtain the channel difference feature according to the maximum channel difference; the color segmentation module is also used to map the pixel value of the image to be processed according to the maximum channel difference corresponding to each pixel point to obtain a color segmented image.

[0150] In one embodiment, the binary segmentation module includes a denoising module, a segmentation threshold acquisition module and a segmentation processing module; wherein: the denoising module is used to perform denoising on the color segmentation image to obtain a denoised image; the segmentation threshold acquisition module is used to obtain a binary segmentation threshold; the segmentation processing module is used to perform binary segmentation on the denoised image based on the binary segmentation threshold and the pixel value corresponding to each pixel point in the denoised image to obtain the element area in the image to be processed.

[0151] In one embodiment, the segmentation processing module includes a pixel value analysis module, a binarization mapping module and a mapping result analysis module; wherein: the pixel value analysis module is used to determine the pixel value corresponding to each pixel point in the denoised image; the binarization mapping module is used to perform pixel binarization mapping on the pixel value corresponding to each pixel point in the denoised image based on the binarization segmentation threshold, and obtain the pixel mapping result corresponding to each pixel point in the denoised image; the mapping result analysis module is used to obtain the element area in the image to be processed according to the pixel mapping result corresponding to each pixel point in the denoised image.

[0152] In one embodiment, the target element region detection module 2206 includes a model query module, a model processing module and a model result processing module; wherein: the model query module is used to query a target element region detection model pre-trained using training image data with marked target element regions; the model processing module is used to perform target element region detection on the image to be processed through the target element region detection model to obtain a target element region probability map; the model result processing module is used to perform target element region determination on the target element region probability map, and obtain the target element region in the image to be processed based on the determination result.

[0153] In one embodiment, the model processing module includes a region of interest module, a probability map acquisition module and a probability map stitching module; wherein: the region of interest module is used to obtain each region of interest obtained by region division processing of the image to be processed; the probability map acquisition module is used to input each region of interest into the target element region detection model for target element region detection, and obtain the region of interest probability map corresponding to each region of interest output by the target element region detection model; the probability map stitching module is used to stitch the probability maps of each region of interest to obtain the target element region probability map corresponding to the image to be processed.

[0154] In one embodiment, the model result processing module includes a probability threshold acquisition module, a probability threshold mapping module and a target element area module; wherein: the probability threshold acquisition module is used to obtain the probability threshold; the probability threshold mapping module is used to perform probability binary mapping on the target element area probability map based on the probability threshold and the probability value corresponding to each pixel point in the target element area probability map, and obtain the probability mapping result of each pixel point in the target element area probability map; the target element area module is used to obtain the target element area in the image to be processed according to the probability mapping result of each pixel point in the target element area probability map.

[0155] In one embodiment, the envelope generation module 2208 includes a region connectivity module, a region filling module and a region expansion module; wherein: the region connectivity module is used to perform region connectivity on the target element region in the processed image to generate a target element connected region; the region filling module is used to remove the non-target element region within the target element connected region to obtain an updated target element connected region; the region expansion module is used to perform morphological expansion on the updated target element connected region to obtain a target element envelope region.

[0156] In one embodiment, the region connectivity module includes a filter region acquisition module, a region type analysis module and a connectivity processing module; wherein: the filter region acquisition module is used to filter and divide the target element region and the non-target region in the image to be processed to obtain each filter region; the region type analysis module is used to perform region type analysis based on the pixel values ​​of the pixel points in each filter region to obtain analysis results; the connectivity processing module is used to connect the filter regions whose analysis results are connected region types to generate target element connected regions.

[0157] In one embodiment, it further includes an overlay processing module for overlaying the target element region outline onto the image to be processed, and identifying the target element region outline in the image to be processed by a predetermined identification method.

[0158] For the specific definition of the image processing device based on artificial intelligence, please refer to the definition of the image processing method based on artificial intelligence above, which will not be repeated here. Each module in the above-mentioned image processing device based on artificial intelligence can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0159] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.23 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method based on artificial intelligence is implemented.

[0160] Those skilled in the art will understand that Fig.23 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0162] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0163] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0164] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0165] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image processing method based on artificial intelligence, characterized in that: The method comprises: Get the image to be processed; Determine the pixel value corresponding to each pixel point in the image to be processed; Obtaining the color channel values ​​of the corresponding pixel points according to the pixel values; Determine a maximum value and a minimum value from the color channel values, and determine a maximum channel difference between the color channel values ​​according to a difference between the maximum value and the minimum value; Generating channel difference pixels corresponding to each pixel point according to the maximum channel difference values ​​corresponding to each pixel point, and performing pixel value mapping on the image to be processed through each channel difference pixel to obtain a color segmentation image; Performing binary segmentation on the color segmented image to obtain an element region in the image to be processed, wherein the element region represents a distribution region of an element in the image to be processed, and the element is an object included in the image to be processed; Adopting an artificial intelligence-based target element region detection method to detect a target element region in the image to be processed, wherein the target element region represents a distribution region of a target element in the image to be processed, and the target element is a specific type of element among the elements; Finding an envelope for the detected target element area to generate a target element envelope area; An intersection between the element region and the target element envelope region is determined, and a target element region contour is obtained according to the intersection, wherein the target element region contour represents a distribution range of the detected target element in the image to be processed.

2. According to the method of claim 1, the step of performing binary segmentation on the color segmentation image to obtain the element region in the image to be processed comprises: Performing denoising processing on the color segmented image to obtain a denoised image; Get the binary segmentation threshold; Based on the binary segmentation threshold and the pixel value corresponding to each pixel point in the denoised image, the denoised image is binary segmented to obtain the element region in the image to be processed.

3. The method according to claim 2, wherein the binary segmentation of the denoised image is performed based on the binary segmentation threshold and the pixel value corresponding to each pixel point in the denoised image to obtain the element region in the image to be processed, comprising: Determine a pixel value corresponding to each pixel in the denoised image; Based on the binary segmentation threshold, pixel values ​​corresponding to each pixel point in the denoised image are respectively mapped into binary pixels to obtain pixel mapping results corresponding to each pixel point in the denoised image; The element region in the image to be processed is obtained according to the pixel mapping result corresponding to each pixel point in the denoised image.

4. The method according to claim 1, wherein the detecting the target element region in the image to be processed by using an artificial intelligence-based target element region detection method comprises: Query a target element region detection model pre-trained using training image data annotating target element regions; Performing target element region detection on the image to be processed by using the target element region detection model to obtain a target element region probability map; The target element region probability map is subjected to target element region determination, and the target element region in the image to be processed is obtained according to the determination result.

5. The method according to claim 4, wherein the target element region detection is performed on the image to be processed by the target element region detection model to obtain a target element region probability map, comprising: Acquire each region of interest obtained by region division processing of the image to be processed; Inputting each region of interest into the target element region detection model to perform target element region detection, and obtaining a region of interest probability map corresponding to each region of interest output by the target element region detection model; The probability maps of the regions of interest are spliced ​​to obtain a probability map of the target element region corresponding to the image to be processed.

6. The method according to claim 4, wherein the target element region determination is performed on the target element region probability map, and the target element region in the image to be processed is obtained according to the determination result, comprising: Get probability threshold; Based on the probability threshold and the probability value corresponding to each pixel in the target element region probability map, the target element region probability map is subjected to probability binary mapping to obtain a probability mapping result of each pixel in the target element region probability map; The target element region in the image to be processed is obtained according to the probability mapping result of each pixel point in the target element region probability map.

7. The method according to claim 1, wherein the step of searching for an envelope of the detected target element region to generate a target element envelope region comprises: Performing regional connectivity on target element regions in the image to be processed to generate target element connected regions; Remove the non-target element region in the target element connected region to obtain an updated target element connected region; The updated connected area of ​​the target element is morphologically expanded to obtain the envelope area of ​​the target element.

8. The method according to claim 7, wherein the step of performing regional connectivity on the target element region in the image to be processed to generate a target element connected region comprises: Filtering and dividing the target element area and the non-target area in the image to be processed to obtain filtering areas; Performing a region type analysis based on the pixel values ​​of the pixel points in each of the filtering regions to obtain an analysis result; The filter areas whose analysis results are of the connected area type are connected to generate the target element connected area.

9. The method according to any one of claims 1 to 8, further comprising: The target element region outline is superimposed on the image to be processed, and the target element region outline is identified in the image to be processed by a predetermined identification method.

10. An image processing device based on artificial intelligence, the device comprising: An image acquisition module, used for acquiring an image to be processed; An element region detection module is used to perform element region detection on the image to be processed to determine an element region in the image to be processed, wherein the element region represents a distribution region of an element in the image to be processed, and the element is an object included in the image to be processed; The element region detection module comprises: A channel difference feature determination module is used to determine the channel difference feature corresponding to each pixel in the image to be processed; A color segmentation module, used to perform color segmentation on the image to be processed based on the channel difference feature corresponding to each pixel point to obtain a color segmented image; A binary segmentation module, used to perform binary segmentation on the color segmentation image to obtain element regions in the image to be processed; The channel difference feature determination module comprises: A pixel value determination module, used to determine the pixel value corresponding to each pixel point in the image to be processed; A channel value determination module, used for obtaining each color channel value of a corresponding pixel point according to the pixel value; A maximum channel difference determination module, used to determine the maximum channel difference between the color channel values, and obtain a channel difference feature according to the maximum channel difference; The color segmentation module is further used to perform pixel value mapping on the image to be processed according to the maximum channel difference corresponding to each pixel point to obtain a color segmented image; A target element region detection module, used to detect a target element region in the image to be processed by using an artificial intelligence-based target element region detection method, wherein the target element region represents a distribution region of a target element in the image to be processed, and the target element is a specific type of element among the elements; An envelope generation module, used to find the envelope of the detected target element area and generate a target element envelope area; The regional fusion module is used to determine the intersection between the element region and the target element envelope region, and obtain the target element region contour according to the intersection, wherein the target element region contour represents the distribution range of the detected target element in the image to be processed.

11. The device according to claim 10, characterized in that The binary segmentation module comprises: A denoising module, used for performing denoising processing on the color segmentation image to obtain a denoised image; A segmentation threshold acquisition module is used to obtain a binary segmentation threshold; The segmentation processing module is used to perform binary segmentation on the denoised image based on the binary segmentation threshold and the pixel value corresponding to each pixel point in the denoised image to obtain the element area in the image to be processed.

12. The device according to claim 11, characterized in that The segmentation processing module comprises: A pixel value analysis module, used to determine the pixel value corresponding to each pixel point in the denoised image; A binary mapping module, used to perform pixel binary mapping on the pixel values ​​corresponding to each pixel point in the denoised image based on the binary segmentation threshold, to obtain a pixel mapping result corresponding to each pixel point in the denoised image; The mapping result analysis module is used to obtain the element area in the image to be processed according to the pixel mapping result corresponding to each pixel point in the denoised image.

13. The device according to claim 10, characterized in that The target element region detection module is further used to query a target element region detection model pre-trained using training image data with target element regions annotated; perform target element region detection on the image to be processed using the target element region detection model to obtain a target element region probability map; The target element region probability map is subjected to target element region determination, and the target element region in the image to be processed is obtained according to the determination result.

14. The device according to claim 13, characterized in that The target element region detection module is also used to obtain the regions of interest obtained by region division processing of the image to be processed; input the regions of interest into the target element region detection model for target element region detection, and obtain the region of interest probability maps corresponding to the regions of interest output by the target element region detection model; and splice the region of interest probability maps to obtain the target element region probability map corresponding to the image to be processed.

15. The device according to claim 13, characterized in that The target element region detection module is further used to obtain a probability threshold; based on the probability threshold and the probability value corresponding to each pixel point in the target element region probability map, the target element region probability map is subjected to probability binary mapping to obtain a probability mapping result of each pixel point in the target element region probability map; The target element region in the image to be processed is obtained according to the probability mapping result of each pixel point in the target element region probability map.

16. The device according to claim 10, characterized in that The envelope generation module is further used to connect the target element region in the image to be processed to generate a target element connected region; remove the non-target element region in the target element connected region to obtain an updated target element connected region; The updated connected area of ​​the target element is morphologically expanded to obtain the envelope area of ​​the target element.

17. The device according to claim 16, characterized in that The envelope generation module is further used to filter and divide the target element area and the non-target area in the image to be processed to obtain each filtering area; perform regional type analysis based on the pixel values ​​of the pixel points in each filtering area to obtain an analysis result; The filter areas whose analysis results are of the connected area type are connected to generate the target element connected area.

18. The device according to any one of claims 10 to 17, characterized in that The device also includes: The overlay processing module is used to overlay the target element region outline onto the image to be processed, and identify the target element region outline in the image to be processed by a predetermined identification method.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images