IMAGE PROCESSING METHOD, APPARATUS, DEVICE, AND PROGRAM

Abstract

An image processing method, apparatus, device, readable storage medium, and program product relating to the field of medical data processing, the method comprising the steps of: acquiring a sample image (210), the sample image including an image obtained by collecting a sample to be analyzed within a preset wavelength band; acquiring a first image of the sample image corresponding to at least one preset wavelength within the preset wavelength band to obtain a pseudocolor image (220); performing region segmentation on the sample image based on differences in sample element types in the sample image to obtain a segmentation result (230); and determining an image region in the sample image including an element type to be identified based on the pseudocolor image and the region segmentation result (240).

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application filed on January 25, 2022, bearing application number 202210086842.8 and titled "Image processing method, apparatus, device, readable storage medium, and program product," the entire contents of which are incorporated herein by reference.

[0002] FIELD OF THE DISCLOSURE Embodiments of the present application relate to the field of medical data processing, and in particular to image processing methods, apparatus, devices, readable storage media and program products. [Background technology]

[0003] After a pathological specimen containing tumor tissue is surgically removed, the collected pathological specimen is usually further studied to locate and identify the boundaries of the tumor tissue in the pathological specimen, thereby enabling a more accurate analysis of the tumor tissue and obtaining more valuable medical analysis results.

[0004] In the related art, after the pathological specimen removed by surgery is fixed in formalin, the pathologist will usually determine the borders of the tumor tissue by observing it with the naked eye, or will use an X-ray device to scan the pathological specimen, and a doctor will interpret the X-ray image to determine the borders of the tumor tissue, and then perform the tumor tissue sampling.

[0005] However, when using the above method to determine the borders of tumor tissue, some lesions with unclear tumor beds are difficult to distinguish with the naked eye, and X-ray equipment is expensive, making it difficult to widely adopt the method. Summary of the Invention [Problem to be solved by the invention]

[0006] Embodiments of the present application provide image processing methods, devices, apparatus, readable storage media, and program products that can improve the accuracy of pathology sampling by utilizing the spectral characteristics of a first sample at different wavelengths to analyze a target sample. [Means for solving the problem]

[0007] The present application provides the following technical solutions.

[0008] According to one aspect of the present invention, acquiring a sample image, the sample image including an image obtained by collecting a sample to be analyzed within a predetermined wavelength range; obtaining a first image of the sample image corresponding to at least one preset wavelength within the preset wavelength band, thereby obtaining a pseudocolor image; obtaining a segmentation result by performing segmentation on the sample image based on differences in sample element types in the sample image, the sample element types including a classification target element type to be classified; determining image regions in the sample image that contain the element type to be identified based on the pseudocolor image and the segmentation result.

[0009] According to another aspect of the invention, a sample acquisition module for acquiring a sample image, the sample image including an image obtained by collecting a sample to be analyzed within a preset wavelength band; an image acquisition module for acquiring a first image of the sample image corresponding to at least one preset wavelength within the preset wavelength band, thereby obtaining a pseudo-color image; a segmentation module for performing segmentation on the sample image based on differences in sample element types in the sample image to obtain a segmentation result, the sample element types including a classification target element type to be classified; and a region determination module for determining an image region in the sample image that includes the element type to be identified based on the pseudocolor image and the region segmentation result.

[0010] According to another aspect of the present application, there is provided a computer device comprising a processor and a memory, wherein the memory stores at least one instruction, at least a portion of a program, code set or instruction set which is loaded and executed by the processor to realize an image processing method according to any one of the above embodiments of the present application.

[0011] According to another aspect of the present application, there is provided a computer-readable storage medium having stored thereon at least one instruction, at least a portion of a program, code set or instruction set which, when loaded and executed by a processor, realises an image processing method as described in any one of the embodiments of the present application.

[0012] According to another aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored on a computer readable storage medium, the computer instructions being read from the computer readable storage medium by a processor of a computing device and executed to cause the computing device to perform the image processing method according to any one of the above embodiments. Effect of the Invention

[0013] The technical solutions provided by the embodiments of the present application can achieve at least the following beneficial effects:

[0014] By combining the pseudo-color image and the region division result to determine the image region, it is possible to avoid judging the size and region of the tumor tissue by only the naked eye observation and explanation by the doctor, and the inaccuracy of the judgment of the tumor tissue region of the patient is reduced. A sample to be analyzed is collected based on a preset wavelength band to obtain a sample image, at least one preset wavelength having a good effect is selected from the preset wavelength band, a first image corresponding to the preset wavelength is determined from the sample image, and the first image is processed to obtain a pseudo-color image that can relatively accurately represent the advantages of the preset wavelength. In addition, a region division is performed on the sample image based on the difference in the type of element to be identified in the sample image, and a region division result is obtained. By combining the pseudo-color image and the region division result to determine the image region including the type of element to be identified, the position information of the region to be identified (e.g., tumor tissue) is determined, improving the accuracy of pathological sampling while reducing the difficulty of pathological sampling, and analyzing using the spectral characteristics corresponding to the sample to be analyzed at different wavelengths, not only is the operation relatively simple, but the cost is relatively low, making it easier to apply widely. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 illustrates an implementation environment according to one exemplary embodiment of the present application. [Diagram 2] 1 is a flowchart of an image processing method according to an exemplary embodiment of the present application. [Diagram 3] FIG. 2 illustrates the capture of a sample image according to one exemplary embodiment of the present application. [Figure 4] FIG. 2 illustrates a sample image within a preset waveband according to one exemplary embodiment of the present application. [Diagram 5] 1 is a graph of spectral features corresponding to a sample image according to one exemplary embodiment of the present application. [Figure 6] FIG. 2 illustrates image processing of an analyzed sample according to one exemplary embodiment of the present application. [Figure 7]FIG. 2 illustrates a pseudo-color image synthesis according to one exemplary embodiment of the present application. [Figure 8] 4 is a flowchart of an image processing method according to another exemplary embodiment of the present application; [Figure 9] FIG. 13 illustrates the collection of a pathology sample according to another exemplary embodiment of the present application. [Figure 10] 1 is a flowchart of performing region segmentation on a sample image according to one exemplary embodiment of the present application; [Figure 11] 1 is a graph of a spectral feature corresponding to a hollow organ according to one exemplary embodiment of the present application; [Figure 12] 1 is a graph of spectral features corresponding to a kidney according to one exemplary embodiment of the present application; [Figure 13] 1 is a graph of a spectral feature corresponding to a mammary gland according to one exemplary embodiment of the present application; [Figure 14] 1 is a graph of spectral characteristics corresponding to a lung according to one exemplary embodiment of the present application. [Figure 15] FIG. 1 illustrates a monochromatic fill hint according to an exemplary embodiment of the present application. [Figure 16] 1 is a flow chart illustrating obtaining a prediction result according to one exemplary embodiment of the present application; [Figure 17] FIG. 2 illustrates a pictorial representation of four tissue classifications according to one exemplary embodiment of the present application. [Figure 18] 1A-1D are diagrams illustrating different image representations of renal cancer according to one exemplary embodiment of the present application. [Figure 19] 1A-1D show different image representations of the mammary gland according to one exemplary embodiment of the present application; [Figure 20] 1A-1C show different image representations of the mammary gland according to another exemplary embodiment of the present application; [Figure 21] 1 is a block diagram showing the structure of an image processing device according to one exemplary embodiment of the present application; [Figure 22] FIG. 2 is a block diagram showing the structure of an image processing device according to another exemplary embodiment of the present application. [Diagram 23]FIG. 2 illustrates a structure of a server according to one exemplary embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] First, the terms used in the embodiments of the present application will be briefly explained.

[0017] Artificial Intelligence (AI) refers to theories, methods, technologies, and application systems that use digital computers or devices controlled by digital computers to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use the knowledge to obtain optimal results. In other words, AI, as a comprehensive technology of computer science, aims to understand the essence of intelligence and create new intelligent machines that can react in a manner similar to human intelligence. In other words, AI gives machines the functions of sensing, reasoning, and decision-making by studying the design principles and realization methods of various intelligent machines.

[0018] AI technology, as a comprehensive discipline, covers a wide range of fields and includes both hardware and software technologies. AI foundation technologies generally include sensors, dedicated AI chips, cloud computing, distributed memory, big data processing technology, operation / interactive systems, electromechanical integration, etc. AI software technology mainly includes several directions such as computer vision technology, voice processing technology, natural language processing technology, and machine learning / deep learning.

[0019] Machine learning (ML) is an interdisciplinary field that involves many disciplines, such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It is a specialized study of how computers simulate or realize human learning behavior, and acquire new knowledge and skills, and reorganize existing knowledge structures to continually improve their own performance. As the core of artificial intelligence, machine learning is the fundamental route to giving computers intelligence, and is applied across various fields of artificial intelligence. Machine learning and deep learning usually include techniques such as artificial neural networks, trust networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0020] In the related art, the border of the tumor tissue is usually determined by a pathologist observing it with the naked eye, or an X-ray device is used to scan the pathological specimen, and a doctor interprets the X-ray image to determine the border of the tumor tissue and then performs the tumor tissue sampling. However, when using the above-mentioned method to determine the border of the tumor tissue, some lesions with unclear tumor beds are difficult to distinguish with the naked eye, and the X-ray device is expensive, so it is difficult to widely use.

[0021] In the embodiment of the present application, an image processing method is provided to improve the accuracy of pathology sampling by analyzing the sample by utilizing the spectral characteristics of the sample at different wavelengths. The image processing method obtained by training in the present application includes at least one of the following scenes in application:

[0022] 1. Medical applications In tumor resection surgery, the location of the tumor margin needs to be accurately known in order to realize the process of completely resecting the tumor area and prevent the recurrence of the patient's condition and avoid secondary surgery, and postoperative histopathological analysis is the gold standard for tumor diagnosis. In order to accurately obtain the patient's lesion information, the selection process of the pathological tissue block by the doctor is particularly important, and the selection of tissue blocks containing lesions will limit the pathologist's more accurate judgment, while the selection of excessive tissue blocks will greatly increase the workload of slice preparation and reduce medical efficiency. For example, when using the above image processing method, a tissue having a lesion (e.g., kidney organ, breast gland, etc.) is used as the analysis target sample, the analysis target sample is collected within a preset wavelength band to obtain a sample image, a first image corresponding to the preset wavelength is selected from the sample image to obtain a pseudo-color image, and a region segmentation result is obtained by performing region segmentation on the sample image based on the sample element type of the sample image, and the region segmentation result and the pseudo-color image are comprehensively analyzed to finally determine the image region corresponding to the tumor tissue relatively accurately, and the identification process for the image region can be realized. The above method can assist pathologists in finding lesion areas more quickly, and can also reduce the cost of equipment using imaging devices such as X-rays. By obtaining sample images using a spectrometer, which has a wider range of applications and is more economical, and analyzing the sample images with spectral information, the medical costs can be reduced and the accuracy of tumor tissue determination can be improved.

[0023] 2. Application to food inspection field Food safety is linked to life safety, and food often contains different components, unhealthy components, or inaccurate component ratios, all of which may cause food safety accidents. For example, the above image processing method is used to take the detected food as a sample to be analyzed, collect the sample to be analyzed within a preset wavelength band to obtain a sample image, select a first image corresponding to a preset wavelength from the sample image to obtain a pseudo-color image, and perform region segmentation on the sample image based on the sample element type of the sample image to obtain a region segmentation result, and comprehensively analyze the region segmentation result and the pseudo-color image to finally determine relatively accurately the regions corresponding to different components in the detected food, and determine the image regions corresponding to unhealthy components, thereby realizing the identification process for the image regions. The above method can help food supervision and management institutions to better supervise food, and can more accurately identify image regions based on the determination of the pseudo-color image based on the preset wavelength and the region segmentation result.

[0024] It should be noted that the above application scenes are merely illustrative, and the image processing method according to the present embodiment can also be applied to other scenes, and the present embodiment is not limited thereto.

[0025] In addition, in specific embodiments of the present application, with regard to related data such as user information, when applying the above examples of the present application to specific products or technologies, permission or consent must be obtained from the user, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0026] Next, an implementation environment according to an embodiment of the present application will be described. Exemplarily, as shown in FIG. 1, the implementation environment includes a terminal 110 and a server 120, and the terminal 110 and the server 120 are connected via a communication network 130.

[0027] In some embodiments, the terminal 110 is installed with an application program having an image collection function. In some embodiments, the terminal 110 is used to send a sample image to the server 120. The server 120 can determine an image region in the sample image that contains an element type to be identified based on the spectral information corresponding to the sample image through an image processing model 121, and mark the image region in a special manner and feed it back to the terminal 110 for display.

[0028] Here, the application method of the image processing model 121 is as follows. That is, a preset wavelength is selected from a preset wavelength band, a first image corresponding to the preset wavelength is determined from a sample image based on the preset wavelength, and the first image is processed to obtain a pseudo-color image. In addition, a region division is performed on the sample image based on the sample element type in the sample image to obtain a region division result corresponding to the sample image, and the region division result and the pseudo-color image are combined to determine an image region for indicating the position information of the element type to be identified in the sample image. For example, the sample to be analyzed is a pathological sample, and the image region determined by analyzing the sample to be analyzed is a region corresponding to tumor tissue, so that the region information corresponding to tumor tissue can be more accurately determined. The above process is an example of a non-unique situation of the application process of the image processing model 121.

[0029] The terminals include mobile terminals such as mobile phones, tablet computers, portable laptop computers, smart voice dialogue devices, smart home appliances, and in-vehicle terminals, but may also be realized as desktop computers, etc. The server may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0030] Among them, cloud technology refers to a type of trusted technology that integrates a series of resources such as hardware, application programs, and networks within a wide area network or local area network to realize data calculation, storage, processing, and sharing.

[0031] In some embodiments, the server may be implemented as a node in a blockchain system.

[0032] The image processing method according to the present application will be described with reference to the above terminology and application scenes. Taking the method as applied to a server as an example, as shown in FIG. 2, the method includes the following steps 210 to 240.

[0033] In step 210, a sample image is acquired.

[0034] Here, the sample image includes an image obtained by collecting a sample to be analyzed within a preset wavelength band.

[0035] The wavelength band indicates a range of wavelengths. For example, the visible light wavelength band indicates a wavelength range of 380 nm to 750 nm, the near-infrared wavelength band indicates a wavelength range of 750 nm to 2500 nm, and the mid-infrared wavelength band indicates a wavelength range of 2500 nm to 25000 nm.

[0036] Alternatively, the illumination light source providing the wavelength may include a halogen lamp, an incandescent lamp, a light emitting diode (LED) light source, etc. Exemplarily, the preset wavelength band is intended to indicate a preset wavelength band, and the illumination light source providing the wavelength covers the preset wavelength band. For example, the preset wavelength band is a wavelength band of 400 nm to 1700 nm, and the selected illumination light source covers the wavelength band of 400 nm to 1700 nm. Alternatively, the preset wavelength band may be a wavelength band of 400 nm to 1700 nm, the illumination light source A provides a wavelength band of 400 nm to 1200 nm, and the illumination light source B provides a wavelength band of 1100 nm to 1800 nm, and the illumination light source A and the illumination light source B may be illumination light sources providing the preset wavelength band, etc.

[0037] The term "analyte sample" refers to a sample to be analyzed. Alternatively, the sample to be analyzed may be a pathological sample excised by surgery, and the location and property information of the pathological sample may be obtained by analyzing the pathological sample excised by surgery. Alternatively, the sample to be analyzed may be a chemical mixture, and the components and ratios of the mixture may be obtained by analyzing the chemical mixture. Alternatively, the sample to be analyzed may be a gemstone, and the structure of the gemstone may be obtained by analyzing the gemstone.

[0038] In one alternative embodiment, a push broom collection operation is performed on the sample to be analyzed to obtain an image of the sample.

[0039] Here, the push broom collection is a collection method of scanning and imaging each point along a scanning line, and the push broom collection operation is performed by a collection device. Exemplarily, FIG. 3 shows a push broom type short wavelength infrared hyperspectral imaging system, which includes a sample holder 310, a short wavelength infrared hyperspectral camera 320, a line light source 330, and an analyzed sample 340. Among them, the short wavelength infrared hyperspectral camera 320 is a combination of imaging technology and spectrum detection technology, and is used to collect a sample image having spectrum information. The sample image is a three-dimensional hyperspectral image, where the x-axis and y-axis indicate the coordinate values ​​of the two-dimensional information, and the z-axis indicates the wavelength information. Compared with general imaging technology, the hyperspectral image increases the spectral information of the image, has a higher spectral resolution, and can reflect the sample situation of the analyzed sample from a wider wavelength band range and more levels of spectral dimensions, and the sample image can simultaneously reflect the spatial information and spectral information of the analyzed sample. When performing analysis on a hyperspectral sample image, it is convenient to select a wavelength with higher resolution and better analysis effect from a wider wavelength band range, and perform subsequent analysis on the sample image, i.e., the sample image is an image having spectral information.

[0040] For example, a reflective hyperspectral camera with an effective photosensitive range of 900 nm to 1700 nm is used to capture a hyperspectral image with a spectral resolution of about 5 nm and an image resolution of 300,000 pixels.

[0041] Optionally, after determining a preset wavelength band, the sample to be analyzed is irradiated with an illumination light source, and the sample to be analyzed is photographed multiple times under illumination conditions of different wavelengths within the preset wavelength band, thereby obtaining sample images corresponding to the multiple samples to be analyzed. Exemplarily, the preset wavelength band is 900 nm to 1700 nm (a wavelength band selected from the near-infrared wavelength band), the sample to be analyzed is a pathological specimen excised during surgery, and the pathological specimen excised during surgery is collected under different wavelengths by using a halogen lamp as an illumination light source and a hyperspectral camera as an image collection device. For example, within the preset wavelength band, one image is collected for each wavelength using the hyperspectral camera, thereby obtaining multiple sample images corresponding to different wavelengths.

[0042] As shown in Fig. 3, a line light source 330 illuminates an analysis target sample 340, and a short-wavelength infrared hyperspectral camera 320 captures the analysis target sample 340 illuminated with different wavelengths, thereby achieving a process of acquiring multiple sample images at multiple different wavelengths. Fig. 4 shows multiple sample images 410 collected using the short-wavelength infrared hyperspectral camera 320, where the multiple sample images are arranged in order from top to bottom according to the wavelength range of 900 nm to 1700 nm, and the multiple sample images 410 are three-dimensional hyperspectral images having spectral information.

[0043] 5, M point 420 and N point 430 in the sample image 410 are arbitrarily selected, and a spectral characteristic curve 510 corresponding to M point 420 and a spectral characteristic curve 520 corresponding to N point 430 are obtained based on the spectral information corresponding to the sample image 410. Here, the spectral characteristic graph is a diagram showing the relationship between light reflectance and wavelength, where the abscissa is wavelength and the ordinate is reflectance, and the reflectance indicates the ratio of the light flux reflected by the sample to be analyzed to the light flux incident on the sample to be analyzed. Optionally, the spectral characteristic curve in FIG. 5 is a curve corrected for reflectance.

[0044] In one optional embodiment, at least one wavelength is determined within a preset wavelength band range using a tunable filter, and a sample image corresponding to the at least one wavelength is obtained by performing a push broom collection operation on the sample to be analyzed using a collection device.

[0045] Optionally, a liquid crystal tunable filter (LCTF) is added before the hyperspectral camera, and the liquid crystal tunable filter can select a wavelength from an illumination light source that covers a preset wavelength band, thereby quickly and vibration-free selecting a wavelength within the visible light wavelength band or the near infrared wavelength band. For example, the wavelength band covered by the illumination light source is 900 nm to 1700 nm, and when the light emitted from the illumination light source is passed through the liquid crystal tunable filter, light with a wavelength of 1130 nm is obtained, that is, the liquid crystal tunable filter filters out light with wavelengths other than the 1130 nm wavelength within the preset wavelength band.

[0046] Alternatively, the collection device is a hyperspectral camera, and the hyperspectral camera is a camera with a built-in lattice push broom structure, and obtains a sample image by performing a push broom collection operation on the sample to be analyzed according to the arrangement of the lattice. Alternatively, the hyperspectral camera is a hyperspectral shooting method with an external push broom structure, and moves the sample holder 310 to perform push broom shooting, as shown in Figure 3. Note that the above is merely an example, and the embodiments of the present application are not limited thereto.

[0047] In some embodiments, the lens in the hyperspectral camera may be a zoom lens that adjusts the field of view by optical magnification. Optionally, the sample image may be acquired by matching the field of view with a physical lift optical holder, or the sample image may be acquired by a combination of a variable magnification lens and a lift holder, etc.

[0048] In step 220, a pseudocolor image is obtained by acquiring a first image of the sample image that corresponds to at least one pre-defined wavelength within the pre-defined wavelength band.

[0049] For example, the sample images are a plurality of images obtained by collecting a sample to be analyzed, and the wavelengths corresponding to the sample images are within a preset wavelength band. At least one preset wavelength is selected from the plurality of wavelengths within the preset wavelength band, and the sample image corresponding to the preset wavelength is taken as a first image, and a pseudocolor image is finally obtained.

[0050] Optionally, for one preset wavelength, there is at least one sample image corresponding to the preset wavelength. Exemplarily, when there are multiple sample images corresponding to the selected preset wavelength, one sample image may be randomly selected from the multiple sample images as the first image corresponding to the preset wavelength, or the first image corresponding to the preset wavelength may be determined by performing a joint analysis on the multiple sample images.

[0051] Or, for one preset wavelength, there is one sample image corresponding to the preset wavelength, and the sample image is the first image. Optionally, taking one preset wavelength as an example corresponding to one first image, there are differences in the processing manner for the first image according to the number of selected preset wavelengths. Exemplarily, the case of selecting one preset wavelength and the case of selecting multiple preset wavelengths are analyzed respectively.

[0052] (1) Select one pre-defined wavelength In one alternative embodiment, a pseudocolor image is obtained by performing a coloring process on the first image corresponding to a preset wavelength within a preset wavelength band.

[0053] For example, the preset wavelength band is a wavelength band of 900 nm to 1700 nm selected from the near-infrared wavelength band, and since the preset wavelength band is an invisible wavelength band, the image corresponding to the preset wavelength band is a grayscale image. One preset wavelength is selected from the preset wavelength band, and the first image corresponding to the preset wavelength is a grayscale image, and a pseudocolor image is obtained by performing a coloring process on the grayscale image.

[0054] Pseudo-color image processing refers to the technical process of converting a black-and-white grayscale image into a color image to improve the visibility of the image content. For example, the pseudo-color image processing is performed using methods such as grayscale division and grayscale conversion.

[0055] Alternatively, a greyscale image may be a single channel image in which each pixel point has only one value representing its colour, with the pixel values ​​ranging from 0 to 255, with 0 representing black, 255 representing white and intermediate values ​​representing different levels of grey, or a three channel image in which the pixel values ​​of all three channels are the same.

[0056] Alternatively, the image relative to the single channel image includes a three-channel image in which each pixel point is represented by three values. For example, an RGB image is a three-channel image in which various colors are obtained by varying and superimposing three color channels, red (R), green (G), and blue (B), in which each pixel point is represented by three values.

[0057] Exemplarily, the sample to be analyzed is a pathological sample excised by surgery, and FIG. 6 is a diagram showing different image processing for the sample to be analyzed. Here, FIG. 610 is for showing the sample to be analyzed (taken by a normal camera for ease of display), FIG. 620 is for showing a hyperspectral image with a wavelength of 1300 nm, and FIG. 631 to FIG. 634 are for showing tissue expression observed by a hematoxylin-eosin staining (HE) method, among which FIG. 631 is for showing cancer tissue (shown at point A in FIG. 610 or FIG. 620), FIG. 632 is for showing fat tissue (shown at point B in FIG. 610 or FIG. 620), FIG. 633 is for showing normal mucosal tissue (shown at point C in FIG. 610 or FIG. 620), and FIG. 634 is for showing muscle tissue (shown at point D in FIG. 610 or FIG. 620).

[0058] For example, a wavelength of 1300 nm is selected as the preset wavelength, the hyperspectral image corresponding to FIG. 620 is taken as the first image corresponding to the preset wavelength, and the above-mentioned coloring process is performed on the first image to obtain a pseudocolor image.

[0059] (2) When selecting multiple preset wavelengths Exemplarily, determine at least two first images corresponding to the at least two preset wavelengths respectively based on the at least two preset wavelengths, where the i-th preset wavelength corresponds to the i-th first image, where i is a positive integer.

[0060] In one optional embodiment, a synthesis process is performed on at least two first images corresponding to at least two predetermined wavelengths within a predetermined wavelength band, and a coloring process is performed on the synthesized image to obtain a pseudocolor image.

[0061] Exemplarily, at least two preset wavelengths are selected from the preset wavelength band, and each of the preset wavelengths corresponds to one first image. Optionally, a candidate image is obtained by performing a synthesis process on the at least two first images.

[0062] Exemplarily, the method of performing the synthesis process on the multiple first images includes at least one of the following methods.

[0063] (1) Pixel value processing In one optional embodiment, an averaging process is performed on first pixel values ​​of pixel points corresponding to at least two first images to obtain second pixel values ​​of the corresponding pixel points, and a candidate image is determined based on the second pixel values ​​corresponding to each pixel point.

[0064] Exemplarily, after obtaining at least two first images corresponding to at least two preset wavelengths, first pixel values ​​of pixel points corresponding to the at least two first images are added and averaged to obtain a second pixel value, which is an average value obtained by comprehensively analyzing the first pixel values ​​of pixel points corresponding to different first images, as the pixel value of the corresponding pixel point. Optionally, after determining the second pixel value corresponding to each pixel point, a candidate image is obtained in which the pixel value of each pixel point is the corresponding second pixel value based on the position information of the pixel point.

[0065] Optionally, when synthesizing first images each corresponding to a different preset wavelength, a first pixel value of a pixel point corresponding to the different first images is used, and a second pixel value obtained by averaging the multiple first pixel values ​​is set as the pixel value of the pixel point corresponding to the second image, thereby performing comprehensive balancing processing on the multiple different first images according to the pixel points corresponding to the images, and better representing the balancing level of the different first images.

[0066] (2) Software processing In one alternative embodiment, after determining the first images corresponding to the preset wavelengths, at least two of the first images are composited by software to obtain a candidate image.

[0067] For example, at least two first images are input into Photoshop, and operations such as alignment, joining and color matching, seam removal, and deriving are performed to obtain a candidate image in which the at least two first images are synthesized.

[0068] The above is merely illustrative and is not limited to the embodiments of the present application.Optionally, when the selected preset wavelength is realized as a plurality of preset wavelengths in a preset wavelength band, a comprehensive analysis is performed on the first images corresponding to different preset wavelengths, i.e., the first images corresponding to different preset wavelengths are synthesized by the above synthesis method, and a more comprehensive analysis process can be performed on the analysis target sample from a plurality of wavelength dimensions; and by synthesizing the candidate image using the first images corresponding to a plurality of wavelengths, the candidate image contains sample information common to a plurality of wavelengths, and the difference in image information due to the wavelength difference is smoothed, thereby avoiding the constraints of analysis.

[0069] For example, based on the brightness values ​​of pixel points in the candidate image, brightness grading is performed on the pixel points in the candidate image to determine at least two brightness levels, and a pseudocolor image is obtained by coloring the pixel points for each of the at least two brightness levels.

[0070] Optionally, the first image is a grayscale image, and the candidate image synthesized based on the first image is a grayscale image, and the second pixel value corresponding to each pixel point in the grayscale image is for indicating the brightness of the candidate image. Exemplarily, the second pixel value is located between 0 and 255, where 0 indicates black (lowest brightness) and 255 indicates white (highest brightness), that is, the smaller the numerical value of the second pixel value, the lower the brightness, and the larger the numerical value of the second pixel value, the higher the brightness.

[0071] For example, FIG. 7 is a diagram showing a process of obtaining a pseudo-color image by performing a synthesis process and a coloring process on a first image corresponding to three preset wavelengths (a wavelength of 1100 nm, a wavelength of 1300 nm, and a wavelength of 1450 nm).

[0072] Among them, Fig. 710 shows a hyperspectral image with a wavelength of 1100 nm, Fig. 720 shows a hyperspectral image with a wavelength of 1300 nm, and Fig. 730 shows a hyperspectral image with a wavelength of 1450 nm. Optionally, the hyperspectral images corresponding to the above three preset wavelengths are subjected to synthesis and coloring processes to obtain a pseudocolor image as shown in Fig. 740.

[0073] For example, after obtaining a first image corresponding to one preset wavelength or a candidate image in which first images corresponding to multiple preset wavelengths are synthesized, the pixel points in the first image or the candidate image are subjected to brightness grading based on the brightness values ​​of the pixel points in the first image or the candidate image, and at least two brightness levels are determined, and a pseudo-color image is obtained by coloring the pixel points corresponding to different brightness levels according to the brightness changes of the pixel points of the image through the above coloring process, so that the colored pseudo-color image is more compatible with the observation habits of images by human eyes, and it is convenient for experts to distinguish different image regions by different colors in the pseudo-color image.

[0074] In step 230, a segmentation result is obtained by performing segmentation on the sample image based on the differences in sample element types in the sample image.

[0075] Here, the sample element types include the target element types to be identified.

[0076] Optionally, the sample element type is for indicating the difference of sample properties corresponding to different sample regions in the sample image. Exemplarily, if the sample image is an image of a pathological sample, the sample element type includes tumor tissue in the pathological sample, fat tissue in the pathological sample, mucosal tissue in the pathological sample, muscle tissue in the pathological sample, etc. If the sample image is an image of a chemical mixture (including compound A, compound B, and impurities), the sample element type includes compound A, compound B, and impurities.

[0077] Exemplarily, when the sample image is a pathology image obtained by photographing a pathology sample, the element type to be identified is a predetermined tumor tissue to be identified (one of the sample element types corresponding to the pathology image), or a predetermined adipose tissue to be identified, etc. Alternatively, when the sample image is a chemical image obtained by photographing a chemical mixture, the element type to be identified is a predetermined B compound to be identified (one of the sample element types corresponding to the chemical image), etc.

[0078] Exemplarily, the sample image is an image having spectral information, and the spectral information differs according to the differences in the properties of different substances, and the sample image is segmented based on the spectral information corresponding to the sample image to obtain a segmentation result.

[0079] For example, when the spectral information in the sample image is displayed differently, for example, when the sample image is a grayscale image, the color of the area corresponding to the A sample element is made the darkest, and the color of the area corresponding to the sample element to be analyzed is made the lightest, thereby obtaining a region segmentation result corresponding to the sample image.

[0080] In one optional embodiment, different regions can be filled with different colors to make them easier to distinguish, resulting in colored segmentation results, or dark contours can be used to segment different regions, resulting in clearly segmented segmentation results, etc.

[0081] It should be noted that the above is merely an example, and the embodiments of the present application are not limited thereto.

[0082] In step 240, image regions that contain the identified element type in the sample image are determined based on the pseudocolor image and the segmentation results.

[0083] Exemplarily, the pseudo-color image is an image obtained by processing a first image corresponding to a preset wavelength, and the segmentation result is a result of performing segmentation based on the sample element type in the sample image. Optionally, the pseudo-color image is segmented with different colors in the pseudo-color image, for example, when the sample image is an image of a pathological sample, the tumor tissue is orange, the fat tissue is bright yellow, the mucous membrane tissue is light orange, and the muscle tissue is dark orange.

[0084] In one alternative embodiment, the overlap area between the false color image and the segmentation result is determined.

[0085] In the sample image, the overlapping regions are taken as image regions that contain the element type to be identified.

[0086] For example, when the sample to be analyzed is a pathological sample, the pathological sample is collected to obtain a sample image having spectral information, and tumor tissue in the sample image is observed, the above-mentioned processing steps are performed on the sample image to obtain a pseudo-color image corresponding to the selected preset wavelength and a region segmentation result for the sample image. Based on the region of tumor tissue determined in the pseudo-color image and the identification result of the element type to be identified (tumor tissue) included in the region segmentation result, the overlapping region is regarded as an image region including the element type to be identified (tumor tissue), and an identification process for the tumor tissue region is realized.

[0087] For example, after obtaining a pseudo-color image of the first image corresponding to a predetermined wavelength with good spectral effect and a segmentation result obtained by performing segmentation on the sample image, the pseudo-color image obtained after the coloring process and the spectral analysis result are integrated to perform a more comprehensive analysis process on the sample image, so that the image area including the element type to be identified not only includes image information represented by the pseudo-color image, but also includes spectral information represented by the segmentation result, thereby significantly improving the accuracy of determining the image area.

[0088] The above are merely illustrative and are not limiting of the embodiments of the present application.

[0089] As described above, a pseudo-color image is obtained by acquiring a first image corresponding to a preset wavelength from a sample image, a region division is performed on the sample image to obtain a region division result, and the image region is determined by combining the pseudo-color image and the region division result. According to the above method, it is possible to avoid judging the size, region, etc. of tumor tissue only by naked eye observation and explanation by a doctor. Based on a predetermined preset wavelength band, an analysis target sample is collected to acquire a sample image, at least one preset wavelength having a good effect is selected from the preset wavelength band, and based on the at least one preset wavelength, a first image corresponding to the preset wavelength from the sample image is determined, and the first image is processed to obtain a pseudo-color image that can relatively accurately represent the advantages of the preset wavelength. Based on the difference in sample element type in the sample image, a region division is performed on the sample image to obtain a region division result. By combining the pseudocolor image and the region segmentation result to determine the image region containing the element type to be identified, the location information of the region to be identified (e.g., tumor tissue) can be determined, improving the accuracy of pathological collection while reducing the difficulty of pathological collection. By analyzing the sample to be analyzed using the spectral characteristics corresponding to the sample to be analyzed at different wavelengths, not only is the operation relatively simple, but the cost is also relatively low, making it easier to apply and widely disseminate.

[0090] In one alternative embodiment, the process of performing region segmentation on the sample image is determined by different spectral information corresponding to different sample element types. For example, as shown in Fig. 8, step 230 in the embodiment shown in Fig. 2 above may be realized as the following steps 810 to 850.

[0091] In step 810, a second image is acquired.

[0092] Here, the second image is a pre-labeled image having spectral information obtained by collecting the sample to be analyzed.

[0093] Optionally, the sample image and the second image are images obtained by collecting a sample to be analyzed, and when the sample to be analyzed is collected, a gold standard of the sample image is determined, i.e., the second image is determined, which is to indicate a reliable method of disease diagnosis that is accepted in the current clinical medical community.

[0094] In one selectable embodiment, the sample to be analyzed is a pathology sample excised by surgery. Illustratively, FIG. 9 is a diagram showing a flow of collecting a pathology sample, in which a doctor first excises a pathology sample from a patient by tumor resection surgery 910 (the pathology sample includes tumor tissue), and then cuts the pathology sample 920 to obtain a tissue block of an appropriate volume, and then fixes the tissue block 930 using a method such as formalin immersion. For example, within 30 minutes after the pathology sample is excised by surgery, the tissue block is placed in a sufficient amount of 3.7% neutral formalin solution to be fixed, and the fixation time is 12h to 48h. Then, a tissue slice (about 5mm on average) with a thickness of 5mm±1mm containing the tumor tissue and 1 to 2cm of normal tissue surrounding it is cut from the fixed tissue block. Alternatively, the tissue block fixed for the pathology sample may be used as the sample to be analyzed, or a tissue slice obtained by slicing the tissue block may be used as the sample to be analyzed.

[0095] In one alternative embodiment, after obtaining the sample to be analyzed, the tissue slice is roughly collected, generally dehydrated, embedded, and processed to prepare HE-stained slices to obtain a stained image 940, and the stained image 940 is scanned by a digital scanner 950 to obtain a number of WSI (Whole Slide Image) images 960. Here, the WSI is an image of the pathological sample collected by the digital scanner 950 (a kind of motorized microscope structure). For example, when the size of a single WSI image is small, the analyzed WSI image may be formed by splicing a number of pathological slices, for example, by splicing a number of WSI fragments using WSI splicing software to reconstruct a virtual large slice 970.

[0096] Optionally, the virtual large slice 970 is labeled using ASAP (Advanced Systems Analysis Program) as a gold standard, and the pathological sample is scanned to obtain a number of labeled WSI images. Here, the labeling may be performed in a manner of labeling a region, and the labeled region may include not only the region where one or more types of lesions are located, but also a special region having a hint function. For example, in a hollow organ, tumor tissue is labeled red, normal mucosa is labeled green, adipose tissue is labeled yellow, and muscle tissue is labeled blue, while in a solid organ, tumor tissue is labeled red, normal tissue is labeled green, and adipose tissue is labeled yellow, etc. Optionally, the above color labels are merely illustrative, and selected tissues may be labeled with different colors, for example, when labeling mammary gland tissue in a solid organ, tumor tissue in the mammary gland tissue may be labeled red, adipose tissue is labeled yellow, and fibrous connective tissue is labeled green, etc. Optionally, if there is no tissue of the corresponding color in the observed organ, no labeling is required, for example, when the above color labeling method is used to label parenchymal organs, if there is no fat tissue in the observed organ, yellow may not be labeled. Exemplarily, the labeled WSI image is used as the second image to realize the process of acquiring the second image.

[0097] In step 820, the candidate segmentation models are trained on the second image.

[0098] Here, the candidate segmentation model is a model that has a certain untrained region segmentation function.Exemplarily, the second image is used as gold standard to train the candidate segmentation model, and by training with a large amount of the second image, the candidate segmentation model is trained, and gradually becomes able to automatically identify special regions such as lesion regions, and gradually becomes equipped with region segmentation function.

[0099] In step 830, an image segmentation model is obtained in response to the training on the candidate segmentation model achieving a training effect.

[0100] Here, the image segmentation model is for performing region segmentation on a first image. Illustratively, in the process of training a candidate segmentation model, the training of the candidate segmentation model achieves a training goal, thereby obtaining an image segmentation model; optionally, the training effect of the candidate segmentation model is judged by a loss value, and the training goal includes at least one of the following cases:

[0101] 1. In response to the loss value reaching a convergence state, the candidate segmentation model obtained in the most recent iterative training is set as the image segmentation model.

[0102] For example, the loss value reaching the convergence state indicates that the numerical value of the loss value obtained by the loss function does not change or the change range is smaller than a preset threshold. For example, when the loss value corresponding to the nth second image is 0.1 and the loss value corresponding to the n+1th second image is also 0.1, the loss value is considered to have reached the convergence state, and the candidate segmentation model in which the loss value corresponding to the nth second image or the n+1th second image is adjusted is used as the image segmentation model, and a training process for the candidate segmentation model is realized.

[0103] 2. In response to the number of times the loss value is obtained reaching a threshold number of times, the candidate segmentation model obtained by the most recent iterative training is set as the image segmentation model.

[0104] For example, one loss value can be obtained by one acquisition, and the number of acquisitions of the loss value for training the image segmentation model is preset, and when one second image corresponds to one loss value, the number of acquisitions of the loss value is the number of second images, or when one second image corresponds to multiple loss values, the number of acquisitions of the loss value is the number of loss values. For example, when one loss value is obtained by one acquisition and the acquisition number threshold of the loss value is preset to 10 times, when the acquisition number threshold is reached, the candidate segmentation model in which the most recent loss value is adjusted is set as the image segmentation model, or the candidate segmentation model in which the minimum loss value in the 10 adjustment processes of the loss value is adjusted is set as the image segmentation model, thereby realizing the training process for the candidate segmentation model.

[0105] It should be noted that the above is merely an example, and the embodiments of the present application are not limited thereto.

[0106] In one alternative embodiment, the deep learning network associated with the candidate segmentation model may be a deep learning network such as Convolutional Networks for Biomedical Image Segmentation (U-net), Generative Adversarial Networks (GAN), Convolutional Neural Networks (CNN), etc., where the deep learning network is a policy for performing segmentation.

[0107] Alternatively, machine learning algorithms other than deep learning, such as Principal Component Analysis (PCA), may be used, or other non-machine learning algorithms, such as Support Vector Machine (SVM), maximum likelihood method, spectral angle, spectral information divergence, Mahalanobis distance, etc. may be used.

[0108] For example, by labeling the pathological area in the pathological sample, a second image can be obtained which can represent the gold standard for disease diagnosis recognized in the current clinical medical community, and the second image can relatively accurately indicate the pathological location corresponding to the pathological sample, which is convenient for more accurate training of the candidate segmentation model, gradually improving the robustness of the candidate segmentation model, and obtaining an image segmentation model which meets the training effect.

[0109] In step 840, the sample images are passed through a pre-trained image segmentation model to determine differential representations of element types.

[0110] In one alternative embodiment, the sample images undergo image pre-processing before being input into a pre-trained image segmentation model.

[0111] 10, after collecting a number of sample images 1010, preprocessing 1020 is performed on the sample images 1010. Here, the process of performing image preprocessing 1020 on the sample images includes performing operations including at least one of a geometric transformation operation, an image enhancement operation, etc. (e.g., image background correction, registration, noise removal, etc.) on the sample images to highlight important features in the sample images. Then, the preprocessed sample images 1010 are passed through a pre-trained image segmentation model 1030, and the image segmentation model 1030 segments regions in the sample images.

[0112] In one optional embodiment, the sample image is an image having spectral information, and a spectral analysis result is obtained by performing a spectral analysis on the sample image, and a differential representation of an element type corresponding to the sample image is determined based on the spectral analysis result.

[0113] According to the difference of the analyzed sample corresponding to the sample image, obtain different spectrum analysis results corresponding to different sample images. Illustratively, the spectrum analysis results are represented as a spectrum feature graph, the abscissa of the spectrum feature graph is wavelength, the ordinate is reflectance, and different spectrum curves are for showing the change situation of reflectance of different analyzed samples at different wavelengths, i.e., spectrum analysis results.

[0114] In one alternative embodiment, after analyzing 62 cases of hyperspectral images of different system tissues, it is preliminarily determined that the wavelength for distinguishing tumor tissue from normal tissue in different organs is 1296-1308 nm (within this wavelength range, the effect is good). For example, hollow organs (e.g., esophagus, stomach, colorectum), kidneys, mammary glands and lungs in which tumor tissue exists are analyzed as analysis target samples to obtain sample images corresponding to the hollow organs, kidneys, mammary glands and lungs, the sample images being three-dimensional hyperspectral images, and spectral feature graphs corresponding to the hollow organs, kidneys, mammary glands and lungs are obtained based on the data corresponding to the three-dimensional hyperspectral images.

[0115] FIG. 11 is a spectral characteristic graph corresponding to a hollow organ 1110, in which the wavelength curve corresponding to tumor tissue (cancerous tissue) is tumor wavelength curve 1120, the wavelength curve corresponding to adipose tissue is fat wavelength curve 1130, the wavelength curve corresponding to normal mucosa is mucosa wavelength curve 1140, and the wavelength curve corresponding to muscle tissue is muscle tissue wavelength curve 1150.

[0116] FIG. 12 is a spectral characteristic graph corresponding to kidney 1210, in which the wavelength curve corresponding to tumor tissue (cancerous tissue) is tumor wavelength curve 1220, the wavelength curve corresponding to adipose tissue is fat wavelength curve 1230, and the wavelength curve corresponding to normal mucosa is mucosa wavelength curve 1240.

[0117] FIG. 13 is a spectral characteristic graph corresponding to a mammary gland 1310, in which the wavelength curve corresponding to tumor tissue (cancerous tissue) is tumor wavelength curve 1320, the wavelength curve corresponding to adipose tissue is fat wavelength curve 1330, and the wavelength curve corresponding to normal mucosa is mucosa wavelength curve 1340.

[0118] FIG. 14 is a spectral characteristic graph corresponding to lung 1410, in which the wavelength curve corresponding to tumor tissue (cancerous tissue) is tumor wavelength curve 1420, and the wavelength curve corresponding to normal lung is normal wavelength curve 1430.

[0119] Here, the difference representation of the element type corresponding to the sample image is the difference between different tissues, such as the difference between tumor tissue and fat tissue, etc. The above is merely exemplary, and the embodiments of the present application are not limited thereto.

[0120] 11 to 14, when the wavelength is about 1300 nm, different tissues in the hollow organ tissue samples show good differentiation, and tumor tissues in solid organs (e.g., breast, kidney, and lung) also show relatively good differentiation from the surrounding normal tissues and adipose tissues.

[0121] Taking colon cancer as an example, when observed with the naked eye, tumor tissue in a 1300 nm hyperspectral image appears gray, normal muscle tissue appears gray-black, which is darker than tumor tissue, fat tissue appears gray-white, and normal mucosa appears dark gray, which is thinner than the muscle layer and slightly darker than tumor tissue, and the 1300 nm hyperspectral image shows good differentiation between fat, muscle layer, and tumor tissue.

[0122] In one alternative embodiment, the peaks and valleys of three waves at 1100 nm, 1300 nm, and 1450 nm in the hyperspectral image are extracted as characteristic wavebands, and a short-wave infrared color composite image is synthesized to provide a pseudo-color image that is more in line with the observation habits of human eyes for doctors to distinguish different tissues. In the short-wave infrared color composite image, the cancer tissue appears orange, the muscle tissue appears darker orange than the tumor tissue, the normal mucosa appears lighter orange than the cancer tissue, and the fat tissue appears bright yellow.

[0123] For example, by performing a spectral analysis on the sample image based on the spectral information contained in the sample image, it is advantageous to use the spectral analysis result corresponding to the sample image to more intuitively determine the reflectance change situation of the analyzed sample at different wavelengths, and further determine the difference situation between different tissues, and perform regional analysis on the sample image based on the difference.

[0124] In step 850, a segmentation result corresponding to the sample image is determined by performing segmentation on the sample image based on the difference representation of the element types.

[0125] Illustratively, after obtaining the spectrum analysis result, a corresponding region information hint is provided to the sample image through an image segmentation model, where the region information hint includes at least one of the following ways:

[0126] (1) Contour Hint Exemplarily, different regions in the sample image are segmented using the contour line to obtain different segmented regions, where the contour line may be a dark curve, a colored curve, etc.

[0127] (2) Heatmap Hints Exemplarily, the area in which the tumor tissue is located is displayed in a special highlighting manner.

[0128] (3) Monochrome filling hint 15, different regions are differentiated by filling them with different colors, e.g., tumor tissue regions are filled with red 1510 and adipose tissue regions are filled with green 1520. Optionally, regions that cannot be accurately segmented are filled with white or not filled at all.

[0129] In one possible embodiment, as shown in Figure 16, deep learning is performed based on a short-wave infrared hyperspectral image 1610 (sample image) and a labeled WSI 1620, and a prediction result 1630 is finally obtained by predicting the short-wave infrared hyperspectral image 1610, and for example, the prediction result 1630 is hinted in a monochromatic filling manner. The above is merely exemplary, and the embodiment of the present application is not limited thereto.

[0130] For example, after determining the differential representation of element types by an image segmentation model, the change in reflectance of the sample to be analyzed at different wavelengths indicated by the differential representation of element types is fully utilized to perform region segmentation on the sample image, and the region segmentation result corresponding to the sample image within a predetermined wavelength band is determined, thereby subdividing the analysis dimension of the sample image as regions, which is advantageous to improving the accuracy of analysis of the sample image.

[0131] In summary, a sample to be analyzed is collected based on a predetermined preset wavelength band to obtain a sample image, at least one preset wavelength having a good effect is selected from the preset wavelength band, a first image corresponding to the preset wavelength is determined from the sample image based on the at least one preset wavelength, and the first image is processed to obtain a pseudo-color image that can relatively accurately represent the advantages of the preset wavelength. A region segmentation result is obtained by performing region segmentation on the sample image based on the difference in sample element type in the sample image. The pseudo-color image and the region segmentation result are combined to determine an image region including the element type to be identified, thereby determining the position information of the region to be identified (e.g., tumor tissue). According to the above method, it is possible to avoid judging the size, region, etc. of tumor tissue only by the doctor's naked eye observation and explanation, and the difficulty of pathological collection can be reduced, and not only the operation is relatively simple, but also the cost is relatively low.

[0132] In the embodiment of the present application, the training process and application process of the region segmentation model are described. When training the region segmentation model, the WSI image is taken as the second image, and the untrained candidate segmentation model is trained on the second image until a convergence condition is reached to obtain the image segmentation model, and the image segmentation model performs region segmentation on the sample image, and determines the region segmentation result corresponding to the sample image according to the difference expression of element types in the sample image. According to the above method, the model learns the lesion region of the resected tissue, and the model automatically identifies special regions such as the lesion region, and the image region is specially marked in the image output from the model in the manner of region information hints, so that the model can better analyze the image and improve the accuracy of pathology collection.

[0133] In one alternative embodiment, the above image processing method is applied to the medical field to process pathological images. After obtaining pathological images with infrared hyperspectral information corresponding to different regions, narrowband synthetic pseudocolor images are combined with deep learning to predict the lesion area of ​​resected tissue, providing a new solution for determining tumor margins during surgery and assisting in pathological collection after surgery. Illustratively, the above image processing method is applied to at least two identification processes: (1) identification of tumor tissue in hollow organs, and (2) identification of tumor tissue in solid organs.

[0134] (1) Identification of tumor tissue in hollow organs Hollow organs are tubular organs that contain a lot of space inside, such as the stomach, intestines, bladder, and gallbladder. Solid organs, in contrast to hollow organs, include the heart, lungs, kidneys, liver, and spleen. The difference between them is that the former are hollow, while the latter are solid. For example, solid abdominal organs include the liver, spleen, kidneys, adrenal glands, and pancreas. Hollow abdominal organs include the gallbladder, stomach, duodenum, jejunum, ileum, appendix, and colon.

[0135] In one alternative embodiment, colon, rectum, stomach and esophagus cancer tissues in hollow organs are studied. In the four different tumor tissues, HSI1300nm shows relatively good discrimination and the imaged colors are similar.

[0136] Compared with X-ray images, hyperspectral imaging shows great advantages in identifying the muscle layer of hollow organs. In determining the borders of tumors, hyperspectral images are obviously clearer than ordinary color images. In particular, by selecting 1100nm, 1300nm and 1450nm HSI images to synthesize color images, the extent of tumor tissue can be clearly displayed, and different tissues have different intensity colors ranging from yellow to orange.

[0137] 17 is a pictorial representation of four tissue classes selected from hollow organs, of which sample 1 is colon cancer tissue, sample 2 is rectal cancer tissue, sample 3 is stomach cancer tissue, and sample 4 is esophageal cancer tissue.

[0138] Here, Figs. 1710 to 1740 shown in the first row are intended to show normal color images (corresponding to naked eye observation) taken with a normal camera.

[0139] Figures 1711 to 1741 in the second row are intended to show X-ray images obtained using an X-ray device, which can display the rough outline of the tumor area, but the effect is not clear and the muscle layer structure cannot be distinguished.

[0140] Figures 1712 to 1742 in the third row are intended to show hyperspectral images (HSI 1300 nm images) with a wavelength of 1300 nm collected by a hyperspectral camera. Hyperspectral images are grayscale images, and different tissues have different shades of color and can be distinguished by the naked eye.

[0141] Figures 1713 to 1743 shown in the fourth row are intended to show pseudo-color images synthesized using a hyperspectral image with a wavelength of 1100 nm, a hyperspectral image with a wavelength of 1300 nm, and a hyperspectral image with a wavelength of 1450 nm.

[0142] The fifth row of figures 1714 to 1744 show images segmented by artificial intelligence, for example, output images obtained using the above-mentioned region segmentation model, and the images can provide more detailed collection information. For example, color A represents tumor tissue, color B represents muscle tissue, color C represents normal mucosa tissue, and color D represents adipose tissue, and selectively, the darker the color, the higher the reliability.

[0143] Figures 1715 to 1745 shown in the sixth row show WSI images (gold standard) and are intended to display the true extent of tumor tissue.

[0144] (2) Identification of tumor tissue in solid organs In one alternative embodiment, the kidney, lung and breast in the parenchymal organs with lesions (e.g., with tumor tissue) are studied, and the tumor tissue is not difficult to distinguish with the naked eye in a relatively large sample. For example, Figure 18 shows different image representations of renal cancer.

[0145] Figure 1810 shows a normal color image (macroscopic observation) of kidney tissue, in which tumor tissue appears gray-white, fat tissue appears yellow, and normal kidney tissue appears light brown. Figure 1811 shows the tumor boundary in a magnified normal color image. Here, the boundary between the tumor tissue and normal kidney tissue displayed in the magnified image is difficult to distinguish.

[0146] Fig. 1820 is to show an X-ray image taken using an X-ray device, where the X-ray image can show the outline of the tumor, but the border is unclear and normal kidney tissue cannot be well distinguished from the tumor tissue.

[0147] Fig. 1830 shows a hyperspectral image with a wavelength of 1300 nm, and Fig. 1831 shows a tumor boundary in a magnified view of the hyperspectral image with a wavelength of 1300 nm. In the 1300 nm hyperspectral image, the tumor tissue appears grayish white, the normal kidney tissue appears gray, and the fat tissue appears light grayish white, and in the magnified view, the boundary between the tumor tissue and the surrounding normal tissue is clear.

[0148] FIG. 1840 is to show a pseudocolor image and FIG. 1832 is to show the tumor border with the combined pseudocolor image enlarged.

[0149] Exemplarily, the pseudo-color image may be obtained by performing a coloring process on one hyperspectral image corresponding to one wavelength, or may be obtained by performing a synthesis process and a coloring process on multiple hyperspectral images corresponding to multiple wavelengths. Here, the wavelength may be selected randomly or may be determined in advance. Exemplarily, at least one wavelength that is effective when judging from a comprehensive set of experimental results is selected in advance, and a pseudo-color image is obtained based on the hyperspectral image corresponding to the at least one wavelength.

[0150] For example, a wavelength of 1300 nm is selected as an effective wavelength, and a hyperspectral image corresponding to the wavelength of 1300 nm is colored to obtain a pseudo-color image, or a wavelength of 1250 nm is randomly selected, and a hyperspectral image corresponding to the wavelength of 1250 nm is colored to obtain a pseudo-color image, or three wavelengths of 1300 nm, 1100 nm, and 1450 nm are selected as effective wavelengths, and a synthesis process and a coloring process are performed on the hyperspectral images corresponding to the three wavelengths, respectively, to obtain a pseudo-color image.

[0151] In the near-infrared composite color image, renal cancer tissue appears orange, fatty tissue appears bright yellow, and normal renal tissue areas appear orange with a darker tint. As shown in the enlarged image, the boundary between the tumor tissue and the surrounding tissue is clear and easy to distinguish.

[0152] FIG. 1850 is for showing an artificial intelligence segmentation image for performing region segmentation on a sample image. Illustratively, different regions are distinguished in different formats, such as tumor tissue being red, normal kidney tissue being green, and adipose tissue being yellow, with the darker the color, the higher the reliability. In the artificial intelligence segmentation image, red is tumor tissue, green is normal kidney tissue, and yellow is adipose tissue, the darker the color, the higher the reliability, and the tumor contour is more consistent with the tumor boundary of the WSI.

[0153] Diagram 1860 is intended to outline the WSI tumor area.

[0154] In one selectable embodiment, the breast in the solid organ is taken as an example. It is not easy to judge the boundary of the tumor with the naked eye, for example, the boundary of the tumor tissue cannot be accurately identified by the photograph taken by the ordinary camera. As shown in FIG. 19, FIG. 1910 is for showing the ordinary color image taken by the ordinary camera, the part surrounded by the circle is the boundary part of the tumor tissue, and the boundary between the tumor tissue and the surrounding tissue in this part cannot be clearly distinguished in the ordinary color image.

[0155] X-ray images have a good effect on the judgment of tumor tissue, and have traditionally been the main supporting tool in pathology sampling, helping pathologists to find the tumor bed area. Figure 1920 is to show the X-ray images collected by the X-ray device, and in the case shown, the tumor outline displayed in the X-ray image has a burr-like edge, and the area is obviously larger than the tumor outline displayed in the WSI.

[0156] Figure 1930 shows a hyperspectral image at a wavelength of 1300 nm, in which the tumor tissue appears dark gray (corresponding to the irregular shape) and the area circled is the surrounding normal breast tissue, which appears light gray.

[0157] FIG. 1940 is intended to illustrate a pseudocolor image determined based on a hyperspectral image corresponding to at least one selected wavelength. Illustratively, in a pseudocolor image synthesized with short wavelength infrared, areas of tumor tissue appear dark orange compared to the surrounding breast tissue, and adipose tissue appears bright yellow.

[0158] Figure 1950 is to show an artificial intelligence segmentation image (the processing result obtained by processing a sample image with a deep learning model), which provides a relatively accurate information reference of the tumor tissue range, and Figure 1960 is to show a WSI image (WSI is the gold standard).

[0159] Optionally, in the case of breast, the X-ray image obtained by the X-ray device can display dot-like calcification. Exemplarily, as shown in FIG. 20, FIG. 2010 is for showing a normal color image taken by a normal camera, in which the tumor area is grayish white, and the outline of the tumor tissue can be identified. FIG. 2020 is for showing an X-ray image collected by an X-ray device, in which the edge of the tumor tissue can be roughly displayed, the edge appears burr-like, and dot-like calcification (position indicated by an arrow in FIG. 2020) can be seen therein. FIG. 2030 is for showing a hyperspectral image with a wavelength of 1300 nm, in which the tumor tissue is dark gray, the normal breast tissue is lighter in tone than the tumor tissue area, and the fat tissue is grayish white. FIG. 2040 is for showing a short-wave infrared color image obtained by processing the hyperspectral image corresponding to at least one selected wavelength, in which the short-wave infrared color image can display a clearer tumor outline. FIG. 2050 is for showing the image result display by artificial segmentation, and FIG. 2060 is for showing the WSI as the gold standard. Illustratively, the short-wave infrared color image and the image result display by artificial segmentation are combined to determine an image region including tumor tissue, and the contour displayed in the image region has the highest degree of agreement with the gold standard (WSI).

[0160] In summary, a sample to be analyzed is collected based on a predetermined preset wavelength band to obtain a sample image, at least one preset wavelength having a good effect is selected from the preset wavelength band, a first image corresponding to the preset wavelength is determined from the sample image based on the at least one preset wavelength, and the first image is processed to obtain a pseudo-color image that can relatively accurately represent the advantages of the preset wavelength. A region segmentation result is obtained by performing region segmentation on the sample image based on the difference in sample element type in the sample image. The pseudo-color image and the region segmentation result are combined to determine an image region including the element type to be identified, thereby determining the position information of the region to be identified (e.g., tumor tissue). According to the above method, it is possible to avoid judging the size, region, etc. of tumor tissue only by the doctor's naked eye observation and explanation, and the difficulty of pathological collection can be reduced, and not only the operation is relatively simple, but also the cost is relatively low.

[0161] In the embodiment of the present application, the above image processing method is applied to the medical field to analyze hollow organs and solid organs, proving the benefits of the above image processing method. Meanwhile, the above image processing method is more reliable than the naked eye observation and hand touch method by doctors, and the image consistency is more ensured. Meanwhile, the hyperspectral imaging system has the characteristics of no damage, no contact, and no ionizing radiation, and its hardware cost is lower than that of an X-ray device.

[0162] FIG. 21 is a block diagram showing the structure of an image processing device according to one exemplary embodiment of the present application. As shown in FIG. 21, the device includes: A sample acquisition module 2110 for acquiring a sample image, the sample image including an image obtained by collecting a sample to be analyzed within a preset wavelength band; an image acquisition module 2120 for acquiring a first image of the sample image corresponding to at least one preset wavelength within the preset wavelength band, thereby obtaining a pseudo-color image; A segmentation module 2130 for performing segmentation on the sample image based on differences in sample element types in the sample image to obtain a segmentation result, the sample element types including a classification target element type to be classified; and a region determination module 2140 for determining an image region including the identified element type from the sample image based on the pseudocolor image and the region segmentation result.

[0163] In one optional embodiment, the image acquisition module 2120 is further used to obtain the pseudo-color image by performing a coloring process on a first image corresponding to a preset wavelength within the preset wavelength band, or to perform a synthesis process on at least two first images corresponding to at least two preset wavelengths within the preset wavelength band, and perform a coloring process on the synthesized image to obtain the pseudo-color image.

[0164] In one optional embodiment, the image acquisition module 2120 is further used to determine at least two first images corresponding respectively to the at least two preset wavelengths based on the at least two preset wavelengths (the i-th preset wavelength corresponds to the i-th first image, where i is a positive integer), obtain a candidate image by performing a synthesis process on the at least two first images, and obtain the pseudo-color image by performing a coloring process on the candidate image.

[0165] In one optional embodiment, the image acquisition module 2120 is further used to obtain second pixel values ​​of corresponding pixel points by performing an averaging process on pixel values ​​of pixel points corresponding to the at least two first images, and determine the candidate image based on the second pixel values ​​corresponding to each pixel point.

[0166] In one optional embodiment, the image acquisition module 2120 is further used to determine at least two brightness levels by performing brightness grading on pixel points in the candidate image based on the brightness values ​​of the pixel points in the candidate image, and to obtain the pseudo-color image by performing coloring for the at least two brightness levels, respectively.

[0167] As shown in FIG. 22, in one alternative embodiment, the region segmentation module 2130: a determining unit 2131 for determining differential representations of the element types by passing the sample images through a pre-trained image segmentation model; a segmentation unit 2132 for performing segmentation on the sample image based on the element type difference representation to determine the segmentation result corresponding to the sample image.

[0168] In one alternative embodiment, the sample image is an image having spectral information; The determining unit 2131 is further used for performing a spectral analysis on the sample image to obtain a spectral analysis result, and determining a differential representation of the element type corresponding to the sample image based on the spectral analysis result.

[0169] In one optional embodiment, the apparatus is further used to obtain a second image, which is a pre-labeled image having spectral information obtained by collecting the sample to be analyzed, train a candidate segmentation model with the second image, and obtain an image segmentation model for performing region segmentation on the first image in response to the training on the candidate segmentation model achieving a training effect.

[0170] In one alternative embodiment, the sample acquisition module 2110 is further used to obtain an image of the sample by performing a push broom collection operation on the sample to be analyzed.

[0171] In one optional embodiment, the push broom collection operation is performed by a collection device, and the sample acquisition module 2110 is further used to determine at least one wavelength within the preset wavelength band range using a wavelength tunable filter, and obtain a sample image corresponding to the at least one wavelength by performing a push broom collection operation on the sample to be analyzed by the collection device.

[0172] In one optional embodiment, the region determination module 2140 is further used to determine an overlap region between the pseudocolor image and the region segmentation result, and to define the overlap region as the image region in the sample image that contains the element type to be identified.

[0173] In summary, a sample to be analyzed is collected based on a predetermined preset wavelength band to obtain a sample image, at least one preset wavelength having a good effect is selected from the preset wavelength band, a first image corresponding to the preset wavelength is determined from the sample image based on the at least one preset wavelength, and the first image is processed to obtain a pseudo-color image that can relatively accurately represent the advantages of the preset wavelength. A region segmentation result is obtained by performing region segmentation on the sample image based on the difference in sample element type in the sample image. The pseudo-color image and the region segmentation result are combined to determine an image region including the element type to be identified, thereby determining the position information of the region to be identified (e.g., tumor tissue). According to the above device, it is possible to avoid judging the size, region, etc. of tumor tissue only by the doctor's naked eye observation and explanation, and the difficulty of pathological collection can be reduced, and not only the operation is relatively simple, but also the cost is relatively low.

[0174] In addition, the image processing device according to the above embodiment has been described by taking the division of each of the above functional modules as an example, but in actual application, the above functions can be assigned to different functional modules to be completed as necessary, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the above functions. In addition, the image processing device according to the above embodiment has the same concept as the embodiment of the image processing method, and the details of the specific implementation process can be referred to the embodiment of the method, so the description will be omitted here.

[0175] 23 is a diagram showing the structure of a server according to one exemplary embodiment of the present application. The server 2300 includes a CPU (Central Processing Unit) 2301, a system memory 2304 including a RAM (Random Access Memory) 2302 and a ROM (Read Only Memory) 2303, and a system bus 2305 connecting the system memory 2304 and the CPU 1201. The server 2300 further includes a mass storage device 2306 that stores an operating system 2313, an application program 2314, and other program modules 2315.

[0176] The mass storage device 2306 is connected to the CPU 1201 via a mass storage controller (not shown) that is connected to the system bus 2305. The mass storage device 2306 and its associated computer readable media provide non-volatile storage for the server 2300.

[0177] Without loss of generality, computer-readable media may include computer storage media and communication media.

[0178] According to various embodiments of the present application, the server 2300 may be connected to a network 2312 by a network interface unit 2311 connected to the system bus 2305, or may be connected to other types of networks or remote computer systems (not shown) by using the network interface unit 2311.

[0179] The memory further includes one or more programs, the one or more programs being stored in the memory and configured to be executed by the CPU.

[0180] An embodiment of the present application further provides a computing device including a processor and a memory, the memory storing at least one instruction, at least a portion of a program, code set or instruction set that is loaded and executed by the processor to realize an image processing method according to each of the method embodiments described above.

[0181] An embodiment of the present application provides a computer-readable storage medium storing at least one instruction, at least a portion of a program, code set or instruction set that can be loaded and executed by a processor to realize an image processing method according to each of the above method embodiments.

[0182] An embodiment of the present application provides a computer program product or computer program comprising computer instructions stored on a computer readable storage medium, the computer instructions being read from the computer readable storage medium by a processor of a computing device and executed to cause the computing device to perform an image processing method according to any of the embodiments described above.

[0183] Optionally, the computer-readable recording medium may include a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drives), an optical disk, etc. Among them, the RAM may include a ReRAM (Resistance Random Access Memory) and a DRAM (Dynamic Random Access Memory). The sequence numbers of the above embodiments of the present application are merely for explanation purposes and do not represent the superiority or inferiority of the embodiments.

Claims

1. An image processing method executed by a server, comprising: acquiring a sample image, the sample image including an image obtained by collecting a sample to be analyzed within a predetermined wavelength range; obtaining a first image of the sample image corresponding to at least one preset wavelength within the preset wavelength band, thereby obtaining a pseudocolor image; obtaining a segmentation result by performing segmentation on the sample image based on differences in sample element types in the sample image, the sample element types including a classification target element type to be classified; determining an image region in the sample image that includes the element type to be identified based on the pseudo color image and the region segmentation result; 13. An image processing method comprising:

2. The step of obtaining a pseudocolor image by acquiring a first image of the sample images corresponding to at least one preset wavelength within the preset wavelength band includes: performing a coloring process on a first image corresponding to a preset wavelength within the preset wavelength band to obtain the pseudocolor image; or performing a synthesis process on at least two first images corresponding to at least two preset wavelengths within the preset wavelength band, and performing a coloring process on the synthesized image to obtain the pseudo-color image; 2. The image processing method according to claim 1.

3. The step of obtaining the pseudo-color image by performing a synthesis process on at least two first images corresponding to at least two preset wavelengths within the preset wavelength band and performing a coloring process on the synthesized image, comprises: determining at least two first images corresponding to the at least two preset wavelengths, respectively, based on the at least two preset wavelengths, where an i-th preset wavelength corresponds to an i-th first image, where i is a positive integer; performing a synthesis process on the at least two first images to obtain a candidate image; and performing a coloring process on the candidate image to obtain the pseudo-color image.

3. The image processing method according to claim 2.

4. The step of obtaining a candidate image by performing a synthesis process on the at least two first images includes: performing an averaging process on first pixel values ​​of corresponding pixel points of the at least two first images to obtain second pixel values ​​of the corresponding pixel points; determining the candidate images based on second pixel values ​​corresponding to each pixel point; 4. The image processing method according to claim 3.

5. The step of obtaining the pseudo-color image by performing a coloring process on the candidate image includes: determining at least two luminance levels by performing luminance grading on pixel points in the candidate image based on luminance values ​​of pixel points in the candidate image; and obtaining the pseudo-color image by applying coloring to each of the at least two luminance levels.

4. The image processing method according to claim 3.

6. The step of obtaining a segmentation result by performing segmentation on the sample image based on differences in sample element types in the sample image includes: determining differential representations of the element types by passing the sample images through a pre-trained image segmentation model; and determining the segmentation result corresponding to the sample image by performing segmentation on the sample image based on the element type difference representation.

6. The image processing method according to claim 1,

7. the sample image is an image having spectral information; determining differential representations of the element types by passing the sample images through a pre-trained image segmentation model; performing a spectral analysis on the sample image to obtain a spectral analysis result; and determining a differential representation of the element type corresponding to the sample image based on the spectral analysis results.

7. The image processing method according to claim 6.

8. acquiring a second image, the second image being a pre-labeled image having spectral information obtained by collecting the sample to be analyzed; training a candidate segmentation model on the second image; and obtaining an image segmentation model in response to the training on the candidate segmentation model achieving a training effect, the image segmentation model being for performing region segmentation on the first image.

8. The image processing method according to claim 7.

9. The step of acquiring a sample image comprises: performing a push broom collection operation on the sample to be analyzed to obtain the sample image; 6. The image processing method according to claim 1,

10. The push broom harvesting operation is performed by a harvesting device; The step of acquiring a sample image comprises: determining at least one preset wavelength within the preset wavelength band using a tunable filter; and performing a push broom collection operation on the sample to be analyzed by the collection device to obtain a sample image corresponding to the at least one preset wavelength.

10. The image processing method according to claim 9.

11. The step of determining an image region including the target element type in the sample image based on the pseudo color image and the region segmentation result includes: determining an overlap area between the pseudo color image and the segmentation result; and determining in the sample image that the overlapping region is the image region that contains the identified element type.

6. The image processing method according to claim 1,

12. a sample acquisition module for acquiring a sample image, the sample image including an image obtained by collecting a sample to be analyzed within a preset wavelength band; an image acquisition module for acquiring a first image of the sample image corresponding to at least one preset wavelength within the preset wavelength band, thereby obtaining a pseudo-color image; a segmentation module for performing segmentation on the sample image based on differences in sample element types in the sample image to obtain a segmentation result, the sample element types including a classification target element type to be classified; and a region determination module for determining an image region including the target element type in the sample image based on the pseudo color image and the region segmentation result.

13. An image processing device comprising:

13. A computing device including a processor and a memory, The memory stores at least a part of a program that is loaded and executed by the processor to realize the image processing method according to any one of claims 1 to 11. A computer apparatus comprising:

14. At least a part of a program that is loaded and executed by a processor to realize the image processing method according to any one of claims 1 to 11 is stored. A computer-readable storage medium comprising:

15. A computer program or instructions which, when executed by a processor, implements the image processing method according to any one of claims 1 to 11, 23. A computer program product comprising:

Images