Image processing method, apparatus, device, and program

Abstract

An image processing method, apparatus, device, readable storage medium, and program product relating to the field of medical data processing, the method comprising the steps of: acquiring a sample image (210), the sample image including an image obtained by collecting a sample to be analyzed within a preset wavelength band; acquiring a first image of the sample image corresponding to at least one preset wavelength within the preset wavelength band to obtain a pseudocolor image (220); performing region segmentation on the sample image based on differences in sample element types in the sample image to obtain a segmentation result (230); and determining an image region in the sample image including an element type to be identified based on the pseudocolor image and the region segmentation result (240).

Description

Technical Field

[0001] This application claims priority to a Chinese patent application with an application number of 202210086842.8 filed on January 25, 2022, and an invention title of "Image Processing Method, Apparatus, Device, Readable Storage Medium, and Program Product", and all of its contents are incorporated herein by reference.

[0002] Embodiments of the present application relate to the field of medical data processing, and particularly to an image processing method, apparatus, device, readable storage medium, and program product.

Background Art

[0003] After surgically removing a pathological specimen containing tumor tissue, usually, the collected pathological specimen is further studied, and the boundary of the tumor tissue in the pathological specimen is found and identified to more accurately analyze the tumor tissue and obtain more valuable medical analysis results.

[0004] In related technologies, after fixing the pathological specimen excised by surgery with formalin, usually, a pathologist determines the boundary of the tumor tissue by observing it with the naked eye, or scans the pathological specimen using an X-ray device, and a doctor interprets the X-ray image to determine the boundary of the tumor tissue and perform the collection operation of the tumor tissue.

[0005] However, when determining the boundary of the tumor tissue using the above method, for some lesions with unclear tumor beds, it is difficult to identify them with the naked eye. On the other hand, X-ray devices are also expensive, so it is difficult to widely spread.

Summary of the Invention

Problems to be Solved by the Invention

[0006] Embodiments of the present application provide an image processing method, apparatus, device, readable storage medium, and program product that can improve the accuracy of pathological sampling by analyzing a sample to be analyzed by utilizing the spectral characteristics of a first sample at different wavelengths.

Means for Solving the Problem

[0007] The present application takes the following technical solutions.

[0008] According to one aspect of the present invention, a step of acquiring a sample image, wherein the sample image includes an image obtained by collecting a sample to be analyzed within a preset wavelength band; a step of obtaining a pseudo-color image by acquiring a first image corresponding to at least one preset wavelength within the preset wavelength band in the sample image; a step of obtaining a region division result by performing region division on the sample image based on differences in sample element types in the sample image, wherein the sample element types include identification target element types to be identified; a step of determining an image region including the identification target element type in the sample image based on the pseudo-color image and the region division result, and providing an image processing method including the steps.

[0009] According to another aspect of the present invention, a sample acquisition module for acquiring a sample image, wherein the sample image includes an image obtained by collecting a sample to be analyzed within a preset wavelength band; an image acquisition module for obtaining a pseudo-color image by acquiring a first image corresponding to at least one preset wavelength within the preset wavelength band in the sample image; A region division module that performs region division on the sample image based on differences in sample element types in the sample image to obtain a region division result, where the sample element type includes the identification target element type to be identified. Provided is an image processing apparatus including: a region determination module that determines an image region including the identification target element type in the sample image based on the pseudo-color image and the region division result.

[0010] According to another aspect of the present application, there is provided a computer device including a processor and a memory, where the memory stores at least one instruction, at least a part of a program, a code set, or an instruction set that, when loaded and executed by the processor, implements the image processing method according to any one of the above embodiments of the present application.

[0011] According to another aspect of the present application, there is provided a computer-readable storage medium storing at least one instruction, at least a part of a program, a code set, or an instruction set that, when loaded and executed by a processor, implements the image processing method according to any one of the above embodiments of the present application.

[0012] According to another aspect of the present application, there is provided a computer program product or a computer program including computer instructions stored in a computer-readable storage medium. When a processor of a computer device reads and executes the computer instructions from the computer-readable storage medium, the computer device is caused to execute the image processing method according to any one of the above embodiments.

Advantages of the Invention

[0013] The technical solutions provided by the embodiments of the present application can achieve at least the following beneficial effects.

[0014] By determining the image region by combining the suspected color image and the region segmentation result, it is possible to avoid judging the size and region of the tumor tissue only by the naked-eye observation and explanation by the doctor, and reduce the inaccuracy of the judgment on the region of the patient's tumor tissue. Based on a preset wavelength band, an analysis target sample is collected to obtain a sample image, at least one preset wavelength with good effect is selected from the preset wavelength band, a first image corresponding to the preset wavelength is determined from the sample image, and by processing the first image, a suspected color image that can relatively accurately represent the advantages of the preset wavelength can be obtained. Also, based on the difference in the type of identification target element in the sample image, region segmentation is performed on the sample image to obtain a region segmentation result. By combining the suspected color image and the region segmentation result and determining the image region including the type of identification target element, the position information of the identification target region (for example, tumor tissue) is determined, the difficulty of pathological sampling is reduced while improving the accuracy of pathological sampling, and by using the spectral characteristics corresponding to the analysis target sample at different wavelengths for analysis, not only the operation becomes relatively simple, but also the cost becomes relatively low, and it becomes easier to be widely applied.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Embodiments for Carrying Out the Invention

[0016] First, terms related to the embodiments of the present application will be briefly described.

[0017] Artificial Intelligence (AI) refers to a theory, method, technology, and application system that uses digital computers or devices controlled by digital computers to simulate, extend, and expand human intelligence, perform environmental perception, knowledge acquisition, and obtain optimal results using knowledge. In other words, as an integrated technology of computer science, artificial intelligence aims to understand the essence of intelligence and manufacture new intelligent machines that can react in a manner similar to human intelligence. That is, artificial intelligence endows devices with functions of perception, inference, and decision-making by researching the design principles and implementation methods of various intelligent machines.

[0018] Artificial intelligence technology, as an integrated discipline, is related to a wide range of fields, including both hardware technology and software technology. The basic technologies of artificial intelligence generally include, for example, technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0019] Machine learning (ML) is an interdisciplinary field that intersects multiple disciplines. It is related to many disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. By specializing in studying how computers simulate or realize human learning behaviors, it acquires new knowledge and skills, and continuously improves its own performance by reorganizing the existing knowledge structure. As the core of artificial intelligence, machine learning is the fundamental route for endowing computers with intelligence and is applied across various fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, trust networks, reinforcement learning, transfer learning, inductive learning, and supervised learning.

[0020] In related technologies, usually, a pathologist determines the boundary of a tumor tissue by observing it with the naked eye, or scans a pathological specimen using an X-ray device, and a doctor interprets the X-ray image to determine the boundary of the tumor tissue and perform the sampling operation of the tumor tissue. However, when using the above methods to determine the boundary of a tumor tissue, for some lesions with an unclear tumor bed, it is difficult to identify them with the naked eye. On the other hand, since X-ray devices are also expensive, it is difficult to widely popularize them.

[0021] In the embodiments of the present application, an image processing method for improving the accuracy of pathological sampling is provided by analyzing an analysis target sample by utilizing the spectral characteristics of the analysis target sample at different wavelengths. The image processing method obtained by the training according to the present application includes at least one of the following scenarios when applied.

[0022] I. Application to the medical field In tumor resection surgery, in order to achieve the process of completely removing the tumor area, prevent the recurrence of the patient's condition, and avoid secondary surgery, it is necessary to accurately know the position of the tumor margin. Postoperative histopathological analysis is the gold standard for tumor diagnosis. In order to accurately obtain the lesion information of the patient, the process of selecting the pathological tissue block by the doctor is particularly important. Missing the tissue block containing the lesion limits the more accurate judgment by the pathologist, while the selection of excessive tissue blocks significantly increases the workload of slicing and reduces the medical efficiency. Exemplarily, when using the above image processing method, a tissue having a lesion (for example, a kidney organ, a breast, etc.) is used as an analysis target sample, the analysis target sample is collected within a preset wavelength band to obtain a sample image, a first image corresponding to the preset wavelength is selected from the sample image to obtain a pseudo-color image, and then, by performing region division on the sample image based on the sample element type of the sample image, a region division result is obtained. By comprehensively analyzing the region division result and the pseudo-color image, finally, the image region corresponding to the tumor tissue can be determined relatively accurately, and the identification process for the image region can be realized. According to the above method, it is possible to assist the pathologist in finding the lesion area more quickly, reduce the equipment cost using imaging devices such as X-rays, use a spectrometer with a wider application range and lower cost to obtain a sample image, and analyze the sample image with spectral information, thereby reducing the medical cost and improving the judgment accuracy of the tumor tissue.

[0023] II. Application in the Field of Food Inspection Food safety is related to life safety. Foods often contain different components, unhealthy components, or inaccurate component ratios, all of which may cause food safety accidents. Exemplarily, using the above image processing method, the food to be detected is used as an analysis target sample, the analysis target sample is collected within a preset wavelength band to obtain a sample image, a first image corresponding to a preset wavelength is selected from the sample image to obtain a pseudo-color image, and then, by performing region division on the sample image based on the sample element type of the sample image to obtain a region division result, and comprehensively analyzing the region division result and the pseudo-color image, finally, the regions corresponding to different components in the food to be detected can be determined relatively accurately, the image region corresponding to the unhealthy component can be determined, and the identification process for the image region can be realized. According to the above method, it can assist food supervision and management agencies to better supervise foods, and based on the determination of the pseudo-color image based on the preset wavelength and the region division result, the image region can be identified more accurately.

[0024] It should be noted that the above application scenarios are only exemplary, and the image processing method according to this embodiment can also be applied to other scenarios, and this is not limited in the embodiments of the present application.

[0025] In the specific embodiments of the present application, regarding related data such as user information, when applying the above embodiments of the present application to specific products or technologies, it is necessary to obtain permission or consent from the user, and the collection, use, and processing of related data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0026] Next, the implementation environment according to the embodiments of the present application will be described. Exemplarily, as shown in FIG. 1, the implementation environment is related to a terminal 110 and a server 120, and the terminal 110 and the server 120 are connected via a communication network 130.

[0027] In some embodiments, an application program with an image acquisition function is installed on the terminal 110. In some embodiments, the terminal 110 is used to send a sample image to the server 120. The server 120 determines, based on the spectral information corresponding to the sample image, an image region including the element type of the object to be identified in the sample image by the image processing model 121, labels the image region in a special manner, and can feedback it to the terminal 110 for display.

[0028] Here, the application method of the image processing model 121 is as follows. That is, a preset wavelength is selected from a preset wavelength band, a first image corresponding to the preset wavelength is determined from the sample image based on the preset wavelength, and the first image is processed to obtain a pseudo-color image. Also, by performing region division on the sample image based on the sample element type in the sample image, a region division result corresponding to the sample image is obtained, and the region division result and the pseudo-color image are combined to determine an image region for indicating the position information of the element type of the object to be identified in the sample image. For example, if the sample to be analyzed is a pathological sample, the image region determined by analyzing the sample to be analyzed is the region corresponding to the tumor tissue, and thus, the region information corresponding to the tumor tissue can be determined more accurately. The above process is an example of a non-unique situation in the application process of the image processing model 121.

[0029] Note that the above terminal includes mobile terminals such as mobile phones, tablet computers, portable laptop computers, smart voice dialogue devices, smart home appliances, in-vehicle terminals, etc., but may also be realized as a desktop computer or the like. The above server may be an independent physical server, or may be a server cluster or distributed system composed of a plurality of physical servers, or may be a cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN), and a cloud server that provides basic cloud computing services such as big data and artificial intelligence platforms.

[0030] Among them, cloud technology refers to a kind of trust technology that integrates a series of resources such as hardware, application programs, and networks within a wide area network or local area network to realize data calculation, storage, processing, and sharing.

[0031] In some embodiments, the above server may be realized as a node in a blockchain system.

[0032] Regarding the image processing method according to the present application, it will be described with reference to the above term explanations and application scenarios. Taking its application to a server as an example, as shown in FIG. 2, the method includes the following steps 210 to 240.

[0033] In step 210, a sample image is acquired.

[0034] Here, the sample image includes an image obtained by collecting an analysis target sample within a preset wavelength band.

[0035] The wavelength band is for indicating the range of wavelengths. For example, the visible light wavelength band is for indicating the wavelength range of 380 nm to 750 nm, the near-infrared wavelength band is for indicating the wavelength range of 750 nm to 2500 nm, and the mid-infrared wavelength band is for indicating the wavelength range of 2500 nm to 25000 nm.

[0036] Optionally, the illumination light source that provides wavelengths may include a halogen lamp, and may also include an incandescent bulb, a light emitting diode (LED) light source, etc. Exemplarily, the preset wavelength band is for indicating a preset wavelength band, and the illumination light source that provides wavelengths covers the preset wavelength band. For example, the preset wavelength band is the wavelength band of 400 nm to 1700 nm, and the selected illumination light source covers the wavelength band of 400 nm to 1700 nm. Alternatively, the preset wavelength band is the wavelength band of 400 nm to 1700 nm, illumination light source A provides the wavelength band of 400 nm to 1200 nm, illumination light source B provides the wavelength band of 1100 nm to 1800 nm, and illumination light source A and illumination light source B may be used as illumination light sources that provide the preset wavelength band, etc.

[0037] The sample to be analyzed is for indicating the sample to be analyzed. Optionally, the sample to be analyzed is a pathological sample excised by surgery. Analyzing the pathological sample excised by surgery can enable one to know the position information, property information, etc. of the pathological sample. Alternatively, the sample to be analyzed is a chemical mixture. Analyzing the chemical mixture can enable one to know the component information, proportion information, etc. in the mixture. Alternatively, the sample to be analyzed is a gemstone. Analyzing the gemstone can enable one to know the structural information, etc. in the gemstone.

[0038] In one selectable embodiment, a sample image is obtained by performing a push-bloom collection operation on the sample to be analyzed.

[0039] Here, push-broom acquisition is an acquisition method that scans and images point by point along a scanning line, and the push-broom acquisition operation is performed by an acquisition device. Exemplarily, FIG. 3 shows a push-broom type short-wavelength infrared hyperspectral imaging system, which includes a sample holder 310, a short-wavelength infrared hyperspectral camera 320, a line light source 330, and an analysis target sample 340. Among them, the short-wavelength infrared hyperspectral camera 320 is a combination of imaging technology and spectral detection technology, and is used to collect a sample image with spectral information. The sample image is a three-dimensional hyperspectral image. The x-axis and y-axis represent the coordinate values of two-dimensional information, and the z-axis represents wavelength information. Compared with general imaging technology, the hyperspectral image increases the spectral information of the image, has a higher spectral resolution, and can reflect the sample situation of the analysis target sample from a wider wavelength band range and more levels of spectral dimensions, enabling the sample image to simultaneously reflect the spatial information and spectral information of the analysis target sample. When analyzing the hyperspectral sample image, it is also convenient to select a wavelength with higher resolution and better analysis effect from a wider wavelength band range and perform subsequent analysis on the sample image. That is, the sample image is an image with spectral information.

[0040] Exemplarily, a reflective hyperspectral camera with an effective photosensitive range of 900 nm to 1700 nm is used to capture a hyperspectral image with a spectral resolution of approximately 5 nm and an image resolution of 300,000 pixels.

[0041] Optionally, after determining a preset wavelength band, irradiate the sample to be analyzed with an illumination light source, and capture the sample to be analyzed multiple times under irradiation conditions of different wavelengths within the preset wavelength band, thereby obtaining sample images corresponding to a plurality of samples to be analyzed. Exemplarily, the preset wavelength band is 900 nm to 1700 nm (a wavelength band selected from the near-infrared wavelength band), the sample to be analyzed is a pathological specimen excised by surgery, a halogen lamp is used as the illumination light source, and a hyperspectral camera is used as the image acquisition device, so as to collect the pathological specimen excised by surgery under different wavelengths. For example, within the preset wavelength band, one image is collected for each wavelength using a hyperspectral camera, thereby obtaining a plurality of sample images corresponding to different wavelengths.

[0042] As shown in FIG. 3, irradiate the sample 340 to be analyzed with the line light source 330, and capture the sample 340 to be analyzed irradiated with different wavelengths by the short-wavelength infrared hyperspectral camera 320, thereby realizing the process of obtaining a plurality of sample images at a plurality of different wavelengths. FIG. 4 shows a plurality of sample images 410 collected using the short-wavelength infrared hyperspectral camera 320. The plurality of sample images are arranged in order from top to bottom according to the wavelength range of 900 nm to 1700 nm. The plurality of sample images 410 are three-dimensional hyperspectral images having spectral information.

[0043] Exemplarily, as shown in FIG. 5, arbitrarily select the M point 420 and the N point 430 in the sample image 410, and obtain a spectral characteristic curve 510 corresponding to the M point 420 and a spectral characteristic curve 520 corresponding to the N point 430 based on the spectral information corresponding to the sample image 410. Here, the spectral characteristic graph is a graph showing the relationship between the light reflectance and the wavelength. The abscissa is the wavelength, the ordinate is the reflectance, and the reflectance indicates the ratio of the light beam reflected by the sample to be analyzed to the light beam incident on the sample to be analyzed. Optionally, the spectral characteristic curve in FIG. 5 is a curve with corrected reflectance.

[0044] In one selectable embodiment, within a preset wavelength band range, at least one wavelength is determined using a wavelength tunable filter, and a push-bloom collection operation is performed on the sample to be analyzed by a collection device, thereby obtaining a sample image corresponding to at least one wavelength.

[0045] Optionally, a liquid crystal tunable filter (LCTF) is added in front of the hyperspectral camera. The liquid crystal tunable filter can quickly and vibrationlessly select wavelengths within the visible light wavelength band or the near-infrared wavelength band by selecting wavelengths from an illumination light source that covers a preset wavelength band. For example, the wavelength band covered by the illumination light source is 900nm - 1700nm. When the light emitted from the illumination light source is passed through the liquid crystal tunable filter, light with a wavelength of 1130nm is obtained. That is, the liquid crystal tunable filter filters out and removes light with wavelengths other than 1130nm within the preset wavelength band.

[0046] Optionally, the collection device is a hyperspectral camera. The hyperspectral camera, as a camera with a built-in grid push-bloom structure, obtains a sample image by performing a push-bloom collection operation on the sample to be analyzed according to the grid arrangement method. Or, the hyperspectral camera is a hyperspectral imaging method with an external push-bloom structure. As shown in Figure 3, the sample holder 310 is moved to perform push-bloom imaging, etc. It should be noted that the above are only illustrative and the embodiments of the present application are not limited thereto.

[0047] In some embodiments, the lens in the hyperspectral camera may be a zoom lens that adjusts the field of view by optical zoom. Optionally, a sample image may be obtained by performing field of view matching with a physical lifting optical holder, or by combining a zoom lens and a lifting holder, etc.

[0048] In step 220, a pseudo-color image is obtained by acquiring a first image corresponding to at least one preset wavelength within a preset wavelength band in the sample image.

[0049] Exemplarily, the sample image is a plurality of images obtained by collecting a sample to be analyzed, and the wavelengths corresponding to the sample image are within a preset wavelength band. At least one preset wavelength is selected from a plurality of wavelengths within the preset wavelength band, and the sample image corresponding to the preset wavelength is used as the first image, and finally a pseudo-color image is obtained.

[0050] Optionally, for one preset wavelength, there is at least one sample image corresponding to the preset wavelength. Exemplarily, when there are a plurality of sample images corresponding to the selected preset wavelength, one sample image may be randomly selected from the plurality of sample images as the first image corresponding to the preset wavelength, or a combined analysis may be performed on the plurality of sample images to determine the first image corresponding to the preset wavelength.

[0051] Or, for one preset wavelength, there is one sample image corresponding to the preset wavelength, and the sample image is used as the first image. Optionally, taking the example that one preset wavelength corresponds to one first image, there are differences in the processing methods for the first image according to the number of selected preset wavelengths. Exemplarily, the cases of selecting one preset wavelength and selecting a plurality of preset wavelengths are analyzed respectively.

[0052] (1) When selecting one preset wavelength In one selectable embodiment, a pseudo-color image is obtained by performing a coloring process on the first image corresponding to one preset wavelength within a preset wavelength band.

[0053] Exemplarily, the preset wavelength band is a wavelength band of 900 nm to 1700 nm selected from the near-infrared wavelength band. Since the preset wavelength band is an invisible wavelength band, the image corresponding to the wavelength band is a grayscale image. One preset wavelength is selected from the preset wavelength band, and the first image corresponding to the preset wavelength is a grayscale image. A pseudo-color image is obtained by performing a coloring process on the grayscale image.

[0054] Pseudo-color image processing is for improving the visibility of image content by showing the technical process of converting a black-and-white grayscale image into a color image. Exemplarily, pseudo-color image processing is performed using methods such as the grayscale segmentation method and the grayscale conversion method.

[0055] Optionally, the grayscale image is a single-channel image where the value representing the color of each pixel point is only one. Its pixel value is located between 0 and 255, where 0 indicates black, 255 indicates white, and intermediate values indicate different levels of gray. Or, when the grayscale image is a three-channel image, the pixel values of the three channels are all the same.

[0056] Optionally, the image relative to the single-channel image includes a three-channel image where each pixel point is represented by three values. Exemplarily, an RGB image is a three-channel image, and various colors are obtained by the changes of the three color channels of red (R), green (G), and blue (B) and their superposition. Here, each pixel point is represented by three values.

[0057] Exemplarily, the sample to be analyzed is a pathological sample excised by surgery, and FIG. 6 is a diagram showing different image processes performed on the sample to be analyzed. Here, FIG. 610 is for showing the sample to be analyzed (in order to make it easier to display, it is taken by a normal camera), FIG. 620 is for showing a hyperspectral image with a wavelength of 1300 nm, and FIGS. 631 to 634 are for showing tissue expressions observed by the hematoxylin-eosin staining (HE) method. Among them, FIG. 631 is for showing a cancer tissue (the one indicated by point A in FIG. 610 or FIG. 620), FIG. 632 is for showing an adipose tissue (the one indicated by point B in FIG. 610 or FIG. 620), FIG. 633 is for showing a normal mucosal tissue (the one indicated by point C in FIG. 610 or FIG. 620), and FIG. 634 is for showing a muscle tissue (the one indicated by point D in FIG. 610 or FIG. 620).

[0058] Exemplarily, a preset wavelength of 1300 nm is selected, the hyperspectral image corresponding to FIG. 620 is used as a first image corresponding to the preset wavelength, and a pseudo-color image is obtained by performing the above coloring process on the first image.

[0059] (2) When selecting a plurality of preset wavelengths Exemplarily, based on at least two preset wavelengths, at least two first images respectively corresponding to the at least two preset wavelengths are determined. Among them, the i-th preset wavelength corresponds to the i-th first image, and i is a positive integer.

[0060] In one selectable embodiment, when a synthesis process is performed on at least two first images corresponding to at least two preset wavelengths within a preset wavelength band and a coloring process is performed on the synthesized image, a pseudo-color image is obtained.

[0061] Exemplarily, at least two preset wavelengths are selected from a preset wavelength band, and each of the preset wavelengths corresponds to one first image. Optionally, a candidate image is obtained by performing a synthesis process on at least two first images.

[0062] Exemplarily, the method of performing a synthesis process on a plurality of first images includes at least one of the following methods.

[0063] (1) Pixel value processing In one selectable embodiment, an averaging process is performed on the first pixel values of the pixel points corresponding to at least two first images to obtain the second pixel values of the corresponding pixel points, and a candidate image is determined based on the second pixel values corresponding to each pixel point.

[0064] Exemplarily, after obtaining at least two first images corresponding to at least two preset wavelengths, the first pixel values of the pixel points corresponding to the at least two first images are added and averaged to obtain, as the pixel value of the corresponding pixel point, a second pixel value that is the average value obtained by comprehensively analyzing the first pixel values of the pixel points corresponding to different first images. Optionally, after determining the second pixel value corresponding to each pixel point, a candidate image is obtained based on the position information of the pixel points, where the pixel value of each pixel point is the corresponding second pixel value.

[0065] Optionally, when synthesizing the first images corresponding to different preset wavelengths, the first pixel values of the pixel points corresponding to different first images are used, and the second pixel value obtained by performing an averaging process on a plurality of first pixel values is used as the pixel value of the pixel point corresponding to the second image, so as to perform a comprehensive balancing process on a plurality of different first images by the pixel points corresponding to the image, and better represent the balance levels of different first images.

[0066] (2) Software processing In one selectable embodiment, after determining a first image corresponding to a preset wavelength, at least two first images are synthetically processed by software to obtain a candidate image.

[0067] Exemplarily, at least two first images are input into Photoshop, and operations such as alignment, splicing, color matching, seam removal, and derivation are performed to obtain a candidate image in which at least two first images are synthesized.

[0068] The above is merely exemplary and is not limited in the embodiments of the present application. Optionally, when the selected preset wavelength is realized as a plurality of preset wavelengths within a preset wavelength band, comprehensive analysis is performed on the first images respectively corresponding to different preset wavelengths, that is, by the above synthesis method, the first images respectively corresponding to different preset wavelengths are synthesized, and a more comprehensive analysis process can be performed on the analysis target sample from multiple wavelength dimensions. By synthesizing a candidate image with the first images respectively corresponding to multiple wavelengths, sample information shared by multiple wavelengths is included in the candidate image, the difference in image information due to wavelength difference is smoothed, and the constraint of analysis is avoided.

[0069] Exemplarily, based on the luminance value of the pixel points in the candidate image, luminance grading is performed on the pixel points in the candidate image to determine at least two luminance levels, and by coloring each of the at least two luminance levels respectively, a pseudo-color image is obtained.

[0070] Optionally, the first image is a grayscale image, the candidate image synthesized based on the first image is a grayscale image, and the second pixel value corresponding to each pixel point in the grayscale image is for indicating the luminance of the candidate image. Exemplarily, the second pixel value is located between 0 and 255, 0 indicates black (the lowest luminance), and 255 indicates white (the highest luminance), that is, the smaller the numerical value of the second pixel value, the lower the luminance, and the larger the numerical value of the second pixel value, the higher the luminance.

[0071] Exemplarily, FIG. 7 shows a process of obtaining a pseudo-color image by performing a synthesis process and a coloring process on a first image corresponding to three preset wavelengths (a wavelength of 1100 nm, a wavelength of 1300 nm, and a wavelength of 1450 nm).

[0072] Among them, FIG. 710 shows a hyperspectral image with a wavelength of 1100 nm, FIG. 720 shows a hyperspectral image with a wavelength of 1300 nm, and FIG. 730 shows a hyperspectral image with a wavelength of 1450 nm. Optionally, when a synthesis process and a coloring process are performed on the hyperspectral images corresponding to the above three preset wavelengths, the pseudo-color image shown in FIG. 740 is obtained.

[0073] Exemplarily, after obtaining a first image corresponding to one preset wavelength or a candidate image obtained by synthesizing first images respectively corresponding to a plurality of preset wavelengths, based on the luminance values of the pixel points in the first image or the candidate image, luminance grading is performed on the pixel points therein, and at least two luminance levels are determined, and then coloring is performed for different luminance levels respectively to obtain a pseudo-color image. By the above coloring process, due to the luminance change of the pixel points of the image, coloring is performed on the pixel points corresponding to different luminance levels, so that the colored pseudo-color image conforms to the image observation habit of the human eye, and it is convenient for experts to distinguish different image regions by different colors in the pseudo-color image.

[0074] In step 230, based on the differences in the sample element types in the sample image, region division is performed on the sample image to obtain a region division result.

[0075] Here, the sample element type includes the identification target element type to be identified.

[0076] Optionally, the sample element type is for indicating differences in sample properties corresponding to different sample regions in the sample image. Exemplarily, when the sample image is an image of a pathological sample, the sample element type includes tumor tissue in the pathological sample, adipose tissue in the pathological sample, mucosal tissue in the pathological sample, muscle tissue in the pathological sample, and the like. When the sample image is an image of a chemical mixture (including compound A, compound B, and impurities), the sample element type includes compound A, compound B, and impurities.

[0077] Exemplarily, when the sample image is a pathological image of a pathological sample, the element type to be identified is a pre-determined tumor tissue to be identified (one of the sample element types corresponding to the pathological image), or a pre-determined adipose tissue, etc. Optionally, when the sample image is a chemical image of a chemical mixture, the element type to be identified is a pre-determined compound B to be identified (one of the sample element types corresponding to the chemical image), etc.

[0078] Exemplarily, the sample image is an image having spectral information, and there are differences in the spectral information according to the differences in the properties of different substances. By performing region division on the sample image based on the spectral information corresponding to the sample image, a region division result is obtained.

[0079] Exemplarily, the display of the spectral information in the sample image is different. For example, when the sample image is a grayscale image, the color of the region corresponding to the A sample element is made the darkest, and the color of the region corresponding to the sample element to be analyzed is made the lightest, thereby obtaining a region division result corresponding to the sample image.

[0080] In one selectable embodiment, for easy distinction, different regions can be filled with different colors to obtain a region division result having colors, or different regions can be divided using thick contour lines to obtain an obviously divided region division result, etc.

[0081] Note that the above is merely exemplary, and the embodiments of the present application are not limited thereto.

[0082] In step 240, based on the pseudo-color image and the region segmentation result, an image region including the element type to be identified in the sample image is determined.

[0083] Exemplarily, the pseudo-color image is an image obtained by processing a first image corresponding to a preset wavelength, and the region segmentation result is the result of performing region segmentation based on the sample element type in the sample image. Optionally, the pseudo-color image is segmented in different colors in the pseudo-color image. For example, when the sample image is an image of a pathological sample, among them, the tumor tissue exhibits orange, the adipose tissue exhibits bright yellow, the mucosal tissue exhibits light orange with a lighter color than the tumor tissue, and the muscle tissue exhibits dark orange with a darker color than the tumor tissue.

[0084] In one selectable embodiment, the overlapping region between the pseudo-color image and the region segmentation result is determined.

[0085] In the sample image, the overlapping region is used as the image region including the element type to be identified.

[0086] Exemplarily, the sample to be analyzed is a pathological sample. When a sample image having spectral information is obtained by collecting the pathological sample and the tumor tissue in the sample image is to be observed, by performing the above processing process on the sample image, a pseudo-color image corresponding to the selected preset wavelength and a region segmentation result for the sample image are obtained. Based on the region of the tumor tissue determined in the pseudo-color image and the identification result of the element type to be identified (tumor tissue) included in the region segmentation result, the overlapping region is used as the image region including the element type to be identified (tumor tissue), and the identification process for the tumor tissue region is realized.

[0087] Exemplarily, after obtaining a pseudo-color image of a first image corresponding to a preset wavelength with good spectral effects and a region division result obtained by performing region division on a sample image, by integrating the pseudo-color image obtained after colorization processing and the spectral analysis result, and performing a more comprehensive analysis process on the sample image, the image region including the element type to be identified not only includes the image information represented by the pseudo-color image, but also includes the spectral information represented by the region division result, sufficiently improving the accuracy of determining the image region.

[0088] The above is only exemplary and is not limited in the embodiments of the present application.

[0089] As described above, a first image corresponding to a preset wavelength is obtained from a sample image to obtain a pseudo-color image, region division is performed on the sample image to obtain a region division result, and the pseudo-color image and the region division result are combined to determine an image region. According to the above method, it is possible to avoid determining the size, region, etc. of a tumor tissue only by naked-eye observation and explanation by a doctor. Based on a preset wavelength band determined in advance, an analysis target sample is collected to obtain a sample image, at least one preset wavelength with good effects is selected from the preset wavelength band, and also, based on at least one preset wavelength, a first image corresponding to the preset wavelength is determined from the sample image, and by processing the first image, a pseudo-color image that can relatively accurately represent the advantages of the preset wavelength is obtained. Based on the differences in the sample element types in the sample image, region division is performed on the sample image to obtain a region division result. By combining the pseudo-color image and the region division result and determining an image region including the element type to be identified, the position information of the region to be identified (for example, a tumor tissue) is determined, improving the accuracy of pathological sampling while reducing the difficulty of pathological sampling. By analyzing the analysis target sample using the spectral characteristics corresponding to the analysis target sample at different wavelengths, not only is the operation relatively simple, but the cost is also relatively low, making application and wide popularization easier.

[0090] In one selectable embodiment, the process of performing region division on a sample image is determined by different spectral information corresponding to different sample element types. Exemplarily, as shown in FIG. 8, step 230 in the embodiment shown in FIG. 2 above may be implemented as the following steps 810 to 850.

[0091] In step 810, a second image is acquired.

[0092] Here, the second image is a pre-labeled image with spectral information obtained by collecting an analysis target sample.

[0093] Optionally, the sample image and the second image are images obtained by collecting an analysis target sample. When collecting an analysis target sample, the gold standard of the sample image is determined, that is, the second image is determined. The gold standard is for indicating a reliable method for disease diagnosis recognized in the current clinical medical field.

[0094] In one selectable embodiment, the analysis target sample is a pathological sample excised by surgery. Exemplarily, FIG. 9 is a diagram showing the flow of collecting a pathological sample. First, a doctor excises a patient's pathological sample by tumor resection surgery 910 (the pathological sample includes tumor tissue), and then, after obtaining an appropriate volume of tissue block by performing cutting 920 on the pathological sample, fixation 930 is performed on the tissue block using methods such as formalin immersion. For example, within 30 minutes after excising the pathological sample by surgery, the tissue block is placed in a sufficient amount of 3.7% neutral formalin solution for fixation, and the fixation time is 12h to 48h. Then, a tissue slice with a thickness of 5 mm ± 1 mm (about 5 mm on average) including the tumor tissue and 1 to 2 cm of normal tissue around it is cut from the fixed tissue block. Optionally, the tissue block fixed for the pathological sample may be used as the analysis target sample, or the tissue slice sliced from the tissue block may be used as the analysis target sample.

[0095] In one selectable embodiment, after obtaining a sample to be analyzed, a stained image 940 is obtained by performing substantially sampling, normal dehydration, embedding, and HE staining slice preparation processes on the tissue slice, and the stained image 940 is scanned with a digital scanner 950 to obtain a plurality of WSI (Whole Slide Image) images 960. Here, a WSI is an image of the pathological sample collected by a digital scanner 950 (a kind of electric microscope structure). Exemplarily, if the size of a single WSI image is small, the WSI image to be analyzed may be formed by splicing a plurality of pathological slices. For example, a virtual large slice 970 is restored and obtained by splicing a plurality of WSI fragments using WSI splicing software.

[0096] Optionally, in the virtual large slice 970, ASAP (Advanced Systems Analysis Program) is used for labeling as a gold standard, and a plurality of labeled WSI images obtained by scanning the pathological sample are acquired. Here, the labeling may be performed in a manner of labeling regions, and the labeled regions include not only regions where one or more types of lesions are located, but also special regions with a hinting effect, etc. Exemplarily, in a hollow organ, the tumor tissue is labeled red, the normal mucosa is labeled green, the adipose tissue is labeled yellow, and the muscle tissue is labeled blue respectively. In a solid organ, the tumor tissue is labeled red, the normal tissue is labeled green, and the adipose tissue is labeled yellow, etc. Optionally, the above color labeling is only exemplary, and the selected tissues may be labeled with different colors. For example, when labeling breast tissue in a solid organ, the tumor tissue in the breast tissue may be labeled red, the adipose tissue may be labeled yellow, and the fibrous connective tissue may be labeled green respectively. Optionally, when there is no corresponding color tissue in the observed organ, labeling may not be necessary. For example, when labeling a solid organ by the above color labeling method, if there is no adipose tissue in the observed organ, yellow may not be labeled. Exemplarily, the labeled WSI image is used as a second image, and the acquisition process of the second image is realized.

[0097] In step 820, train the candidate segmentation model with the second image.

[0098] Here, the candidate segmentation model is a model with a certain region segmentation function that has not been trained. Exemplarily, using the second image as the gold standard, training the candidate segmentation model and training it with a large number of second images enables the candidate segmentation model to learn, gradually making it possible to automatically identify special regions such as lesion regions, and gradually equipping it with a region segmentation function.

[0099] In step 830, in response to the training of the candidate segmentation model achieving the training effect, obtain an image segmentation model.

[0100] Here, the image segmentation model is for performing region segmentation on the first image. Exemplarily, in the process of training the candidate segmentation model, when the training of the candidate segmentation model achieves the training goal, an image segmentation model is obtained. Optionally, the training effect of the candidate segmentation model is judged by the loss value, and the training goal includes at least one of the following cases.

[0101] 1. In response to the loss value reaching the convergence state, use the candidate segmentation model obtained in the most recent round of iterative training as the image segmentation model.

[0102] Exemplarily, the loss value reaching the convergence state indicates that the numerical value of the loss value obtained by the loss function no longer changes or the change range is smaller than a preset threshold. For example, if the loss value corresponding to the nth second image is 0.1 and the loss value corresponding to the (n + 1)th second image is also 0.1, it is considered that the loss value has reached the convergence state, and the candidate segmentation model with the loss value adjusted corresponding to the nth second image or the (n + 1)th second image is used as the image segmentation model to realize the training process of the candidate segmentation model.

[0103] 2. In response to the number of times of obtaining the loss value reaching the number threshold, use the candidate segmentation model obtained by the most recent round of iterative training as the image segmentation model.

[0104] Exemplarily, one loss value can be obtained by one acquisition. The number of acquisitions of the loss value for training the image segmentation model is preset. When one second image corresponds to one loss value, the number of acquisitions of the loss value is the number of second images. Or, when one second image corresponds to a plurality of loss values, the number of acquisitions of the loss value is the number of loss values. For example, if one loss value is obtained by one acquisition and the acquisition number threshold of the loss value is preset to be 10 times, when the acquisition number threshold is reached, the candidate segmentation model with the most recently adjusted loss value is used as the image segmentation model, or the candidate segmentation model with the minimum loss value in the 10 adjustment processes of the loss value is used as the image segmentation model, thereby realizing the training process for the candidate segmentation model.

[0105] It should be noted that the above is only exemplary and the embodiments of the present application do not limit this.

[0106] In one selectable embodiment, the deep learning network related to the candidate segmentation model may be a deep learning network such as a Convolutional Networks for Biomedical Image Segmentation (U-net), a Generative Adversarial Networks (GAN), or a Convolutional Neural Networks (CNN) for biomedical image segmentation. Here, the deep learning network is a policy for performing region segmentation.

[0107] Optionally, a machine learning algorithm other than deep learning, such as a Principal Component Analysis (PCA) method, etc., may be used, or other non-machine learning algorithms, such as a Support Vector Machine (SVM), a maximum likelihood method, a spectral angle, a spectral information divergence, a Mahalanobis distance, etc., may be used.

[0108] Exemplarily, by labeling the pathological region in the pathological sample, a second image that can represent the disease diagnosis gold standard recognized in the current clinical medical field is obtained. The second image can relatively accurately indicate the pathological position corresponding to the pathological sample, and is further convenient for more accurate training of the candidate segmentation model, gradually improving the robustness of the candidate segmentation model, and obtaining an image segmentation model that meets the training effect.

[0109] In step 840, by passing the sample image through a pre-trained image segmentation model, the differential representation of the element type is determined.

[0110] In one selectable embodiment, after performing preprocessing on the sample image, it is input into a pre-trained image segmentation model.

[0111] Exemplarily, as shown in FIG. 10, after collecting a plurality of sample images 1010, preprocessing 1020 is performed on the sample images 1010. Here, the process of performing preprocessing 1020 on the sample image includes performing at least one operation among geometric transformation operations, image enhancement operations, etc. (such as image background correction, registration, noise removal, etc.) on the sample image to emphasize the important features in the sample image. Then, the preprocessed sample image 1010 is passed through a pre-trained image segmentation model 1030, and the image segmentation model 1030 divides the regions in the sample image.

[0112] In one selectable embodiment, the sample image is an image having spectral information. By performing spectral analysis on the sample image, a spectral analysis result is obtained, and based on the spectral analysis result, the differential representation of the element type corresponding to the sample image is determined.

[0113] Based on the differences in the samples to be analyzed corresponding to the sample images, different spectral analysis results corresponding to different sample images are obtained. Exemplarily, the spectral analysis results are represented as spectral feature graphs. The abscissa of the spectral feature graph is the wavelength, the ordinate is the reflectance, and the different spectral curves indicate the change situations of the reflectances of different samples to be analyzed at different wavelengths, that is, for showing the spectral analysis results.

[0114] In one selectable embodiment, as a result of analyzing the hyperspectral images of 62 different system tissues, it was initially determined that the wavelengths for distinguishing tumor tissues from normal tissues in different organs are 1296 - 1308 nm (the effect is good within this wavelength range). Exemplarily, by analyzing the hollow organs (such as the esophagus, stomach, colorectal) where tumor tissues exist, the kidneys, mammary glands, and lungs as the samples to be analyzed, sample images corresponding to the hollow organs, kidneys, mammary glands, and lungs are obtained. The sample images are three-dimensional hyperspectral images. Based on the data corresponding to the three-dimensional hyperspectral images, spectral feature graphs corresponding to the hollow organs, kidneys, mammary glands, and lungs are obtained respectively.

[0115] FIG. 11 is a spectral characteristic graph corresponding to the hollow organ 1110. Among them, the wavelength curve corresponding to the tumor tissue (cancer tissue) is the tumor wavelength curve 1120, the wavelength curve corresponding to the adipose tissue is the adipose wavelength curve 1130, the wavelength curve corresponding to the normal mucosa is the mucosal wavelength curve 1140, and the wavelength curve corresponding to the muscular tissue is the muscular tissue wavelength curve 1150.

[0116] FIG. 12 is a spectral characteristic graph corresponding to the kidney 1210. Among them, the wavelength curve corresponding to the tumor tissue (cancer tissue) is the tumor wavelength curve 1220, the wavelength curve corresponding to the adipose tissue is the adipose wavelength curve 1230, and the wavelength curve corresponding to the normal mucosa is the mucosal wavelength curve 1240.

[0117] FIG. 13 is a spectral characteristic graph corresponding to a mammary gland 1310, in which the wavelength curve corresponding to tumor tissue (cancerous tissue) is tumor wavelength curve 1320, the wavelength curve corresponding to adipose tissue is fat wavelength curve 1330, and the wavelength curve corresponding to normal mucosa is mucosa wavelength curve 1340.

[0118] FIG. 14 is a spectral characteristic graph corresponding to lung 1410, in which the wavelength curve corresponding to tumor tissue (cancerous tissue) is tumor wavelength curve 1420, and the wavelength curve corresponding to normal lung is normal wavelength curve 1430.

[0119] Here, the difference representation of the element type corresponding to the sample image is the difference between different tissues, such as the difference between tumor tissue and fat tissue, etc. The above is merely exemplary, and the embodiments of the present application are not limited thereto.

[0120] 11 to 14, when the wavelength is about 1300 nm, different tissues in the hollow organ tissue samples show good differentiation, and tumor tissues in solid organs (e.g., breast, kidney, and lung) also show relatively good differentiation from the surrounding normal tissues and adipose tissues.

[0121] Taking colon cancer as an example, when observed with the naked eye, tumor tissue in a 1300 nm hyperspectral image appears gray, normal muscle tissue appears gray-black, which is darker than tumor tissue, fat tissue appears gray-white, and normal mucosa appears dark gray, which is thinner than the muscle layer and slightly darker than tumor tissue, and the 1300 nm hyperspectral image shows good differentiation between fat, muscle layer, and tumor tissue.

[0122] In one selectable embodiment, three wave valleys at 1100 nm, 1300 nm, and 1450 nm in the hyperspectral image are extracted as characteristic wavelength bands, and a short-wavelength infrared color composite image is synthesized to provide a pseudo-color image that conforms to the observation habits of the human eye for the identification of different tissues by doctors. In the short-wavelength infrared color composite image, cancerous tissue appears orange, muscle tissue appears darker orange than tumor tissue, normal mucosa appears lighter orange than cancerous tissue, and adipose tissue appears bright yellow.

[0123] Exemplarily, based on the spectral information of the sample image, spectral analysis is performed on the sample image, and the spectral analysis result corresponding to the sample image is used to more intuitively determine the change situation of the reflectance of the sample to be analyzed at different wavelengths, and further determine the difference situation between different tissues, which is advantageous for performing region analysis on the sample image based on the difference.

[0124] In step 850, based on the differential expression of the element type, region division is performed on the sample image to determine the region division result corresponding to the sample image.

[0125] Exemplarily, after obtaining the spectral analysis result, the corresponding region information hint is provided to the sample image by the image segmentation model, and the region information hint includes at least one of the following methods.

[0126] (1) Contour hint Exemplarily, different regions in the sample image are segmented by using the contour line to obtain different segmented regions. Here, the contour line may be a thick curve or a colored curve, etc.

[0127] (2) Heatmap hint Exemplarily, the region where the tumor tissue is located is displayed in a special highlighting manner.

[0128] (3) Monochromatic filling hint Exemplarily, as shown in FIG. 15, different regions are distinguished by being filled with different colors. For example, the tumor tissue region is filled with red 1510, and the adipose tissue region is filled with green 1520. Optionally, for regions that cannot be accurately segmented, they can be filled with white or not filled at all.

[0129] In one selectable embodiment, as shown in FIG. 16, deep learning is performed based on the short-wavelength infrared hyperspectral image 1610 (sample image) and the labeled WSI 1620, and finally the prediction result 1630 that predicts the short-wavelength infrared hyperspectral image 1610 is obtained. Exemplarily, the prediction result 1630 is hinted in a single-color filling manner. The above is only exemplary and is not limited in the embodiments of the present application.

[0130] Exemplarily, after the difference representation of the element type is determined by the image segmentation model, the change situation of the reflectance of the analysis target sample at different wavelengths indicated by the difference representation of the element type is fully utilized to perform region segmentation on the sample image, and by determining the region segmentation result corresponding to the sample image within a preset wavelength band, the analysis dimension of the sample image is subdivided as a region, which is advantageous for improving the accuracy of the analysis of the sample image.

[0131] Summarizing the above, an analysis target sample is collected based on a predetermined preset wavelength band to obtain a sample image, at least one preset wavelength with good effects is selected from the preset wavelength band, a first image corresponding to the preset wavelength is determined from the sample image based on the at least one preset wavelength, and by processing the first image, a pseudo-color image that can relatively accurately represent the advantages of the preset wavelength is obtained. Based on the differences in the sample element types in the sample image, region division is performed on the sample image to obtain a region division result. By combining the pseudo-color image and the region division result and determining the image region including the element type to be identified, the position information of the region to be identified (for example, tumor tissue) is determined. According to the above method, it is possible to avoid judging the size, region, etc. of tumor tissue only by the naked-eye observation and explanation of a doctor, reduce the difficulty of pathological collection, not only make the operation relatively simple, but also make the cost relatively low.

[0132] In the embodiment of the present application, the training process and application process of the region division model are described. When training the region division model, the WSI image is used as the second image, and the untrained candidate division model is trained with the second image until the convergence condition is reached to obtain an image division model. The image division model is used to perform region division on the sample image, and based on the difference representation of the element types in the sample image, the region division result corresponding to the sample image is determined. According to the above method, the model learns the lesion region of the excised tissue, automatically identifies special regions such as the lesion region by the model, and specially labels the image region in the form of region information hints in the image output from the model, so that the model can better analyze the image and improve the accuracy of pathological collection.

[0133] In one selectable embodiment, the above image processing method is applied to the medical field to process pathological images. After obtaining pathological images with infrared hyperspectral information corresponding to different parts, a new solution is provided for assisting in determining the tumor margin during surgery and pathological collection after surgery by combining a narrow-band synthesized pseudo-color image and deep learning to predict the lesion area of the excised tissue. Exemplarily, the above image processing method is applied to at least two discrimination processes: (i) discrimination of tumor tissues in hollow organs, and (ii) discrimination of tumor tissues in solid organs.

[0134] (i) Discrimination of tumor tissues in hollow organs Hollow organs are tubular, such as the stomach, intestines, bladder, gallbladder, etc., and organs that contain a lot of space inside the organ. Solid organs include the heart, lungs, kidneys, liver, spleen, etc. compared to hollow organs. The difference between them is that the former is hollow while the latter is solid. For example, solid organs in the abdomen include the liver, spleen, kidneys, adrenal glands, pancreas, etc. Hollow organs in the abdomen include the gallbladder, stomach, duodenum, jejunum, ileum, appendix, colon, etc.

[0135] In one selectable embodiment, colon cancer tissue, rectal cancer tissue, gastric cancer tissue, and esophageal cancer tissue in hollow organs are studied. Among the four different types of tumor tissues, HSI1300nm shows relatively good discriminability and the imaged colors are similar.

[0136] Compared with X-ray images, hyperspectral imaging shows great advantages in the discrimination of the muscular layer of hollow organs. When judging the boundary of a tumor, the hyperspectral image is significantly clearer than a normal color image. In particular, by selecting HSI images at 1100nm, 1300nm, and 1450nm and synthesizing color images, the range of tumor tissue can be clearly displayed, and different tissues exhibit different intensities of colors from yellow to orange.

[0137] Exemplarily, FIG. 17 shows image representations of four types of tissue classifications selected from hollow organs. Among them, Sample 1 is colon cancer tissue, Sample 2 is rectal cancer tissue, Sample 3 is gastric cancer tissue, and Sample 4 is esophageal cancer tissue.

[0138] Here, FIGS. 1710 to 1740 shown in the first row are for showing normal color images (corresponding to naked-eye observation) taken with an ordinary camera.

[0139] FIGS. 1711 to 1741 shown in the second row are for showing X-ray images obtained using an X-ray device. Although they can display the approximate contour of the tumor region, the effect is not clear, and the muscular layer structure cannot be distinguished either.

[0140] FIGS. 1712 to 1742 shown in the third row are for showing hyperspectral images (HSI1300nm images) with a wavelength of 1300nm collected by a hyperspectral camera. The hyperspectral images are grayscale images, and different tissues show different shades of color and can be distinguished by the naked eye.

[0141] FIGS. 1713 to 1743 shown in the fourth row are for showing pseudo-color images synthesized using hyperspectral images with a wavelength of 1100nm, hyperspectral images with a wavelength of 1300nm, and hyperspectral images with a wavelength of 1450nm.

[0142] FIGS. 1714 to 1744 shown in the fifth row are for showing images segmented by artificial intelligence. For example, they are output images obtained using the above-mentioned region segmentation model, and the images can provide more detailed acquisition information. For example, color A represents tumor tissue, color B represents muscular layer tissue, color C represents normal mucosal tissue, color D represents adipose tissue, and optionally, the darker the color, the higher the reliability.

[0143] Figures 1715 to 1745 shown in the sixth row show WSI images (gold standard) and are intended to display the true extent of tumor tissue.

[0144] (2) Identification of tumor tissue in solid organs In one alternative embodiment, the kidney, lung and breast in the parenchymal organs with lesions (e.g., with tumor tissue) are studied, and the tumor tissue is not difficult to distinguish with the naked eye in a relatively large sample. For example, Figure 18 shows different image representations of renal cancer.

[0145] Figure 1810 shows a normal color image (macroscopic observation) of kidney tissue, in which tumor tissue appears gray-white, fat tissue appears yellow, and normal kidney tissue appears light brown. Figure 1811 shows the tumor boundary in a magnified normal color image. Here, the boundary between the tumor tissue and normal kidney tissue displayed in the magnified image is difficult to distinguish.

[0146] Fig. 1820 is to show an X-ray image taken using an X-ray device, where the X-ray image can show the outline of the tumor, but the border is unclear and normal kidney tissue cannot be well distinguished from the tumor tissue.

[0147] Fig. 1830 shows a hyperspectral image with a wavelength of 1300 nm, and Fig. 1831 shows a tumor boundary in a magnified view of the hyperspectral image with a wavelength of 1300 nm. In the 1300 nm hyperspectral image, the tumor tissue appears grayish white, the normal kidney tissue appears gray, and the fat tissue appears light grayish white, and in the magnified view, the boundary between the tumor tissue and the surrounding normal tissue is clear.

[0148] FIG. 1840 is to show a pseudocolor image and FIG. 1832 is to show the tumor border with the combined pseudocolor image enlarged.

[0149] Exemplarily, the pseudo-color image may be obtained by performing a coloring process on one hyperspectral image corresponding to one wavelength, or may be obtained by performing a synthesis process and a coloring process on multiple hyperspectral images corresponding to multiple wavelengths. Here, the wavelength may be selected randomly or may be determined in advance. Exemplarily, at least one wavelength that is effective when judging from a comprehensive set of experimental results is selected in advance, and a pseudo-color image is obtained based on the hyperspectral image corresponding to the at least one wavelength.

[0150] For example, a wavelength of 1300 nm is selected as an effective wavelength, and a hyperspectral image corresponding to the wavelength of 1300 nm is colored to obtain a pseudo-color image, or a wavelength of 1250 nm is randomly selected, and a hyperspectral image corresponding to the wavelength of 1250 nm is colored to obtain a pseudo-color image, or three wavelengths of 1300 nm, 1100 nm, and 1450 nm are selected as effective wavelengths, and a synthesis process and a coloring process are performed on the hyperspectral images corresponding to the three wavelengths, respectively, to obtain a pseudo-color image.

[0151] In the near-infrared composite color image, renal cancer tissue appears orange, fatty tissue appears bright yellow, and normal renal tissue areas appear orange with a darker tint. As shown in the enlarged image, the boundary between the tumor tissue and the surrounding tissue is clear and easy to distinguish.

[0152] FIG. 1850 is for showing an artificial intelligence segmentation image for performing region segmentation on a sample image. Illustratively, different regions are distinguished in different formats, such as tumor tissue being red, normal kidney tissue being green, and adipose tissue being yellow, with the darker the color, the higher the reliability. In the artificial intelligence segmentation image, red is tumor tissue, green is normal kidney tissue, and yellow is adipose tissue, the darker the color, the higher the reliability, and the tumor contour is more consistent with the tumor boundary of the WSI.

[0153] Diagram 1860 is intended to outline the WSI tumor area.

[0154] In one selectable embodiment, the breast in the solid organ is taken as an example. It is not easy to judge the boundary of the tumor with the naked eye, for example, the boundary of the tumor tissue cannot be accurately identified by the photograph taken by the ordinary camera. As shown in FIG. 19, FIG. 1910 is for showing the ordinary color image taken by the ordinary camera, the part surrounded by the circle is the boundary part of the tumor tissue, and the boundary between the tumor tissue and the surrounding tissue in this part cannot be clearly distinguished in the ordinary color image.

[0155] X-ray images have a good effect on the judgment of tumor tissue, and have traditionally been the main supporting tool in pathology sampling, helping pathologists to find the tumor bed area. Figure 1920 is to show the X-ray images collected by the X-ray device, and in the case shown, the tumor outline displayed in the X-ray image has a burr-like edge, and the area is obviously larger than the tumor outline displayed in the WSI.

[0156] Figure 1930 shows a hyperspectral image at a wavelength of 1300 nm, in which the tumor tissue appears dark gray (corresponding to the irregular shape) and the area circled is the surrounding normal breast tissue, which appears light gray.

[0157] FIG. 1940 is intended to illustrate a pseudocolor image determined based on a hyperspectral image corresponding to at least one selected wavelength. Illustratively, in a pseudocolor image synthesized with short wavelength infrared, areas of tumor tissue appear dark orange compared to the surrounding breast tissue, and adipose tissue appears bright yellow.

[0158] Figure 1950 is to show an artificial intelligence segmentation image (the processing result obtained by processing a sample image with a deep learning model), which provides a relatively accurate information reference of the tumor tissue range, and Figure 1960 is to show a WSI image (WSI is the gold standard).

[0159] Optionally, in the case of breast, the X-ray image obtained by the X-ray device can display dot-like calcification. Exemplarily, as shown in FIG. 20, FIG. 2010 is for showing a normal color image taken by a normal camera, in which the tumor area is grayish white, and the outline of the tumor tissue can be identified. FIG. 2020 is for showing an X-ray image collected by an X-ray device, in which the edge of the tumor tissue can be roughly displayed, the edge appears burr-like, and dot-like calcification (position indicated by an arrow in FIG. 2020) can be seen therein. FIG. 2030 is for showing a hyperspectral image with a wavelength of 1300 nm, in which the tumor tissue is dark gray, the normal breast tissue is lighter in tone than the tumor tissue area, and the fat tissue is grayish white. FIG. 2040 is for showing a short-wave infrared color image obtained by processing the hyperspectral image corresponding to at least one selected wavelength, in which the short-wave infrared color image can display a clearer tumor outline. FIG. 2050 is for showing the image result display by artificial segmentation, and FIG. 2060 is for showing the WSI as the gold standard. Illustratively, the short-wave infrared color image and the image result display by artificial segmentation are combined to determine an image region including tumor tissue, and the contour displayed in the image region has the highest degree of agreement with the gold standard (WSI).

[0160] Summarizing the above, an analysis target sample is collected based on a predetermined preset wavelength band to obtain a sample image, at least one preset wavelength with good effects is selected from the preset wavelength band, a first image corresponding to the preset wavelength is determined from the sample image based on the at least one preset wavelength, and by processing the first image, a pseudo-color image that can relatively accurately represent the advantages of the preset wavelength is obtained. Based on the differences in the sample element types in the sample image, region division is performed on the sample image to obtain a region division result. By combining the pseudo-color image and the region division result to determine an image region including the element type to be identified, the position information of the region to be identified (for example, a tumor tissue) is determined. According to the above method, it is possible to avoid judging the size, region, etc. of a tumor tissue only by a doctor's naked-eye observation and description, reduce the difficulty of pathological collection, not only make the operation relatively simple, but also make the cost relatively low.

[0161] In the embodiments of the present application, the above image processing method is applied to the medical field to analyze hollow organs and solid organs, and the benefits of the above image processing method are proved. On the one hand, the above image processing method is more reliable than the doctor's naked-eye observation and manual touch method, and the consistency of the images is more ensured. On the other hand, the hyperspectral imaging system has the characteristics of no damage, contact and ionizing radiation, and its hardware cost is lower than that of an X-ray device.

[0162] FIG. 21 is a block diagram showing the structure of an image processing apparatus according to an exemplary embodiment of the present application. As shown in FIG. 21, the apparatus includes a sample acquisition module for acquiring a sample image, where the sample image includes an image obtained by collecting an analysis target sample within a preset wavelength band, namely, sample acquisition module 2110 an image acquisition module 2120 for obtaining a pseudo-color image by obtaining a first image corresponding to at least one preset wavelength within the preset wavelength band in the sample image A region division module that performs region division on the sample image based on the differences in sample element types in the sample image to obtain a region division result. The sample element type includes the identification target element type to be identified, and the region division module 2130, and a region determination module 2140 that determines an image region including the identification target element type from the sample image based on the pseudo-color image and the region division result.

[0163] In one selectable embodiment, the image acquisition module 2120 is further used to obtain the pseudo-color image by performing colorization processing on a first image corresponding to a preset wavelength within the preset wavelength band, or by performing synthesis processing on at least two first images corresponding to at least two preset wavelengths within the preset wavelength band and then performing colorization processing on the synthesized image.

[0164] In one selectable embodiment, the image acquisition module 2120 is further used to determine at least two first images respectively corresponding to the at least two preset wavelengths based on the at least two preset wavelengths (the i-th preset wavelength corresponds to the i-th first image, and i is a positive integer), obtain a candidate image by performing synthesis processing on the at least two first images, and obtain the pseudo-color image by performing colorization processing on the candidate image.

[0165] In one selectable embodiment, the image acquisition module 2120 is further used to obtain a second pixel value of the corresponding pixel point by performing averaging processing on the pixel values of the pixel points corresponding to the at least two first images, and determine the candidate image based on the second pixel value corresponding to each pixel point.

[0166] In one selectable embodiment, the image acquisition module 2120 is further configured to perform brightness grading on the pixel points in the candidate image based on the brightness values of the pixel points in the candidate image, thereby determining at least two brightness levels, and by performing color coding on each of the at least two brightness levels, it is also used to obtain the pseudo-color image.

[0167] As shown in FIG. 22, in one selectable embodiment, the region division module 2130 includes a determination unit 2131 that determines the differential representation of the element type by passing the sample image through a pre-trained image segmentation model, and a division unit 2132 that determines the region division result corresponding to the sample image by performing region division on the sample image based on the differential representation of the element type.

[0168] In one selectable embodiment, the sample image is an image having spectral information, and the determination unit 2131 is further configured to perform spectral analysis on the sample image to obtain a spectral analysis result, and is also used to determine the differential representation of the element type corresponding to the sample image based on the spectral analysis result.

[0169] In one selectable embodiment, the apparatus is further configured to obtain a second image, which is a pre-labeled image having spectral information obtained by collecting the analysis target sample, train a candidate segmentation model with the second image, and obtain an image segmentation model for performing region division on the first image in response to the training of the candidate segmentation model achieving a training effect.

[0170] In one selectable embodiment, the sample acquisition module 2110 is further configured to obtain the sample image by performing a push-bloom collection operation on the analysis target sample.

[0171] In one selectable embodiment, the push-bloom collection operation is performed by a collection device, and the sample acquisition module 2110 further determines at least one wavelength using a wavelength-variable filter within the preset wavelength band range, and is also used to obtain a sample image corresponding to the at least one wavelength by performing a push-bloom collection operation on the sample to be analyzed by the collection device.

[0172] In one selectable embodiment, the region determination module 2140 further determines an overlapping region between the pseudo-color image and the region segmentation result, and is also used to set the overlapping region as the image region including the element type to be identified in the sample image.

[0173] To sum up, a sample image is obtained by collecting a sample to be analyzed based on a preset wavelength band determined in advance, at least one preset wavelength with good effect is selected from the preset wavelength band, a first image corresponding to the preset wavelength is determined from the sample image based on the at least one preset wavelength, and by processing the first image, a pseudo-color image that can relatively accurately represent the advantages of the preset wavelength is obtained. Based on the differences in the sample element types in the sample image, region segmentation is performed on the sample image to obtain a region segmentation result. By combining the pseudo-color image and the region segmentation result and determining an image region including the element type to be identified, the position information of the region to be identified (for example, a tumor tissue) is determined. According to the above device, it is possible to avoid judging the size, region, etc. of a tumor tissue only by a doctor's naked-eye observation and description, reduce the difficulty of pathological collection, not only make the operation relatively simple, but also make the cost relatively low.

[0174] Note that the image processing apparatus according to the above embodiment has been described by taking the division of each of the above functional modules as an example. However, in actual applications, if necessary, the above functions can be assigned to different functional modules to be completed. That is, the internal structure of the device can be divided into different functional modules to complete all or part of the above functions. In addition, the image processing apparatus according to the above embodiment has the same idea as the embodiment of the image processing method. Since the details of the specific realization process can be referred to the embodiment of the method, the description is omitted here.

[0175] FIG. 23 is a diagram showing the structure of a server according to one exemplary embodiment of the present application. The server 2300 includes a CPU (Central Processing Unit) 2301, a system memory 2304 including a RAM (Random Access Memory) 2302 and a ROM (Read Only Memory) 2303, and a system bus 2305 connecting the system memory 2304 and the CPU 1201. The server 2300 further includes a mass storage device 2306 that stores an operating system 2313, an application program 2314, and other program modules 2315.

[0176] The mass storage device 2306 is connected to the CPU 1201 via a mass storage controller (not shown) connected to the system bus 2305. The mass storage device 2306 and the computer-readable medium associated therewith provide non-volatile memory for the server 2300.

[0177] Without loss of generality, the computer-readable medium may include computer storage media and communication media.

[0178] According to various embodiments of the present application, the server 2300 may be connected to the network 2312 by a network interface unit 2311 connected to the system bus 2305, or may be connected to another type of network or a remote computer system (not shown) by using the network interface unit 2311.

[0179] The memory further includes one or more programs, and the one or more programs are configured to be stored in the memory and executed by the CPU.

[0180] Embodiments of the present application further provide a computer device including a processor and a memory, where at least one instruction, at least a part of a program, a code set, or an instruction set for implementing the image processing method according to each of the above method embodiments is stored in the memory and is configured to be loaded and executed by the processor.

[0181] Embodiments of the present application provide a computer-readable storage medium storing at least one instruction, at least a part of a program, a code set, or an instruction set for implementing the image processing method according to each of the above method embodiments, and the at least one instruction, at least a part of the program, the code set, or the instruction set is configured to be loaded and executed by the processor.

[0182] Embodiments of the present application provide a computer program product or a computer program including computer instructions stored in a computer-readable storage medium. When a processor of a computer device reads and executes the computer instructions from the computer-readable storage medium, the computer device is caused to execute the image processing method according to any one of the above embodiments.

[0183] Optionally, the computer-readable recording medium may include a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drives), or an optical disk, etc. Among them, the RAM can include a ReRAM (Resistance Random Access Memory) and a DRAM (Dynamic Random Access Memory). The serial numbers of the embodiments of the present application above are only for the purpose of explanation and do not represent the superiority or inferiority of the embodiments.

Claims

1. An image processing method executed by a server, a step of acquiring a sample image, wherein the sample image includes an image obtained by collecting an analysis target sample within a preset wavelength band; a step of obtaining a pseudo-color image by obtaining at least two first images corresponding to wavelengths in at least two preset infrared regions within the preset wavelength band; a step of performing luminance grading on pixel points in the candidate image obtained by performing a compositing process on at least two first images corresponding to wavelengths in at least two preset infrared regions within the preset wavelength band, based on the luminance values of the pixel points, to determine at least two luminance levels, and performing color coding on each of the at least two luminance levels to obtain the pseudo-color image, a step of performing region division on the sample image based on the difference in sample element types in the sample image and the color coding result of the pseudo-color image, to obtain a region division result for each sample element type; a step of determining, based on the color coding result of the pseudo-color image and the region division result, in the sample image, an image region including the color coding result corresponding to the identification target element type among the region-divided image regions as the identification target region, wherein the sample element type includes the identification target element type set as the sample element type to be identified and the other sample element type set as the sample element type not to be identified; An image processing method characterized by the above.

2. The step of obtaining the pseudo-color image by performing a compositing process on at least two first images corresponding to at least two preset wavelengths within the preset wavelength band and performing a color coding process on the composite image is A step of determining at least two first images respectively corresponding to the at least two preset wavelengths based on the at least two preset wavelengths, wherein the i-th preset wavelength corresponds to the i-th first image, and i is a positive integer; A step of obtaining a candidate image by performing a synthesis process on the at least two first images; A step of obtaining the pseudo-color image by performing a coloring process on the candidate image, including: The image processing method according to claim 1, characterized in that.

3. The step of obtaining a candidate image by performing a synthesis process on the at least two first images includes: A step of obtaining a second pixel value of the corresponding pixel point by performing an averaging process on the first pixel value of the pixel point corresponding to the at least two first images; A step of determining the candidate image based on the second pixel value corresponding to each pixel point, including: The image processing method according to claim 2, characterized in that.

4. The step of obtaining a region division result by performing region division on the sample image based on the difference in sample element types in the sample image includes: A step of determining the region division result corresponding to the sample image by passing the sample image through a pre-trained image division model, wherein The sample image is an image having spectral information, A step of determining a difference representation of the sample element types corresponding to the sample image based on the difference in the spectral analysis in the spectral analysis result obtained by performing spectral analysis on the sample image in advance; A step of determining the region division result corresponding to the sample image by performing region division on the sample image based on the difference representation of the sample element types, including: The image processing method according to any one of claims 1 to 3, characterized in that...

5. A step of obtaining a second image, wherein the second image is a pre-labeled image having spectral information obtained by collecting the sample to be analyzed; A step of training a candidate segmentation model with the second image, wherein the candidate segmentation model is a candidate model for obtaining an image segmentation model that performs region segmentation on the first image, and when the candidate segmentation model inputs the second image, a step of obtaining a region segmentation result of the second image based on the spectral information; A step of obtaining the image segmentation model in response to the training of the candidate segmentation model achieving a training effect, further comprising: The image processing method according to claim 4, characterized in that...

6. The step of obtaining a sample image is: Including a step of causing a collection device to perform a push-bloom collection operation on the sample to be analyzed to obtain the sample image. The image processing method according to any one of claims 1 to 3, characterized in that...

7. The step of obtaining a sample image is: A step of determining at least one preset wavelength using a wavelength-variable filter within the preset wavelength band range; A step of obtaining a sample image corresponding to the at least one preset wavelength by causing the collection device to perform a push-bloom collection operation on the sample to be analyzed, including: The image processing method according to claim 6, characterized in that...

8. Based on the coloring result of the pseudo-color image and the region segmentation result, in the sample image, the step of determining, as an identification target region, an image region including the coloring result corresponding to the identification target element type among the region-segmented image regions is: Determining an overlapping region between the pseudo-color image and an image region including a coloring result corresponding to the element type to be identified; In the sample image, using the overlapping region as the image region including the element type to be identified, including: The image processing method according to any one of claims 1 to 3, characterized by the above.

9. A sample acquisition module for acquiring a sample image, wherein the sample image includes an image obtained by collecting an analysis target sample within a preset wavelength band; An image acquisition module for obtaining a pseudo-color image by acquiring at least two first images corresponding to wavelengths of at least two preset infrared regions within the preset wavelength band in the sample image; For a candidate image obtained by performing a synthesis process on at least two first images corresponding to wavelengths of at least two preset infrared regions within the preset wavelength band, based on the luminance value of a pixel point, performing luminance grading on the pixel points in the candidate image to determine at least two luminance levels, and performing coloring on each of the at least two luminance levels to obtain the pseudo-color image; A region division module for performing region division on the sample image based on the difference in sample element types in the sample image and the coloring result of the pseudo-color image to obtain a region division result for each sample element type; A region determination module for determining, in the sample image, an image region including a coloring result corresponding to the element type to be identified as the region to be identified based on the coloring result of the pseudo-color image and the region division result, wherein the sample element type includes an element type to be identified set as the sample element type to be identified and another sample element type set as a sample element type not to be identified; An image processing apparatus characterized by the following.

10. A computer device including a processor and a memory, wherein at least a part of a program for realizing the image processing method according to any one of Claims 1 to 3 is stored in the memory so as to be loaded and executed by the processor. A computer device characterized by the following.

11. including instructions for realizing the image processing method according to any one of Claims 1 to 3 when executed by a processor. A computer program characterized by the following.

Images