Image processing method, device, equipment and storage medium

Abstract

The present application provides an image processing method, apparatus, device and storage medium, the method comprising: obtaining at least one reflected light image of a tissue cut, the reflected light image comprising an image of the tissue cut converted by a camera device to the reflected light of a first light source, the first light source and the camera device being located on the same side of the tissue cut; obtaining at least one transmitted light image of the tissue cut, the transmitted light image comprising an image of the tissue cut converted by the transmitted light transmitted by a second light source through the camera device, wherein the second light source and the camera device are located on different sides of the tissue cut; determining the lesion area in the tissue cut based on the learned light absorption characteristics of the diseased tissue, and the light absorption characteristics at different positions in the tissue cut represented by at least one reflected light image and at least one transmitted light image of the tissue cut. The scheme of the present application can more accurately identify the lesion area in the tissue cut.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] Pathological examination has been widely used in clinical work and scientific research. For example, postoperative tissue analysis of tumor patients can help accurately diagnose the tumor condition.

[0003] Among them, pathological examination means that after cutting off tissue from a certain part of the human or animal body, the cut tissue is divided into multiple tissue blocks of appropriate volume, and the doctor selects the tissue block containing the lesion and makes a pathological section, and observes the morphology of the pathological section under a microscope to check the lesion. In order to obtain accurate lesion information, the process of selecting the tissue block containing the lesion and making the pathological section is particularly important.

[0004] However, doctors mainly identify the lesion area in the tissue block by observing the tissue block with naked eyes and touching the tissue block. This method requires high experience of doctors and is prone to lesion identification errors. Moreover, if the lesion is hidden inside the tissue block, the doctor cannot identify the lesion in the tissue block by naked eye observation alone, and the doctor's touch of the tissue block to identify the lesion is highly subjective and not very accurate. Summary of the invention

[0005] In view of this, the present application provides an image processing method, apparatus, device and storage medium, so as to more accurately identify the lesion area in the tissue section.

[0006] To achieve the above objectives, this application provides the following technical solutions:

[0007] On the one hand, the present application provides an image processing method, comprising:

[0008] Obtaining at least one reflected light image of the tissue cut, the reflected light image comprising an image of the tissue cut after the reflected light of the first light source is converted by an imaging device, wherein the first light source and the imaging device are located on the same side of the tissue cut;

[0009] Obtaining at least one transmitted light image of the tissue cut, the transmitted light image comprising an image of the tissue cut converted by the transmitted light transmitted by the second light source through the imaging device, wherein the second light source and the imaging device are located on different sides of the tissue cut;

[0010] The lesion area in the tissue block is determined based on the learned light absorption characteristics of the diseased tissue and the light absorption characteristics at different positions in the tissue block represented by at least one reflected light image and at least one transmitted light image of the tissue block.

[0011] In a possible implementation, determining the lesion area in the tissue block based on the learned light absorption characteristics of the diseased tissue and the light absorption characteristics at different positions in the tissue block represented by at least one reflected light image and at least one transmitted light image of the tissue block includes:

[0012] Determining a lesion area in the tissue section based on at least one reflected light image and at least one transmitted light image of the tissue section and using a lesion recognition model;

[0013] Among them, the lesion recognition model is obtained through deep learning training using at least one reflected light image and at least one transmitted light image of each of multiple tissue cutting samples marked with actual lesion areas. The lesion recognition model has learned the light absorption characteristics of the diseased tissue.

[0014] In yet another possible implementation, obtaining at least one reflected light image of the tissue section includes:

[0015] Obtaining at least one first reflected light image and at least one second reflected light image of a tissue cut, wherein the first reflected light image includes an image of the reflected light of the tissue cut to a first light source converted by a first imaging device, and the second reflected light image includes an image of the reflected light of the tissue cut to the first light source converted by a second imaging device, the first imaging device is an imaging device whose photosensitive range includes a visible light spectrum, the second imaging device is an imaging device whose photosensitive range includes a short-wave infrared band spectrum, and the first imaging device, the second imaging device and the first light source are located on the same side of the tissue cut;

[0016] The step of obtaining at least one transmitted light image of the tissue section comprises:

[0017] At least one first transmitted light image and at least one second transmitted light image of a tissue cut are obtained, wherein the first transmitted light image includes an image of the tissue cut after being converted by the first imaging device through the transmitted light transmitted by the second light source, and the second transmitted light image includes an image of the tissue cut after being converted by the second imaging device through the transmitted light transmitted by the second light source, and the first imaging device and the second imaging device are both located on different sides of the tissue cut as compared with the second light source.

[0018] In another aspect, the present application further provides an image processing device, comprising:

[0019] A first image acquisition unit, configured to acquire at least one reflected light image of a tissue cut, wherein the reflected light image comprises an image of the tissue cut converted by a camera device to reflected light of a first light source, wherein the first light source and the camera device are located on the same side of the tissue cut;

[0020] A second image acquisition unit, configured to acquire at least one transmitted light image of the tissue cut block, wherein the transmitted light image comprises an image of the tissue cut block after the transmitted light transmitted by the second light source is converted by the imaging device, wherein the second light source and the imaging device are located on different sides of the tissue cut block;

[0021] A lesion determination unit is used to determine a lesion area in the tissue block based on the learned light absorption characteristics of the diseased tissue and the light absorption characteristics at different positions in the tissue block represented by at least one reflected light image and at least one transmitted light image of the tissue block.

[0022] In yet another aspect, the present application also provides a computer device, including a memory and a processor;

[0023] Wherein, the memory is used to store programs;

[0024] The processor is used to execute the program, and when the program is executed, it is specifically used to implement any one of the image processing methods described above.

[0025] On the other hand, the present application also provides a storage medium for storing a program, which, when executed, is used to implement any of the image processing methods described above.

[0026] From the above content, it can be seen that based on the imaging method of the reflected light image and the transmitted light image of the tissue cut, the reflected light image of the tissue cut can reflect the absorption characteristics of the light on the surface of the tissue cut, while the transmitted light image of the tissue cut reflects the absorption characteristics of the light inside the tissue cut. Therefore, the reflected light image and the transmitted light image based on the tissue cut can more comprehensively reflect the tissue characteristics on the surface and inside of the tissue cut. On this basis, since the absorption characteristics of light of different types of tissues such as diseased tissue and normal tissue will also be different, this application combines the light absorption characteristics presented by the diseased tissue, and based on the light absorption characteristics at different positions on the surface and inside of the tissue cut, it can more comprehensively analyze the area where the lesion exists in the tissue cut, and also reduce the situation where the lesion cannot be identified because it is hidden inside the tissue cut, thereby improving the accuracy of identifying the lesion area in the tissue cut. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0028] Figure 1 A schematic diagram of a system architecture applicable to the present application is shown;

[0029] Figure 2 A schematic diagram showing a flow chart of an embodiment of the image processing method provided by the present application;

[0030] Figure 3 A flowchart of another embodiment of the image processing method provided by the present application;

[0031] Figure 4 A schematic diagram of an implementation principle of the image processing method of the present application;

[0032] Figure 5 A schematic diagram of an implementation principle for improving the resolution of the second reflected light image in the image processing method of the present application;

[0033] Figure 6 A schematic diagram of a process for training a lesion recognition model in this application;

[0034] Figure 7 A schematic diagram of the composition structure of an embodiment of an image processing device provided by the present application;

[0035] Figure 8 A schematic diagram of the composition structure of a computer device provided by the present application is shown. DETAILED DESCRIPTION

[0036] The solution of the present application is suitable for analyzing and processing images of human or animal tissue sections using a personal computer, server or cloud platform to determine the lesion area in the tissue section and improve the accuracy of determining the lesion area in the tissue section.

[0037] In order to improve the accuracy of determining the lesion area in the present application, images of the tissue slices are obtained respectively under the conditions of reflected illumination and transmitted illumination, and the images of the two conditions are analyzed.

[0038] To facilitate understanding, the system architecture to which the solution of the present application is applicable is first introduced below.

[0039] like Figure 1 , which shows a schematic diagram of the composition structure of a system architecture applicable to the solution of the present application.

[0040] exist Figure 1 The system architecture includes: an image acquisition system 10 and an image analysis system 20;

[0041] The image acquisition system 10 includes: an imaging device 11, a first light source 12, a second light source 13, and a glass plate 14 for carrying tissue cuts.

[0042] Depend on Figure 1 It can be seen that the first light source 12 and the camera 11 are located on one side of the glass plate 14, and the second light source 13 is located on the other side of the glass plate 14. It can be seen that when a tissue block is placed on the glass block, the first light source 12 and the camera 11 are located on one side of the tissue block, and the second light source 13 is located on the other side of the tissue block.

[0043] Wherein, the tissue block can be a tissue block taken from a human body or an animal body. Of course, although the present application is called a tissue block, in fact, the thickness of the tissue block is also thinner, so it can also be called a tissue section.

[0044] There may be one or more imaging devices 11, and the imaging device may be a camera, a webcam, or an intelligent video camera, etc. For example, the imaging device may be any imaging device capable of collecting hyperspectral images.

[0045] In the case where there are more camera devices, the multiple camera devices may be the same. Alternatively, the multiple camera devices may include at least two camera devices with different photosensitive ranges. For example, the multiple camera devices may include at least one camera device whose photosensitive range includes the visible light spectrum range and at least one camera device whose photosensitive range includes the short-wave infrared band range. For example, the camera device whose photosensitive range includes the visible light spectrum range may be a hyperspectral camera with a photosensitive range of 400nm-1000nm; the camera device whose photosensitive range includes the short-wave infrared band range may be a hyperspectral camera with a photosensitive range of 900nm-1700nm.

[0046] The first light source and the second light source may be light sources output by the same light emitting device, except that the light emitting devices corresponding to the first light source and the second light source need to be deployed on different sides of the tissue block.

[0047] For example, the first light source and the second light source can be halogen lamps, that is, light sources are generated by halogen lamps, wherein when the halogen lamps are used to irradiate the tissue cut pieces, no loss of the tissue cut pieces can be achieved, and no non-ionizing radiation will be generated. Of course, the first light source and the second light source can be incandescent lamps or LED light sources.

[0048] In a possible implementation, the light output by the first light source is aligned with the tissue cut (eg Figure 1The surface of the glass plate is not perpendicular, that is, the angle between the light output by the first light source and the surface of the tissue cut is less than 90 degrees, so that when the light output by the first light source is irradiated on the tissue cut, reflected light (reflection of light) is generated on the surface of the tissue cut. For example, the angle between the light output by the first light source and the surface of the tissue cut is 45 degrees.

[0049] In a possible implementation, the light output by the second light source is perpendicular to the surface of the tissue block, and the light of the second light source can be projected onto the tissue block, so that the light of the second light source can generate transmitted light (transmission of light) in the tissue block. Figure 1 The second light source is located directly below the tissue block.

[0050] In the present application, the glass plate is a high-transmittance material that can be used for light diffusion.

[0051] For example, in one implementation, it may be translucent frosted glass. After light passes through the frosted glass, it can provide more uniform illumination for the tissue cut block on the frosted glass, which can help the second light source provide more uniform illumination for the tissue cut block.

[0052] Of course, the glass plate can also be a frosted acrylic plate, a polycarbonate plate (PC plate), white paper or other high-transmittance material that can be used for light diffusion.

[0053] It is understandable that if the tissue block can be fixed by other means so that the first light source and the second light source are respectively located on two sides of the tissue block, then the glass plate may not be required.

[0054] In the embodiment of the present application, in order to obtain the texture information of the surface of the tissue cut, the present application can use a first light source to illuminate the tissue cut, and obtain a reflected light image of the tissue cut by a camera device. The reflected light image of the tissue cut refers to the image of the reflected light of the tissue cut to the first light source after being converted by the camera device.

[0055] Since the light of the first light source will generate reflected light on the surface of the tissue block after the first light source illuminates the tissue block, the reflected light image is actually an image of the tissue block captured when the first light source is used to illuminate the tissue block and the light of the first light source is reflected on the tissue block.

[0056] When there are multiple first light sources, the multiple first light sources can be used to irradiate the tissue block simultaneously.

[0057] It is understandable that when the first light source illuminates the tissue block, in order to avoid interference, the second light source can be turned off so that the second light source does not emit light.

[0058] It is understandable that, since the reflected light image of the tissue block is the situation of the tissue block collected when the light of the first light source is reflected on the surface of the tissue block, the reflected light image can reflect the absorption of light at different positions on the surface of the tissue block. Different types of tissues have different absorption of light, so the texture information of the surface of the tissue block can be reflected through the reflected light image.

[0059] On the other hand, in order to obtain the internal information of the tissue cut, the second light source can be used to illuminate the tissue cut, and the transmitted light image of the tissue cut can be obtained by the camera device, wherein the transmitted light image includes the image of the tissue cut after the transmitted light transmitted by the second light source is converted by the camera device.

[0060] It is understandable that, since the second light source is located below the tissue cut, and the thickness of the tissue cut is usually between 2mm and 8mm, some light will be able to pass through the tissue cut under the illumination of the second light source. Specifically, when the second light source illuminates the tissue cut, the light of the second light source will be incident on the surface of the tissue cut, so there is a phenomenon of the incident light being refracted and passing through the tissue cut, that is, a transmission phenomenon. It can be seen that when the second light source is used to illuminate the tissue cut, and the light of the second light source is transmitted through the tissue cut, the image of the tissue cut obtained by the camera device is the transmitted light image of the tissue cut.

[0061] It is understandable that when the second light source illuminates the tissue block, in order to avoid interference, the first light source can be turned off so that the first light source does not emit light.

[0062] It is understandable that different types of tissues absorb light differently, and the transmitted light image can reflect the absorption of light at different locations inside the tissue block. Therefore, the transmitted light image can reflect the internal information of the tissue block. For example, the crystal clear part in the transmitted light image is generally fat in the tissue block.

[0063] It should be noted that, since there may be one or more imaging devices, the present application can obtain at least one reflected light image of the tissue cut through at least one imaging device; correspondingly, at least one transmitted light image of the tissue cut can be obtained.

[0064] After obtaining at least one reflected light image and at least one transmitted light image of the tissue segment, the image analysis system 20 may analyze the at least one reflected light image and the at least one transmitted light image to determine the lesion area in the tissue segment.

[0065] The image analysis system 20 includes at least one computer device 21. For example, the image analysis system can be an independent personal computer or server, or a cluster or cloud platform composed of multiple servers.

[0066] In one implementation, in an embodiment of the present application, in order to determine the lesion area of ​​the tissue section, the image analysis system performs image analysis and model machine learning based on artificial intelligence technology.

[0067] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0068] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0069] In the embodiments of the present application, at least the model will be trained based on machine learning. Among them, machine learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning generally include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by formula.

[0070] It is understandable that after the transmitted light image and reflected light image of the tissue section are obtained by the image acquisition system 20, they can be input into the computer device of the image analysis system by the user, or transmitted to the computer device of the image analysis system by the user through the user terminal.

[0071] Of course, when the camera device of the image analysis system has a communication function, the camera device may also transmit the captured transmitted light image and reflected light image to the computer device of the image analysis system.

[0072] In combination with the above content, the image processing method of the present application is introduced below with reference to a flowchart.

[0073] The image processing method of the present application can be executed by a computer device, such as a personal computer, a server or a cloud platform in the image analysis system mentioned above.

[0074] like Figure 2 As shown, it shows a schematic flow chart of an embodiment of an image processing method of the present application. The method of this embodiment may include:

[0075] S201, obtaining at least one reflected light image of a tissue section.

[0076] As can be seen from the above description, the reflected light image includes the image of the reflected light of the tissue block to the first light source after being converted by the camera device. For example, when there are multiple cameras, multiple reflected light images can be obtained.

[0077] The first light source and the camera are located on the same side of the tissue block. If there are multiple cameras, then the multiple cameras and the first light source are also located on the same side of the tissue block.

[0078] For the reflected light image, the first light source and the camera device, reference may be made to the related introduction of the previous embodiments.

[0079] S202, obtaining at least one transmitted light image of the tissue section.

[0080] The transmitted light image includes an image of the tissue cut block after the transmitted light transmitted by the second light source is converted by the camera device. The second light source and the camera device are located on different sides of the tissue cut block.

[0081] It can be understood that, regarding the transmitted light image and the second light source, reference can be made to the related introduction of the previous embodiments, which will not be repeated here.

[0082] S203, determining the lesion area in the tissue block based on the learned light absorption characteristics of the diseased tissue and the light absorption characteristics at different positions in the tissue block represented by at least one reflected light image and at least one transmitted light image of the tissue block.

[0083] The light absorption characteristics of tissues (pathological tissues or normal tissues of human or animal bodies) refer to characteristics such as the degree to which tissues absorb light of different wavelengths.

[0084] Diseased tissue refers to the human or animal tissue that has undergone lesions, that is, the diseased tissue where lesions exist.

[0085] In the present application, the light absorption characteristics of the diseased tissue can be obtained based on the light absorption characteristics of multiple diseased tissue slices. For example, the light absorption characteristics of the diseased tissue can be extracted by performing feature analysis on the transmitted light images and reflected light images of multiple diseased tissue slices. For another example, the light absorption characteristics of the diseased tissue can be determined based on the transmitted light images and reflected light images of multiple diseased tissue slices and using machine learning or deep learning technology.

[0086] Among them, for different diseases, the light absorption characteristics of the lesion tissue under each disease can be learned, or the light absorption characteristics of the lesion tissue applicable to different diseases can be learned, and there is no restriction on this.

[0087] From the previous introduction to transmitted light images and reflected light images, we can see that reflected light images can reflect the light absorption characteristics at different positions on the surface of tissue cuts, which actually reflects the texture information on the surface of tissue cuts; while transmitted light images reflect the light absorption characteristics at different positions inside the tissue cuts, that is, the tissue structure characteristics at different positions inside the tissue cuts.

[0088] It is understandable that the light absorption characteristics of different types of tissues in normal animal tissues without lesions are different, and the light absorption characteristics of animal tissues with lesions are also significantly different from those of normal animal tissues. Based on this, in a possible implementation, by combining the learned light absorption characteristics of the lesion tissue and the light absorption characteristics at various positions of the tissue cut, the lesion area with abnormal light absorption characteristics (e.g., consistent with the light absorption characteristics of the lesion tissue) can be determined from the tissue cut.

[0089] In a possible implementation, the present application may pre-train a lesion recognition model for identifying the lesion area of ​​a tissue segment. On this basis, the lesion recognition model may be used to determine the lesion area in the tissue segment based on at least one reflected light image and at least one transmitted light image of the tissue segment.

[0090] The lesion recognition model is obtained through deep learning training using at least one reflected light image and at least one transmitted light image of each of multiple tissue cutting samples marked with actual lesion areas.

[0091] The lesion recognition model can be trained by a network model such as a convolutional neural network (CNN), a generative adversarial network (GAN), or a U-net, without limitation. This application does not limit the specific process of using deep learning to train the lesion model, and a possible method will be introduced as an example later.

[0092] It can be understood that by training the lesion recognition model, the lesion recognition model can learn the light absorption characteristics of the diseased tissue.

[0093] Wherein, determining the lesion area of ​​the tissue section may be to output the position area information of the lesion area in the tissue section. Determining the lesion area may also be to determine the lesion area in the reflected light image or the transmitted light image of the tissue section. For example, the lesion area is marked by a heat map, a specified color filling, or an outline, so that the user can intuitively view the position of the lesion area. Of course, it is also possible to generate a two-dimensional or three-dimensional image of the tissue section based on the transmitted light image and the reflected light image, and determine the lesion area in the two-dimensional or three-dimensional image of the tissue section.

[0094] From the above content, it can be seen that based on the imaging method of the reflected light image and the transmitted light image of the tissue cut, the reflected light image of the tissue cut can reflect the absorption characteristics of the light on the surface of the tissue cut, while the transmitted light image of the tissue cut reflects the absorption characteristics of the light inside the tissue cut. Therefore, the reflected light image and the transmitted light image based on the tissue cut can more comprehensively reflect the tissue characteristics on the surface and inside of the tissue cut. On this basis, since the absorption characteristics of light of different types of tissues such as diseased tissue and normal tissue will also be different, this application combines the light absorption characteristics presented by the diseased tissue, and based on the light absorption characteristics at different positions on the surface and inside of the tissue cut, it can more comprehensively analyze the area where the lesion exists in the tissue cut, and also reduce the situation where the lesion cannot be identified because it is hidden inside the tissue cut, thereby improving the accuracy of identifying the lesion area in the tissue cut.

[0095] It is understandable that in visible light, the absorption of light by some tissues in the human body or organisms does not show much difference, but in the short-wave infrared light band, more obvious intensity or light color differences may be seen.

[0096] Based on this, in order to make the reflected light image and transmitted light image of the tissue section more intuitively reflect the light absorption characteristics of the tissue section, the present application can set at least one first camera device whose photosensitive range includes the visible light spectrum and at least one second camera device whose photosensitive range includes the short-wave infrared band spectrum.

[0097] For example, the first imaging device is a hyperspectral camera with a sensitivity range of 400nm-1000nm; the second imaging device is a hyperspectral camera with a sensitivity range of 900nm-1700nm.

[0098] The first camera device and the second camera device are also located on the same side of the tissue cut block as the first light source. The first camera device and the second camera device are both located on different sides of the tissue cut block as the second light source.

[0099] In this case, the present application obtains at least one first reflected light image, at least one second reflected light image, at least one first transmitted light image, and at least one second transmitted light image of the tissue cut.

[0100] Specifically, in the present application, when the first light source illuminates the tissue block (the second light source does not emit light), the image of the tissue block is collected by at least one first camera device and at least one second camera device.

[0101] Since there will be reflected light on the surface of the tissue block when the first light source illuminates the tissue block, the first camera device and the second camera device both capture images of the tissue block reflecting the light of the first light source, except that the first camera device and the second camera device capture images of reflected light of the tissue block in different light wave bands.

[0102] For the sake of distinction, the reflected light image of the tissue cut collected by the first camera device is referred to as the first reflected light image, and the reflected light image of the tissue cut collected by the second camera device is referred to as the second reflected light image. It can be seen that the first reflected light image may include the imaging of the reflected light of the first light source of the tissue cut in the visible light range (i.e., converted in the first camera device). The second reflected light image may include the imaging of the reflected light of the first light source of the tissue cut in the short-wave infrared band range (i.e., converted in the second camera device).

[0103] Figure 1 As shown, it can be assumed Figure 1 Among the two cameras, one is a 400nm-1000nm hyperspectral camera, and the other is a 900nm-1700nm hyperspectral camera. After at least one first light source 12 is turned on, the 400nm-1000nm hyperspectral camera can be used to collect a hyperspectral image of the reflected light of the tissue cut in the 400nm-1000nm band to obtain a first reflected light image; at the same time, the 900nm-1700nm hyperspectral camera can be used to collect a hyperspectral image of the reflected light of the tissue cut in the 900nm-1700nm band to obtain a second reflected light image.

[0104] In order to achieve better shooting effect for the 900-1700nm camera device, an infrared long-pass filter may be added in front of the camera device.

[0105] Similarly, in the present application, when collecting the second light source irradiating the tissue cut (the first light source is not emitting light), the first camera can be used to collect the first transmitted light image of the tissue cut, and the second camera can be used to collect the second transmitted light image of the tissue cut. It can be seen that the second transmitted light image includes: the imaging of the tissue slice through the transmitted light transmitted by the second light source within the visible light band; the second transmitted light image includes the imaging of the tissue cut through the transmitted light transmitted by the second light source within the short-wave infrared band.

[0106] Since the first reflected light image and the second reflected light image can respectively reflect the reflection of light on the surface of the tissue cut within the visible light and short infrared light bands, and thus can comprehensively reflect the light absorption on the surface of the tissue cut, the texture information on the surface of the tissue cut can be more comprehensively reflected through the first reflected light image and the second reflected light image.

[0107] Correspondingly, the first transmitted light image and the second transmitted light image can respectively reflect the light absorption inside the tissue block when visible light and short infrared band light are transmitted through the tissue block, thereby more comprehensively reflecting the tissue structure characteristics inside the tissue block.

[0108] On this basis, the lesion area of ​​the tissue section can be determined more accurately according to the first reflected light image, the second reflected light image, the first transmitted light image and the second transmitted light image of the tissue section.

[0109] It is understandable that in order to obtain the above-mentioned hyperspectral images, a first camera device and a second camera device with a built-in grating push-scan structure can be used. If the first camera device and the second camera device do not have a built-in grating push-scan structure, a liquid crystal tunable filter can be added in front of the two camera devices, or a push-scan structure can be installed outside the camera device, and the image can be taken by moving the moving table where the glass plate on which the tissue cut is placed is located.

[0110] In the present application, the lens of the camera device may be a fixed-focus lens. In this case, in order to ensure that the tissue cut is completely photographed, the lifting range of the optical lifting bracket may be used to select a suitable focal length. Alternatively, the present application may select a camera device with a zoom lens.

[0111] It should be noted that the above is an example of obtaining reflected light images and transmitted light images through a first camera device such as a 400nm-1000nm hyperspectral camera whose sensitivity range includes the visible light range and a second camera device such as a 900nm-1700nm hyperspectral camera whose sensitivity range includes the shortwave infrared band range.

[0112] However, if the photosensitivity range of the imaging device covers both the visible light range and the short-wave infrared band range, then the present application may also only set up one or more such imaging devices, and when the first light source or the second light source illuminates the tissue cut, the reflected light image or the transmitted light image of the tissue cut is collected by the imaging device. The reflected light image may include the imaging of the tissue cut of the reflected light of the first light source in the visible light range and the short-wave infrared range; and the transmitted light image includes the imaging of the tissue cut of the transmitted light of the second light source in the visible light range and the short-wave infrared range.

[0113] In the present application, in order to further improve the accuracy of determining the lesion area in the tissue section, on the premise of obtaining the reflected light image and the transmitted light image of the tissue section, machine learning can also be applied to determine the lesion identification in the tissue section. As mentioned above, the present application can be combined with the lesion identification model trained by machine learning to further improve the accuracy of determining the lesion area.

[0114] For ease of understanding, the image processing method of the present application is introduced below by taking the use of a trained lesion recognition model to determine the lesion area of ​​a tissue section as an example.

[0115] like Figure 3 As shown, it shows a flow chart of another embodiment of an image processing method of the present application. The method of this embodiment is applied to a computer device in an image analysis system. This embodiment may include:

[0116] S301, obtaining at least one first reflected light image and at least one second reflected light image of a tissue segment.

[0117] S302, obtaining at least one first transmitted light image and at least one second transmitted light image of the tissue section.

[0118] For example, in an optional manner, the first reflected light image is a first hyperspectral image acquired by a 400nm-1000nm hyperspectral camera when the first light source is used to illuminate the tissue cut and the light of the first light source is reflected on the surface of the tissue cut. The second reflected light image is a second hyperspectral image acquired by a 900nm-1700nm hyperspectral camera when the first light source is used to illuminate the tissue cut and the first light source is reflected on the surface of the tissue cut.

[0119] Similarly, the first transmitted light image is a third hyperspectral image acquired by a 400nm-1000nm hyperspectral camera when the second light source is used to vertically illuminate the tissue cut and the light of the second light source is transmitted through the tissue cut. The second transmitted light image is a fourth hyperspectral image acquired by a 900nm-1700nm hyperspectral camera when the second light source is used to vertically illuminate the tissue cut and the light of the second light source is transmitted through the tissue cut.

[0120] The 400nm-1000nm hyperspectral camera and the 900nm-1700nm hyperspectral camera are located on the same side of the tissue cut as the first light source, and on a different side of the tissue cut as the second light source.

[0121] The acquisition of the above reflected light image and the transmitted light image can refer to the relevant introduction of the previous embodiment, which will not be repeated here.

[0122] It should be noted that this embodiment is described by taking the acquisition of two reflected light images and two transmitted light images of a tissue cut as an example. If only one reflected light image and one transmitted light image are acquired, this embodiment is also applicable.

[0123] S303, inputting the first reflected light image, the second reflected light image, the first transmitted light image and the second transmitted light image of the tissue segment into a lesion recognition model to obtain a lesion prediction image of the tissue segment output by the lesion recognition model.

[0124] The lesion prediction image is an image of a tissue segment, and the lesion region in the predicted tissue segment is marked in the lesion prediction image.

[0125] In an embodiment of the present application, in order to intuitively see and understand the position of the lesion area in the tissue segment, the present application uses a trained lesion recognition model to output a tissue segment image marked with the lesion area, that is, a lesion prediction image.

[0126] Among them, the lesion prediction image output by the lesion recognition model can be an image selected from at least one reflected light image (such as the first reflected light image and the second reflected light image) and at least one transmitted light image (such as the first transmitted light image and the second transmitted light image) as a reference image, and the predicted lesion area is marked in the reference image.

[0127] The lesion recognition model can also synthesize an image of a tissue cut according to the input reflected light image and transmitted light image, and mark the lesion area in the synthesized tissue cut to obtain a lesion prediction image. Of course, there can be other ways to obtain a lesion prediction image, which are not limited.

[0128] There are also many ways to mark the lesion area in the lesion prediction image, such as marking the outline of the lesion area in the tissue section in the lesion prediction image, filling the lesion area with a specific color in the lesion prediction image, or marking the lesion area in the lesion prediction image using a heat map.

[0129] For example, see Figure 4 The images input into the lesion recognition model of the present application include: a 400nm-1000nm reflected light image (first reflected light image) 401 of the tissue section collected by a 400nm-1000nm hyperspectral camera, a 900nm-1700nm hyperspectral camera reflected light image (second reflected light image) 402 of the tissue section collected by a 900nm-1700nm hyperspectral camera, a 400nm-1000nm transmitted light image (first transmitted light image) 403 of the tissue section collected by a 400nm-1000nm hyperspectral camera, and a 900nm-1700nm transmitted light image (second transmitted light image) 404 of the tissue section collected by a 900nm-1700nm hyperspectral camera.

[0130] After the four hyperspectral images are input into the lesion recognition model 405 obtained based on machine learning, the lesion recognition model 405 outputs a lesion prediction image 406 of the tissue segment, wherein the lesion region 407 of the tissue segment is marked in the lesion prediction image 406 .

[0131] In a possible implementation, considering that the resolution of images captured by some camera devices with a photosensitivity range of the short-wave infrared band may be low, in order to improve the resolution of the second reflected light image, before step S303, the present application can also convert each second reflected light image of the tissue section into a synthesized second reflected light image based on the second reflected light image and the first reflected light image of the tissue section and using the first image synthesis model. Accordingly, in step S303, the first reflected light image and the synthesized second reflected light image are input to the lesion recognition model.

[0132] The synthesized second reflected light image has the same content as the second reflected light image before synthesis, and the resolution of the synthesized second reflected light image is higher than the resolution of the second reflected light image before synthesis. The first image synthesis model improves the resolution of the imaging of the reflected light of the first light source in the short-wave infrared band by the tissue cut, which is conducive to improving the accuracy of the lesion recognition model in identifying the lesion area in the tissue cut.

[0133] The first image synthesis model is based on high-resolution reflected light images corresponding to each of the plurality of tissue cutting samples, and is trained using the first reflected light images and the second reflected light images of each of the plurality of tissue cutting samples.

[0134] The high-resolution reflected light image is a reflected light image within the short infrared band corresponding to the tissue cut sample, and the resolution of the high-resolution reflected light image is higher than the resolution of the second reflected light image of the tissue cut sample. That is, the high-resolution reflected light image represents the imaging of the tissue cut in the short infrared band for the reflected light of the first light source when the tissue cut sample is irradiated by the first light source.

[0135] For example, the high-resolution reflected light image of the tissue cut sample may be an image of the tissue cut collected by a camera device with a photosensitivity range in the short infrared band and a pixel count exceeding a set value when the tissue cut sample is illuminated by the first light source.

[0136] For another example, the high-resolution reflected light image of the tissue cutting sample can be stitched from the reflected light images of multiple block areas of the tissue cutting sample. The tissue cutting sample can be divided into the multiple block areas. The reflected light image of each block area is an image of the block area captured by the first camera device when the tissue cutting sample is illuminated by the first light source.

[0137] It can be understood that, for the same object, reducing the camera range of the camera device for the object can improve the resolution of the object part in the captured image. Therefore, the resolution of the image formed by stitching the reflected light images of each block of the tissue block sample is higher than the reflected light image of the tissue block sample directly captured by the first camera device.

[0138] Among them, the first image synthesis model can be a network model such as a GAN network model. The training of the first image synthesis model is based on the training goal that the synthesized second reflected light image obtained by the first image synthesis model for the tissue cutting sample is similar to the high-resolution reflected light image of the tissue cutting sample. There can be many possibilities for the specific training process, and there is no limitation to this.

[0139] like Figure 5 As shown, the present application can obtain a hyperspectral image 501 of the tissue section sample in the range of 400nm-1000nm and a hyperspectral image 502 in the range of 900nm-1700nm of the reflected light of the first light source. The two hyperspectral images are reflected light images obtained by a 400nm-1000nm hyperspectral camera and a 900nm-1700nm hyperspectral camera respectively when the tissue section sample is irradiated with the first light source.

[0140] Among them, since the resolution of the hyperspectral camera of 400nm-1000nm is generally high, the hyperspectral image 501 in the range of 400nm-1000nm has a higher resolution. However, the resolution of some hyperspectral cameras of 900nm-1700nm is relatively low, so the resolution of the hyperspectral image 502 in the range of 900nm-1700nm is relatively low. On this basis, the present application can input the hyperspectral image 501 and the hyperspectral image 502 into the first image synthesis model 503 obtained by machine learning training, so as to obtain the hyperspectral image 504 in the range of 900nm-1700nm corresponding to the tissue section sample, and the resolution of the hyperspectral image 504 in the range of 900nm-1700nm is higher than the resolution of the hyperspectral image 504 in the range of 900nm-1700nm.

[0141] Similarly, in order to improve the resolution of the second transmitted light image, before step S303, the present application can also convert each second transmitted light image of the tissue segment into a synthesized second transmitted light image based on the second transmitted light image and the first transmitted light image of the tissue segment and using the second image synthesis model. Accordingly, in step S303, the first transmitted light image and the synthesized second transmitted light image are input to the lesion recognition model.

[0142] The content of the synthesized second transmitted light image is the same as that of the second transmitted light image before the synthesis, and the resolution of the synthesized second transmitted light image is higher than that of the second transmitted light image before the synthesis.

[0143] The second image synthesis model is obtained by training based on high-resolution transmitted light images corresponding to each of the multiple tissue cutting samples and using the first transmitted light images and the second transmitted light images of each of the multiple tissue cutting samples.

[0144] The high-resolution transmitted light image is a transmitted light image within the short infrared band corresponding to the tissue cut sample, and the resolution of the high-resolution transmitted light image is higher than the resolution of the second transmitted light image of the tissue cut sample.

[0145] Among them, the method of obtaining the high-resolution transmitted light image is similar to the process of obtaining the high-resolution reflected light image, and the process of training the second image synthesis model is similar to the process of training the first image synthesis model, which will not be repeated here.

[0146] In order to facilitate understanding of the training process of the lesion recognition model of the present application, a training process is introduced below as an example. Figure 6 As shown, it shows a schematic diagram of a process of training a lesion recognition model in this application. This process may include:

[0147] S601, obtaining at least one reflected light image and at least one transmitted light image of each of a plurality of tissue cutting samples.

[0148] Each tissue cut sample corresponds to a cut sample image with annotated actual lesion area.

[0149] The tissue cutting sample is a tissue cutting used to train the lesion recognition model. Correspondingly, the reflected light image of the tissue cutting sample refers to the reflected light image of the tissue cutting used as the training sample, and the transparent light image of the tissue cutting sample is the transmitted light image of the tissue cutting used as the training sample. Based on this, it can be seen that the method for obtaining the reflected light image and the transmitted light image of the tissue cutting sample can refer to the method for obtaining the reflected light image and the transmitted light image of the tissue cutting.

[0150] In a possible implementation, the present application can obtain at least one first reflected light image, at least one second reflected light image, at least one first transmitted light image, and at least one second transmitted light image of a tissue cutting sample.

[0151] The cut sample image is also an image of the tissue cut sample, but the actual lesion area in the tissue cut is manually marked in the image. Therefore, there are also multiple ways to obtain the cut sample image.

[0152] In a possible implementation, in order to improve the training accuracy, the cut sample image can be a whole slide digital scan (WSI) image of the tissue cut sample, wherein the WSI is an image of the tissue cut sample acquired under a digital scanner (actually an electric microscope structure).

[0153] S602, for each tissue cutting sample, respectively aligning the cutting sample image of the tissue cutting sample with each reflected light image and each transmitted light image of the tissue cutting sample, and combining the alignment results to mark the corresponding actual lesion area in the cutting sample image in the reflected light image and the transmitted light image.

[0154] Among them, different image registrations refer to matching and superimposing different images. It can be understood that by registering the tissue cutting sample image with the reflected light image of the tissue cutting sample, the position area of ​​the actual lesion area in the tissue cutting sample in the reflected light image can be determined, so that the actual lesion area of ​​the tissue cutting sample can be marked in the reflected light image. Similarly, the actual lesion area of ​​the tissue cutting sample can be marked in the transmitted light image.

[0155] S603, for each tissue cutting sample, input at least one reflected light image and at least one transmitted light image with annotated actual lesion area corresponding to the tissue cutting sample into the network model to be trained, and obtain a lesion prediction image of the tissue cutting sample output by the network model.

[0156] Among them, the lesion prediction image of the tissue section sample is marked with the actual lesion area and the predicted lesion area.

[0157] It can be understood that, considering that the resolution of the image captured by the camera device with a photosensitivity range of the short infrared band may be low, when at least one reflected light image of the tissue cut sample includes the first reflected light image and the second reflected light image of the tissue cut sample, the present application can also use the aforementioned first image synthesis model to convert the second reflected light image of the tissue cut sample into a synthesized second reflected light image, and then input the synthesized second reflected light image corresponding to the tissue cut sample into the lesion recognition model to be trained.

[0158] Similarly, in the case where at least one transmitted light image of the tissue cutting sample includes a first transmitted light image and a second transmitted light image of the tissue cutting sample, the second transmitted light image can also be converted into a synthesized second transmitted light image using the second image synthesis model to obtain a second transmitted light image with a relatively high resolution. Accordingly, the synthesized second transmitted light image corresponding to the tissue cutting sample can be input into the lesion recognition model to be trained.

[0159] S604, based on the actual lesion area and the predicted lesion area in the lesion prediction image of each tissue cutting sample, determine whether the prediction accuracy of the network model meets the training requirements. If yes, determine the trained network model as the lesion recognition model and the training ends; if not, adjust the internal parameters of the network model and return to step S603 until the prediction accuracy of the network model meets the training requirements.

[0160] There are many possible ways to determine whether the prediction accuracy of the network model meets the training requirements. For example, based on the actual lesion area and the predicted lesion area in the lesion prediction image of each tissue cutting sample, the accuracy of the network model predicting the lesion area is calculated; for example, the similarity between the actual lesion area and the predicted lesion area is calculated, and the average of the similarities of each tissue cutting sample is used as the accuracy of the network model predicting the lesion area. Among them, the similarity between the actual lesion area and the predicted lesion area mainly focuses on whether the actual lesion area and the predicted lesion area are in the same position area in the lesion prediction image. The similarity between the two areas can be calculated in combination with any image similarity algorithm, such as the cosine similarity algorithm or the histogram algorithm, etc., and there is no restriction on this. Correspondingly, if the accuracy of the network model predicting the lesion area exceeds the set threshold, it is determined that the training requirements are met.

[0161] For another example, the loss function value may be calculated based on the set loss function and in combination with the actual lesion area and the prediction model area in the lesion prediction image of each tissue cut sample. If the loss function value reaches convergence, it is confirmed that the training requirements are met. Of course, there may be other possibilities for whether the prediction accuracy of the network model meets the training requirements, and there is no restriction on this.

[0162] It should be noted that Figure 6 This is only one implementation method of training the lesion recognition model in the present application. Training the lesion recognition model by other methods is also applicable to the present application.

[0163] Corresponding to an image processing method of the present application, the present application also provides an image processing device.

[0164] like Figure 7 As shown, it shows a schematic diagram of the composition structure of an embodiment of an image processing device of the present application, and the device may include:

[0165] A first image acquisition unit 701 is used to acquire at least one reflected light image of a tissue block, wherein the reflected light image includes an image of the tissue block after the reflected light of a first light source is converted by an imaging device, and the first light source and the imaging device are located on the same side of the tissue block;

[0166] A second image acquisition unit 702 is used to acquire at least one transmitted light image of the tissue cut block, wherein the transmitted light image includes an image of the tissue cut block after the transmitted light transmitted by the second light source is converted by the imaging device, wherein the second light source and the imaging device are located on different sides of the tissue cut block;

[0167] The lesion determination unit 703 is used to determine the lesion area in the tissue block based on the learned light absorption characteristics of the diseased tissue and the light absorption characteristics at different positions in the tissue block represented by at least one reflected light image and at least one transmitted light image of the tissue block.

[0168] In a possible implementation, the lesion determination unit includes:

[0169] A lesion model processing unit is used to determine the lesion area in the tissue cut according to at least one reflected light image and at least one transmitted light image of the tissue cut and using a lesion recognition model; wherein the lesion recognition model is obtained through deep learning training using at least one reflected light image and at least one transmitted light image of each of a plurality of tissue cut samples marked with actual lesion areas, and the lesion recognition model has learned the light absorption characteristics of the diseased tissue.

[0170] As an optional method, the lesion model processing unit is specifically used to input at least one reflected light image and at least one transmitted light image of the tissue block into a lesion recognition model to obtain a lesion prediction image of the tissue block output by the lesion recognition model, wherein the lesion prediction image is an image of the tissue block, and the predicted lesion area in the tissue block is marked in the lesion prediction image.

[0171] In another possible implementation, the first image acquisition unit is specifically used to obtain at least one first reflected light image and at least one second reflected light image of the tissue cut, wherein the first reflected light image includes an image of the reflected light of the tissue cut to the first light source after being converted by a first camera device, and the second reflected light image includes an image of the reflected light of the tissue cut to the first light source after being converted by a second camera device, the first camera device is an image device whose photosensitive range includes a visible light spectrum, and the second camera device is an image device whose photosensitive range includes a short-wave infrared band spectrum, and the first camera device, the second camera device and the first light source are located on the same side of the tissue cut;

[0172] The second image acquisition unit is specifically used to obtain at least one first transmitted light image and at least one second transmitted light image of the tissue cut, the first transmitted light image includes an image of the tissue cut after the transmitted light transmitted by the second light source is converted by the first camera device, and the second transmitted light image includes an image of the tissue cut after the transmitted light transmitted by the second light source is converted by the second camera device, and the first camera device and the second camera device are both located on different sides of the tissue cut with the second light source.

[0173] In an optional manner, the device further includes:

[0174] a first resolution enhancement unit, configured to, before the lesion recognition model determines the lesion area in the tissue segment, for each second reflected light image of the tissue segment, convert the second reflected light image into a synthesized second reflected light image according to the second reflected light image and the first reflected light image of the tissue segment and using the first image synthesis model;

[0175] The synthesized second reflected light image has the same content as the second reflected light image before the synthesis, and the resolution of the synthesized second reflected light image is higher than the resolution of the second reflected light image before the synthesis;

[0176] The first image synthesis model is based on the high-resolution reflected light images corresponding to each of the multiple tissue cutting samples, and is trained using the first reflected light images and the second reflected light images of each of the multiple tissue cutting samples. The high-resolution reflected light image is a reflected light image within the short infrared band corresponding to the tissue cutting sample, and the resolution of the high-resolution reflected light image is higher than the resolution of the second reflected light image of the tissue cutting sample.

[0177] In yet another optional embodiment, the device may further include:

[0178] A second resolution enhancement unit is used for, before the lesion recognition model determines the lesion area in the tissue segment, for each second transmitted light image of the tissue segment, converting the second transmitted light image into a synthesized second transmitted light image based on the second transmitted light image and the first transmitted light image of the tissue segment and using a second image synthesis model;

[0179] The synthesized second transmitted light image has the same content as the second transmitted light image before the synthesis, and the resolution of the synthesized second transmitted light image is higher than the resolution of the second transmitted light image before the synthesis;

[0180] The second image synthesis model is based on the high-resolution transmitted light images corresponding to each of the multiple tissue cutting samples, and is trained using the first transmitted light images and the second transmitted light images of each of the multiple tissue cutting samples. The high-resolution transmitted light image is a transmitted light image within the short infrared band corresponding to the tissue cutting sample, and the resolution of the high-resolution transmitted light image is higher than the resolution of the second transmitted light image of the tissue cutting sample.

[0181] In another possible implementation, the device further includes: a recognition model training unit, configured to train the lesion recognition model in the following manner:

[0182] Obtaining at least one reflected light image and at least one transmitted light image of each of a plurality of tissue cutting samples, wherein each tissue cutting sample corresponds to a cutting sample image with an actual lesion area marked thereon;

[0183] For each tissue cutting sample, the cutting sample image of the tissue cutting sample is registered with each reflected light image and each transmitted light image of the tissue cutting sample, and based on the registration results, the corresponding actual lesion area in the cutting sample image is marked in the reflected light image and the transmitted light image respectively;

[0184] For each tissue section sample, at least one reflected light image and at least one transmitted light image corresponding to the tissue section sample and marked with an actual lesion area are input into the network model to be trained, and a lesion prediction image of the tissue section output by the network model is obtained, wherein the lesion prediction image is marked with an actual lesion area and a predicted lesion area;

[0185] Based on the actual lesion area and the predicted lesion area in the lesion prediction image of each tissue section sample, it is judged whether the prediction accuracy of the network model meets the training requirements;

[0186] If the prediction accuracy of the network model meets the training requirements, the trained network model is determined as the lesion recognition model;

[0187] If the prediction accuracy of the network model does not meet the training requirements, after adjusting the internal parameters of the network model, the operation of inputting at least one reflected light image and at least one transmitted light image corresponding to the tissue section sample and marked with the actual lesion area into the network model to be trained is returned to the training until the prediction accuracy of the network model meets the training requirements.

[0188] In another aspect, the present application also provides a computer device, which may be a personal computer, a server, a server in a server cluster, or a node in a cloud platform in an image processing system. Figure 8 , which shows a schematic diagram of the composition architecture of a computer device provided by the present application. Figure 8 In the embodiment, the computer device 800 may include: a processor 801 and a memory 802.

[0189] Optionally, the computer device may further include: a communication interface 803 , an input unit 804 , a display 805 and a communication bus 806 .

[0190] The processor 801 , the memory 802 , the communication interface 803 , the input unit 804 and the display 805 all communicate with each other via the communication bus 806 .

[0191] In the embodiment of the present application, the processor 801 may be a central processing unit, a specific application integrated circuit, etc.

[0192] The processor may call the program stored in the memory 802. Specifically, the processor may execute the operations executed by the computer device in the image analysis system in the above embodiment.

[0193] The memory 802 is used to store one or more programs. The program may include program code, and the program code includes computer operation instructions. In the embodiment of the present application, the memory at least stores a program for implementing the image processing method in any of the above embodiments.

[0194] In one possible implementation, the memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, the programs mentioned above, and application programs required for image processing functions, etc.; the data storage area may store data created during the use of the computer device.

[0195] The communication interface 803 may be an interface of a communication module.

[0196] The present application may further include an input unit 804, which may include a touch sensing unit, a keyboard, and the like.

[0197] The display 805 includes a display panel, such as a touch display panel.

[0198] certainly, Figure 8 The computer device structure shown does not constitute a limitation on the computer device in the embodiment of the present application. In actual applications, the computer device may include Figure 8 More or fewer components than shown, or combinations of certain components.

[0199] On the other hand, the present application further provides a storage medium storing computer executable instructions, which, when loaded and executed by a processor, implement the image processing method in any of the above embodiments.

[0200] The present application also proposes a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations of the above-mentioned image processing method or image processing device. The specific implementation process can refer to the description of the above-mentioned corresponding embodiment, and will not be repeated here.

[0201] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. At the same time, the features recorded in each embodiment in this specification can be replaced or combined with each other, so that professionals in the field can implement or use this application. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0202] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0203] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0204] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: include: Obtaining at least one reflected light image of the tissue cut, the reflected light image comprising an image of the tissue cut after the reflected light of the first light source is converted by an imaging device, wherein the first light source and the imaging device are located on the same side of the tissue cut; Obtaining at least one transmitted light image of the tissue cut, the transmitted light image comprising an image of the tissue cut converted by the transmitted light transmitted by the second light source through the imaging device, wherein the second light source and the imaging device are located on different sides of the tissue cut; Determine a lesion area in the tissue section according to the learned light absorption characteristics of the diseased tissue and the light absorption characteristics at different positions in the tissue section represented by at least one reflected light image and at least one transmitted light image of the tissue section; Wherein, obtaining at least one reflected light image of the tissue section comprises: Obtaining at least one first reflected light image and at least one second reflected light image of a tissue cut, wherein the first reflected light image includes an image of the reflected light of the tissue cut to a first light source converted by a first imaging device, and the second reflected light image includes an image of the reflected light of the tissue cut to the first light source converted by a second imaging device, the first imaging device is an imaging device whose photosensitive range includes a visible light spectrum, the second imaging device is an imaging device whose photosensitive range includes a short-wave infrared band spectrum, and the first imaging device, the second imaging device and the first light source are located on the same side of the tissue cut; The step of obtaining at least one transmitted light image of the tissue section comprises: At least one first transmitted light image and at least one second transmitted light image of a tissue cut are obtained, wherein the first transmitted light image includes an image of the tissue cut after being converted by the first imaging device through the transmitted light transmitted by the second light source, and the second transmitted light image includes an image of the tissue cut after being converted by the second imaging device through the transmitted light transmitted by the second light source, and the first imaging device and the second imaging device are both located on different sides of the tissue cut as compared with the second light source.

2. The method according to claim 1, characterized in that: The determining of the lesion area in the tissue block based on the learned light absorption characteristics of the diseased tissue and the light absorption characteristics at different positions in the tissue block represented by at least one reflected light image and at least one transmitted light image of the tissue block, comprises: Determining a lesion area in the tissue section based on at least one reflected light image and at least one transmitted light image of the tissue section and using a lesion recognition model; Among them, the lesion recognition model is obtained through deep learning training using at least one reflected light image and at least one transmitted light image of each of multiple tissue cutting samples marked with actual lesion areas. The lesion recognition model has learned the light absorption characteristics of the diseased tissue.

3. The method according to claim 2, characterized in that The determining of the lesion area in the tissue section based on at least one reflected light image and at least one transmitted light image of the tissue section and using a lesion recognition model comprises: At least one reflected light image and at least one transmitted light image of the tissue block are input into a lesion recognition model to obtain a lesion prediction image of the tissue block output by the lesion recognition model. The lesion prediction image is an image of the tissue block, and the predicted lesion area in the tissue block is marked in the lesion prediction image.

4. The method according to claim 1, characterized in that: Before determining the lesion area in the tissue block, the method further includes: For each second reflected light image of the tissue section, according to the second reflected light image and the first reflected light image of the tissue section, and using the first image synthesis model, converting the second reflected light image into a synthesized second reflected light image; The synthesized second reflected light image has the same content as the second reflected light image before the synthesis, and the resolution of the synthesized second reflected light image is higher than the resolution of the second reflected light image before the synthesis; The first image synthesis model is based on the high-resolution reflected light images corresponding to each of the multiple tissue cutting samples, and is trained using the first reflected light images and the second reflected light images of each of the multiple tissue cutting samples. The high-resolution reflected light images are reflected light images within the short infrared band corresponding to the tissue cutting samples, and the resolution of the high-resolution reflected light image is higher than the resolution of the second reflected light image of the tissue cutting samples.

5. The method according to claim 1, characterized in that Before determining the lesion area in the tissue block, the method further includes: For each second transmitted light image of the tissue section, according to the second transmitted light image and the first transmitted light image of the tissue section, and using a second image synthesis model, converting the second transmitted light image into a synthesized second transmitted light image; The synthesized second transmitted light image has the same content as the second transmitted light image before the synthesis, and the resolution of the synthesized second transmitted light image is higher than the resolution of the second transmitted light image before the synthesis; The second image synthesis model is based on the high-resolution transmitted light images corresponding to each of the multiple tissue cutting samples, and is trained using the first transmitted light images and the second transmitted light images of each of the multiple tissue cutting samples. The high-resolution transmitted light image is a transmitted light image within the short infrared band corresponding to the tissue cutting sample, and the resolution of the high-resolution transmitted light image is higher than the resolution of the second transmitted light image of the tissue cutting sample.

6. The method according to claim 2 or 3, characterized in that: The lesion recognition model is trained in the following way: Obtaining at least one reflected light image and at least one transmitted light image of each of a plurality of tissue cutting samples, wherein each tissue cutting sample corresponds to a cutting sample image with an actual lesion area marked thereon; For each tissue cutting sample, respectively registering the cutting sample image of the tissue cutting sample with each reflected light image and each transmitted light image of the tissue cutting sample, and marking the corresponding actual lesion area in the cutting sample image in the reflected light image and the transmitted light image in combination with the registration results; For each tissue cutting sample, at least one reflected light image and at least one transmitted light image corresponding to the tissue cutting sample and marked with the actual lesion area are input into the network model to be trained, so as to obtain a lesion prediction image of the tissue cutting output by the network model, wherein the actual lesion area and the predicted lesion area are marked in the lesion prediction image; Based on the actual lesion area and the predicted lesion area in the lesion prediction image of each tissue section sample, it is judged whether the prediction accuracy of the network model meets the training requirements; If the prediction accuracy of the network model meets the training requirements, the trained network model is determined as a lesion recognition model; If the prediction accuracy of the network model does not meet the training requirements, after adjusting the internal parameters of the network model, return to the operation of inputting at least one reflected light image and at least one transmitted light image corresponding to the tissue section sample and marked with the actual lesion area into the network model to be trained until the prediction accuracy of the network model meets the training requirements.

7. An image processing device, characterized in that: include: A first image acquisition unit, configured to acquire at least one reflected light image of a tissue cut, wherein the reflected light image comprises an image of the tissue cut converted by a camera device to reflected light of a first light source, wherein the first light source and the camera device are located on the same side of the tissue cut; A second image acquisition unit, configured to acquire at least one transmitted light image of the tissue cut block, wherein the transmitted light image comprises an image of the tissue cut block after the transmitted light transmitted by the second light source is converted by the imaging device, wherein the second light source and the imaging device are located on different sides of the tissue cut block; a lesion determination unit, configured to determine a lesion area in the tissue cut block according to the learned light absorption characteristics of the diseased tissue and the light absorption characteristics at different positions in the tissue cut block represented by at least one reflected light image and at least one transmitted light image of the tissue cut block; Wherein, the first image acquisition unit is specifically used for: Obtaining at least one first reflected light image and at least one second reflected light image of a tissue cut, wherein the first reflected light image includes an image of the reflected light of the tissue cut to a first light source converted by a first imaging device, and the second reflected light image includes an image of the reflected light of the tissue cut to the first light source converted by a second imaging device, the first imaging device is an imaging device whose photosensitive range includes a visible light spectrum, the second imaging device is an imaging device whose photosensitive range includes a short-wave infrared band spectrum, and the first imaging device, the second imaging device and the first light source are located on the same side of the tissue cut; The second image acquisition unit is specifically used for: At least one first transmitted light image and at least one second transmitted light image of a tissue cut are obtained, wherein the first transmitted light image includes an image of the tissue cut after being converted by the first imaging device through the transmitted light transmitted by the second light source, and the second transmitted light image includes an image of the tissue cut after being converted by the second imaging device through the transmitted light transmitted by the second light source, and the first imaging device and the second imaging device are both located on different sides of the tissue cut as compared with the second light source.

8. The device according to claim 7, characterized in that The lesion determination unit comprises: A lesion model processing unit is used to determine the lesion area in the tissue section based on at least one reflected light image and at least one transmitted light image of the tissue section and using a lesion recognition model; wherein the lesion recognition model is obtained through deep learning training using at least one reflected light image and at least one transmitted light image of each of a plurality of tissue section samples marked with actual lesion areas, and the lesion recognition model has learned the light absorption characteristics of the diseased tissue.

9. The device according to claim 7, characterized in that The device also includes: a first resolution enhancement unit, configured to, before determining the lesion area in the tissue segment, for each second reflected light image of the tissue segment, convert the second reflected light image into a synthesized second reflected light image according to the second reflected light image and the first reflected light image of the tissue segment and using a first image synthesis model; The synthesized second reflected light image has the same content as the second reflected light image before the synthesis, and the resolution of the synthesized second reflected light image is higher than the resolution of the second reflected light image before the synthesis; The first image synthesis model is based on the high-resolution reflected light images corresponding to each of the multiple tissue cutting samples, and is trained using the first reflected light images and the second reflected light images of each of the multiple tissue cutting samples. The high-resolution reflected light images are reflected light images within the short infrared band corresponding to the tissue cutting samples, and the resolution of the high-resolution reflected light image is higher than the resolution of the second reflected light image of the tissue cutting samples.

10. The device according to claim 7, characterized in that The device also includes: a second resolution enhancement unit, for converting each second transmitted light image of the tissue segment into a synthesized second transmitted light image according to the second transmitted light image and the first transmitted light image of the tissue segment and using a second image synthesis model before determining the lesion area in the tissue segment; The synthesized second transmitted light image has the same content as the second transmitted light image before the synthesis, and the resolution of the synthesized second transmitted light image is higher than the resolution of the second transmitted light image before the synthesis; The second image synthesis model is based on the high-resolution transmitted light images corresponding to each of the multiple tissue cutting samples, and is trained using the first transmitted light images and the second transmitted light images of each of the multiple tissue cutting samples. The high-resolution transmitted light image is a transmitted light image within the short infrared band corresponding to the tissue cutting sample, and the resolution of the high-resolution transmitted light image is higher than the resolution of the second transmitted light image of the tissue cutting sample.

11. A computer device, characterized in that: including memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the image processing method according to any one of claims 1 to 6.

12. A storage medium, characterized in that: Used to store a program, which, when executed, is used to implement the image processing method as claimed in any one of claims 1 to 6.

13. A computer program product, characterized in that The computer program product includes computer instructions, and a processor of a computer device executes the computer instructions, so that the computer device performs the image processing method according to any one of claims 1 to 6.

Images