A method and device for self-supervised staining normalization of pathological images

Abstract

The present invention discloses a method and device for self-supervised color normalization of pathological images, which belongs to the technical field of combining digital pathology and image processing, and includes: constructing a pathological image database; designing a color normalization model, which includes a common feature domain extraction module, a content sensor, a scene sensor, and a fusion module, and the color normalization model can perform color conversion on the pathological image to obtain a color normalized image; based on the pixel-level difference between the input original pathological image and the color normalized image, a self-supervised loss function is constructed and the color normalization model is trained, and the model can realize adaptive color style normalization of an image of any unknown pathological data center toward an image of a selected pathological data center.

Description

Technical Field

[0001] The invention belongs to the technical field of combining digital pathology and image processing, and in particular relates to a method and device for self-supervised staining normalization of pathological images. Background Art

[0002] The development of digital pathology and the advancement of computer vision algorithms have promoted the application of artificial intelligence technology in the field of pathology. However, due to the privacy protection of medical data, pathology data is usually not centralized, forming a de facto "data island". In this context, training customized intelligent auxiliary models with a single large medical institution as the main body is a direct and safe solution. In the real world, large medical institutions have accumulated rich medical resources and patient data because they have relatively more professional knowledge and experience in pathological analysis and diagnosis. Designing a single-center pathology intelligent auxiliary model with the goal of serving it can avoid sharing multi-center data with developers, maximize service to itself and protect the data privacy security of patients.

[0003] However, different sample preparation procedures and different imaging equipment used among pathology centers will lead to differences in staining styles in the final pathology images. This difference makes the pathology intelligent assisted model trained in a single large medical institution less versatile and difficult to generalize to other medical institutions that did not participate in the training, hindering the sharing and promotion of advanced medical technologies. Therefore, there is an urgent need for a preprocessing method that can achieve staining normalization of images from unknown pathology centers.

[0004] In order to solve the problem of staining differences in pathological images of different medical institutions, the existing staining normalization methods are mainly divided into two categories: (1) traditional staining normalization methods; (2) deep learning-based staining normalization methods. Among them, traditional staining normalization methods mainly rely on mathematical algorithms and image processing techniques, and use the color distribution or statistical characteristics of the image for staining normalization. These methods require the selection of template images from the target staining style dataset in advance, so they are mainly suitable for small-scale datasets and situations where the image staining styles are relatively consistent. Deep learning-based methods are the mainstream technology in the current field of staining normalization. They can automatically learn complex staining style conversion from the data involved in training, and then achieve more accurate staining style normalization, such as the style transfer technology based on generative adversarial networks.

[0005] However, although the above-mentioned type (1) scheme is easy to operate, it usually only involves the normalization of two staining styles and is heavily dependent on the selection of the template image in the target style. The template image usually cannot fully represent the staining characteristics of all samples in a single pathology center, resulting in deviations in the results of staining normalization. The above-mentioned type (2) scheme is usually based on a generative adversarial network, which can normalize a variety of known staining styles, but its training process is complicated, consumes a lot of computing resources, and the generated images may have artifacts or distortions, especially in the processing of high-frequency textures or details of complex tissue structures.

[0006] More importantly, both types of solutions require the participation of target staining style data and the need to centralize the data, which makes it impossible to protect the privacy of patient data on the one hand, and on the other hand, it is impossible to normalize the staining of data from unknown pathology centers that did not participate in the training. This makes it difficult to help more small and medium-sized hospitals use advanced pathology intelligent assisted models developed by a single large medical center without the need for open source patient data. Summary of the invention

[0007] In view of the above, an object of the present invention is to provide a method and device for self-supervised coloring normalization of pathological images, which can realize adaptive coloring style normalization of images of any unknown pathological data center towards images of a selected pathological data center.

[0008] To achieve the above-mentioned purpose of the invention, an embodiment provides a pathological image self-supervised staining normalization method, comprising the following steps:

[0009] Build a pathology image database;

[0010] Designing a dye normalization model and training, which converts a pathological image into a common feature domain image after color randomization, and uses the common feature domain image and the input pathological image as self-pairing to perform self-supervised training of the model to obtain a dye normalization model of a single medical center, wherein the model includes a content sensor, a scene sensor, and a fusion module, wherein the content sensor is used to extract a content feature map based on the common feature domain image, and merge the content feature map with the common feature domain image to obtain a merged map, the scene sensor is used to predict the adaptive weighting parameters of multiple color conversion physical models based on the common feature domain image, the fusion module is used to perform weighted summation of multiple color conversion physical models based on the adaptive weighting parameters to obtain a fusion model, and the fusion model is used to perform color conversion on the merged map to obtain a dye normalized image;

[0011] Verification and application evaluation were carried out based on the dyeing normalization model after parameter optimization.

[0012] Preferably, in the common feature domain extraction module, the color randomization processing includes at least one of brightness, contrast, saturation, and hue random processing. When the pathological image is converted into a common feature domain image after the randomization processing, based on the prior that the structures or histological features of the same type of pathological images across medical centers are similar, the pathological image is converted into a common feature domain image based on the prior.

[0013] Preferably, the content sensor adopts a first lightweight convolutional network, and the scene sensor adopts a second lightweight convolutional network.

[0014] Preferably, when merging the content feature map with the common feature domain image, the number of channels N of the merged image is taken as the target. When the common feature domain image is a one-channel grayscale image and the content feature map is an n-channel image, the common feature domain image is copied Nn times to form an Nn-channel common feature domain image that is merged with the content feature map to form an N-channel merged image, where n<N, and n and N are both natural numbers.

[0015] Preferably, the color conversion physical model uses a learnable color conversion lookup table, and the number of dimensions of the color conversion physical model is equal to the number of channels of the merged image.

[0016] Preferably, when the fusion model performs color conversion on the merged image to obtain a dyed normalized image, the fused N-dimensional color conversion lookup table is used to perform N linear interpolations on the merged image to obtain the dyed normalized image.

[0017] Preferably, the dyeing normalization model after parameter optimization is used for verification and application evaluation, including:

[0018] After using the pathological image database of the first medical center to train the staining normalization model, the pathological images of the second medical center were color normalized using the staining normalization model with optimized parameters. Then, the pathology auxiliary model of the first medical center was used to perform auxiliary diagnosis prediction based on the staining normalized images.

[0019] To achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a pathological image self-supervised staining normalization device, comprising:

[0020] A database construction unit, which is used to construct a pathological image database;

[0021] A model design and training unit, which is used to convert a pathological image into a common feature domain image after color randomization, and to perform self-supervised training of the model using the common feature domain image and the input pathological image as self-pairing to obtain a dye normalization model of a single medical center, wherein the unit includes a content sensor, a scene sensor, and a fusion module, wherein the content sensor is used to extract a content feature map based on the common feature domain image, and merge the content feature map with the common feature domain image to obtain a merged map, wherein the scene sensor is used to predict adaptive weighted parameters of multiple color conversion physical models based on the common feature domain image, wherein the fusion module is used to perform weighted summation of multiple color conversion physical models based on the adaptive weighted parameters to obtain a fusion model, and wherein the fusion model is used to perform color conversion on the merged map to obtain a dye normalized image;

[0022] The model verification and application evaluation unit is used to perform verification and application evaluation using the dyeing normalization model after parameter optimization.

[0023] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned pathological image self-supervised staining normalization method.

[0024] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned pathological image self-supervised staining normalization method is implemented.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention adopts a self-supervised scheme. When training the staining normalization model, no other staining style images except the pathological images of the selected training center are required as templates, which means that data transmission among multiple pathology centers is not involved, effectively protecting the privacy of patient data.

[0027] The self-supervised staining normalization scheme of the present invention can realize the normalization of any unknown staining style toward the selected pathological center staining style, and based on this, further realize the staining normalization of ultra-large-size whole slide images, which is a unique advantage.

[0028] The self-supervised staining normalization scheme of the present invention adopts a lightweight and interpretable model framework design, has a simple training process, consumes little computing resources, and can adaptively complete real-time and efficient staining normalization according to the pathological image block scene and content. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a flow chart of a method for self-supervised staining normalization of pathological images provided in an embodiment;

[0031] Figure 2 is a schematic diagram of constructing a pathological image database provided by an embodiment;

[0032] Figure 3 Schematic diagram of the training process of the dyeing normalization model provided in the embodiment;

[0033] Figure 4 is a schematic diagram of self-supervised pairing of pathological images provided by an embodiment;

[0034] Figure 5 is a schematic diagram of the structure of the dyeing normalization model provided in the embodiment;

[0035] Figure 6 is a schematic diagram of a model-oriented application provided by an embodiment;

[0036] Figure 7 is a schematic diagram of model verification and application evaluation provided by an embodiment;

[0037] Figure 8 Schematic diagram of the structure of the pathological image self-supervised staining normalization device provided in the embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0039] The data of large pathology medical centers usually have consistent collection standards, equipment settings, and pathology diagnosis specifications. The single-center pathology intelligent auxiliary model built on it can assist itself to the greatest extent. However, due to the differences in staining styles of pathology data imaging in different medical centers, the single-center pathology intelligent auxiliary model may not be well applied to pathology images of other hospitals.

[0040] Based on this, the present invention proposes to design an adaptive staining style normalization scheme based on the prior knowledge that the same tumor type in pathological images has similar structural features from the interpretable self-supervised technical level to realize image staining normalization of images of any unknown pathological medical center towards the images of the selected pathological medical center, thereby helping small and medium-sized hospital centers to use the advanced pathology intelligent auxiliary model developed by the large medical center without the need for open source patient data, thereby narrowing the resource gap and promoting medical equity while strictly protecting patient data privacy.

[0041] The training of the dyeing style normalization model in the solution of the present invention only needs to be performed locally based on pathological images in the selected medical center, and there is no need to obtain the original pathological images of other medical centers, thereby ensuring the privacy of patient data in each pathological center. The solution of the present invention avoids the problem of dyeing normalization style deviation easily occurring in the traditional dyeing normalization method of the first category in the background technology due to the selection of template images, the problem of large computational resource consumption and easy artifact generation in the deep learning method of the second category, and the problem that both methods can only implement data dyeing normalization in a limited number of known pathological centers.

[0042] like Figure 1 As shown, the embodiment provides a pathological image self-supervised staining normalization method, comprising the following steps:

[0043] S110, constructing a pathological image database.

[0044] In the embodiment, different medical centers build their own pathology image databases. The data obtained by imaging in a single pathology medical center usually has a consistent staining style, and different medical centers can complete the preparation and imaging of the latest patient pathology staining slides locally. The process is as follows Figure 2 As shown, it mainly includes: pathological tissue extraction, hematoxylin and eosin staining, biomedical microscopic imaging, and construction of a pathological image database with corresponding staining styles in different pathological medical centers. The general steps for implementing hematoxylin and eosin staining include: first, tissue sampling and fixation in formalin buffer to preserve its structure; then gradient ethanol dehydration, xylene transparency, immersion in molten paraffin, slicer slicing, special adhesive patch, xylene dewaxing and hydration in water; finally, hematoxylin is used to dye the cell nucleus into blue-purple, and eosin is used to dye the cytoplasm into pink. The stained slices are dehydrated and transparentized again with gradient ethanol and xylene, and neutral gum is used to seal the slices under a coverslip to complete the sealing. The present invention images the newly prepared stained glass slides on a special microscope in a single medical center, divides the imaging results of the stained glass slides into multiple pathological images, and together with the digital data accumulated by the hospital, constitutes an initial pathological image database of the same style. As an example, the present invention will Figure 2Hospital A of traditional Chinese medicine was selected as the first pathological medical center, and hospital B was regarded as an arbitrary unknown pathological medical center.

[0045] S120, designing a dye normalization model and training, randomizing the color of the pathological image and converting it into a common feature domain image, using the common feature domain image and the input pathological image as self-pairing to perform self-supervised training of the model to obtain a dye normalization model for a single medical center.

[0046] In the embodiment, the design and training of the staining normalization model are performed in a selected pathology medical center. Figure 3 As shown in the figure, the input pathology image is firstly randomized in color, then converted into an image in the public feature domain, and finally the public feature domain image and the input pathology image are used as self-pairings for self-supervised training to obtain a single-center staining normalization model. This model can realize staining normalization of data from the public feature domain to the selected pathology center data style of any unknown pathology center.

[0047] In an embodiment, color randomization is achieved by randomly performing one or more synchronous perturbations on brightness, contrast, saturation, and hue, with the goal of simulating the differences in foreground and background of data from different pathology centers to enhance the adaptability of the self-supervised staining normalization model. Specifically, the operation of converting to a common feature domain in the present invention is based on the prior that the structure or histological features of the same type of pathology images across medical centers are similar. Based on this prior, the actual operation is to convert the input color randomized image to a common feature domain image that can characterize these features, wherein the common feature domain can be a grayscale domain. It should be noted that modifications based on this prior conversion to other common domains that can characterize these features should also be within the scope of protection of the present invention. Examples of conversion of color randomization and common feature domain images are as follows: Figure 4 shown.

[0048] In the embodiment, Figure 5 As shown, the designed color normalization model includes a content sensor, a scene sensor, and a fusion module, wherein the content sensor is used to extract a content feature map based on a common feature domain image, and merge the content feature map with the common feature domain image to obtain a merged map. The content sensor is used to distinguish different colors corresponding to similar content in the pathological image to achieve more accurate self-supervised mapping. Specifically, the image converted to the common feature domain is sent to the content sensor to generate the corresponding content feature map, and then merged with the corresponding common feature domain image to obtain a merged map. When merging specifically, the number of channels N of the merged map is taken as the target. When the common feature domain image is a one-channel grayscale image and the content feature map is n-channel, the common feature domain image is copied Nn times to form a common feature domain image of Nn channels and merged with the content feature map to form an N-channel merged map, wherein n<N, and n and N are both natural numbers.

[0049] For example, when the merged image has 4 channels, there are two merging methods. The first is to adaptively generate a three-channel content feature map by the content sensor and merge it with the corresponding single-channel grayscale image. The second is to adaptively generate a single-channel content feature map by the content sensor and merge it with the corresponding three-channel grayscale image. Both can equivalently achieve the purpose of the present invention.

[0050] In the embodiment, the content sensor adopts the first lightweight convolutional network, which specifically includes four convolution blocks with a kernel size of 3 × 3 and one convolution block with a kernel size of 1 × 1. The stride of the 3 × 3 convolution block is 1 and LeakyReLU activation and instance regularization are applied, wherein the first convolution block and the fourth convolution block are residually connected; the purpose of the first four 3 × 3 convolution blocks is to extract high-level image features of the same resolution from the input common feature domain image, and the 1 × 1 convolution block is used to compress image features and output content feature maps. Note that any modification to the content sensor to achieve the same purpose of the present invention should be within the scope of protection of the present invention.

[0051] In the embodiment, the scene sensor is used to predict the adaptive weighted parameters of multiple color conversion physical models based on the common feature domain image, and the fusion module is used to perform weighted summation on the multiple color conversion physical models based on the adaptive weighted parameters to obtain a fusion model, and the fusion model is used to perform color conversion on the merged image to obtain a dyed normalized image. Specifically, the common feature domain image is simultaneously fed into the scene sensor and the adaptive parameters output by the scene sensor are used to weight multiple identical learnable color conversion physical models in the real world to obtain a fusion model of the conversion merged image; at this time, for each input pathological image, a fusion model corresponding to the conversion merged image can be adaptively obtained, so that the overall design has the advantages of traceability and explainability.

[0052] In the embodiment, the scene perception uses a second lightweight convolutional network, specifically including first using bilinear interpolation to downsample the input image to a resolution of 256 × 256, followed by 5 convolutional blocks, a dropout layer and a fully connected layer; each convolutional block includes a convolutional layer with a kernel size of 3 × 3, a LeakyReLU and an instance normalization; the dropout rate is set to 0.5 to suppress overfitting. Note that any modification to the scene perception to achieve the same purpose of the network architecture in the present invention should be within the scope of protection of the present invention.

[0053] In the embodiment, the number of color conversion physical models is set to 1 at least, 3 by default, and there may be no upper limit (the more the number, the larger the model). If the number of channels of the merged image is N, multiple identical learnable color conversion physical models use a color conversion lookup table that can be operated in N-dimensional (e.g., four-dimensional) space, and the mapping relationship of all elements therein is initialized to an identity mapping. The N-dimensional color conversion lookup table is a fusion model obtained by weighting the parameters output by the scene sensor, that is, the fusion model is also an N-dimensional color conversion lookup table, which is used to convert the merged image of N channels and obtain a stained normalized image through N linear interpolation operations. Since the interpolation operation is differentiable, the N-dimensional color conversion lookup table can update the element mapping relationship therein according to the back-propagated gradient during training, just like the content sensor and the scene sensor.

[0054] The dye-normalized image output by the fusion model is optimized and trained with the original pathological image as a label, and a dye-normalized model is finally obtained for realizing the style of any unknown pathological medical center data to the selected pathological medical center data. The present invention uses the pixel-level difference of the self-paired input original pathological image and the dye-normalized image, such as the mean square error, to construct a self-supervised loss function for optimization training, with the purpose of making the predicted dye-normalized image closer to the original pathological image, which is expressed as:

[0055] ;

[0056] in, represents the self-supervised loss function, where represents the batch size, and Respectively represent The original pathological image and the predicted stained normalized image in the pairing, Note that any modification of selecting other pixel-level loss functions to achieve the same purpose of the present invention should be within the protection scope of the present invention.

[0057] S130, performing dyeing normalization verification and application evaluation using the dyeing normalization model after parameter optimization.

[0058] In the embodiment, the trained dye normalization model is verified and applied in an unknown medical center (Hospital B). First, under the premise of protecting the privacy of patient data, the dye normalization of the present invention is verified on pathological data with style differences but unknown models. Specifically, the present invention implements two levels of dye normalization, namely, adaptive dye normalization of pathological image blocks and dye normalization of pathological whole slide images. The process and results are shown in the following examples. Figure 6As shown. Regarding the verification of the adaptive staining normalization capability of the pathology image block, specifically, although the whole slide images with the same center usually have the same staining style, the image blocks extracted from different tissue regions within the same whole slide image may have color differences. The design of the scene sensor and content sensor of the present invention enables the present invention to not only have the ability to perform staining normalization toward a certain style, but also to achieve adaptive staining normalization based on the scene and content of the pathology image block. Based on this, the present invention can further achieve staining normalization of pathology whole slide images, such as Figure 6 shown.

[0059] Secondly, under the premise of protecting patient data privacy, the single-center pathology intelligent auxiliary model is verified on pathology data with style differences but unknown models. The process is as follows: Figure 7 As shown. First, a single-center pathology intelligent auxiliary model that can identify tumor a image blocks is trained in hospital A. The model can give reasonable auxiliary diagnosis suggestions in hospital A, which is easy to achieve in the field of digital pathology, such as auxiliary tasks such as tumor detection, classification and segmentation; but due to differences in data staining styles, it is usually difficult to apply to the auxiliary diagnosis of the same type of tumors in an unknown hospital B. The present invention can achieve the auxiliary diagnosis of the pathology of the same type of disease in hospital B by using the self-supervised staining normalization model trained in hospital A on the same type of tumor data in hospital B, and then using the pathology intelligent auxiliary model of hospital A to complete the pathology auxiliary diagnosis of the same type of disease in hospital B.

[0060] like Figure 8 As shown, the self-supervised staining normalization 80 of pathological images provided in the embodiment includes a database construction unit 81, a model design and training unit 82, and a model verification and application evaluation unit 83, wherein the database construction unit 81 is used to construct a pathological image database; the model design and training unit 82 is used to convert the pathological image into a common feature domain image after color randomization, and use the common feature domain image and the input pathological image as self-pairing to perform self-supervised training of the model to obtain a staining normalization model of a single medical center; the model verification and application evaluation unit 83 is used to construct a self-supervised loss function based on the pixel-level difference between the input original pathological image and the staining normalized image, train the staining normalization model, and then use the staining normalization model with optimized parameters to perform staining normalization.

[0061] It should be noted that when the pathological image self-supervised coloring normalization device provided in the above embodiment is constructing a human-computer interaction cognitive model, the division of the above-mentioned functional modules or units should be used as an example to illustrate that the above-mentioned functions can be assigned to different functional modules or units as needed, that is, the internal structure of the terminal or server is divided into different functional modules or units to complete all or part of the functions described above. In addition, the pathological image self-supervised coloring normalization device provided in the above embodiment and the pathological image self-supervised coloring normalization method embodiment belong to the same concept, and the specific implementation process is detailed in the pathological image self-supervised coloring normalization method embodiment, which will not be repeated here.

[0062] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein executable code is stored in the memory, and when the one or more processors execute the executable code, the method for self-supervised staining normalization of pathological images is implemented, specifically including the following steps:

[0063] S110, constructing a pathological image database;

[0064] S120, designing a dye normalization model and training, randomizing the color of the pathological image and converting it into a common feature domain image, using the common feature domain image and the input pathological image as self-pairing to perform self-supervised training of the model to obtain a dye normalization model for a single medical center.

[0065] S130, using the dyeing normalization model after parameter optimization for verification and application evaluation.

[0066] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the self-supervised staining normalization method for pathological images described in S110-S130 above. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0067] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned pathological image self-supervised staining normalization method is implemented, which specifically includes the following steps:

[0068] S110, constructing a pathological image database;

[0069] S120, designing a dye normalization model and training, randomizing the color of the pathological image and converting it into a common feature domain image, using the common feature domain image and the input pathological image as self-pairing to perform self-supervised training of the model to obtain a dye normalization model for a single medical center.

[0070] S130, using the dyeing normalization model after parameter optimization for verification and application evaluation.

[0071] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0072] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A self-supervised staining normalization method for pathological images, characterized in that: The following steps are involved: Build a pathology image database; Designing a dye normalization model and training it, which converts the pathological image into a common feature domain image after color randomization, wherein the color randomization processing includes at least one of brightness, contrast, saturation, and hue randomization processing, and when the pathological image is converted into the common feature domain image after the randomization processing, based on the prior that the structure or histological features of the same type of pathological images across medical centers are similar, the pathological image is converted into the common feature domain image based on the prior; A single medical center's dye normalization model is obtained by self-supervising training of a model using a common feature domain image and an input pathological image as self-pairings, wherein the dye normalization model includes a content sensor, a scene sensor, and a fusion module, wherein the content sensor is used to extract a content feature map based on the common feature domain image, and merge the content feature map with the common feature domain image to obtain a merged map, wherein the scene sensor is used to predict adaptive weighted parameters of multiple color conversion physical models based on the common feature domain image, wherein the fusion module is used to perform weighted summation of multiple color conversion physical models based on the adaptive weighted parameters to obtain a fusion model, wherein the fusion model is used to perform color conversion on the merged map to obtain a dye normalized image; wherein the color conversion physical model uses a learnable color conversion lookup table, and the number of dimensions of the color conversion physical model is equal to the number of channels of the merged map; The dyeing normalization model with optimized parameters was used for verification and application evaluation.

2. The method for self-supervised color normalization of pathological images according to claim 1, characterized in that: The content sensor adopts a first lightweight convolutional network, and the scene sensor adopts a second lightweight convolutional network.

3. The pathological image self-supervised staining normalization method according to claim 1, characterized in that: When merging the content feature map with the public feature domain image, the number of channels N of the merged image is taken as the target. When the public feature domain image is a one-channel grayscale image and the content feature map is n-channel, the public feature domain image is copied Nn times to form an Nn-channel public feature domain image that is merged with the content feature map to form an N-channel merged image, where n<N, and n and N are both natural numbers.

4. The method for self-supervised color normalization of pathological images according to claim 1, characterized in that: When the fusion model performs color conversion on the merged image to obtain a dyed normalized image, the fused N-dimensional color conversion lookup table is used to perform N linear interpolations on the merged image to obtain a dyed normalized image.

5. The method for self-supervised color normalization of pathological images according to claim 1, characterized in that: The dye normalization model with optimized parameters was used for verification and application evaluation, including: After using the pathological image database of the first medical center to train the staining normalization model, the pathological images of the second medical center were color normalized using the staining normalization model with optimized parameters. Then, the pathology auxiliary model of the first medical center was used to perform auxiliary diagnosis prediction based on the staining normalized images.

6. A pathological image self-supervised staining normalization device, characterized in that: include: A database construction unit, which is used to construct a pathological image database; A model design and training unit, which is used to convert a pathological image into a common feature domain image after color randomization, wherein the color randomization processing includes at least one of brightness, contrast, saturation, and hue randomization processing, and when the pathological image is converted into the common feature domain image after the randomization processing, based on the prior that the structures or histological features of the same type of pathological images across medical centers are similar, the pathological image is converted into the common feature domain image based on the prior; A single medical center's dye normalization model is obtained by self-supervising training of a model using a common feature domain image and an input pathological image as self-pairings, wherein the model includes a content sensor, a scene sensor, and a fusion module. The content sensor is used to extract a content feature map based on the common feature domain image, and merge the content feature map with the common feature domain image to obtain a merged map. The scene sensor is used to predict adaptive weighting parameters of multiple color conversion physical models based on the common feature domain image. The fusion module is used to perform weighted summation of multiple color conversion physical models based on the adaptive weighting parameters to obtain a fusion model. The fusion model is used to perform color conversion on the merged map to obtain a dye normalized image. The color conversion physical model uses a learnable color conversion lookup table, and the number of dimensions of the color conversion physical model is equal to the number of channels of the merged map. The model verification and application evaluation unit is used to perform verification and application evaluation using the dyeing normalization model after parameter optimization.

7. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the pathological image self-supervised staining normalization method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the pathological image self-supervised staining normalization method described in any one of claims 1 to 5 is implemented.

Images