A method for separating a staining signal, an electronic device, and a medium

Abstract

The present invention discloses a method for separating staining signals. First, the image to be separated is inverted to obtain a preprocessed image. Then, the gray values of each pixel point in the preprocessed image in the red, green, and blue channels are determined to obtain the vector expression of each pixel point in the C_RGB coordinate system, where the X, Y, and Z axes of the C_RGB coordinate system represent the red, green, and blue channels respectively. Finally, the first and second projection values of the vector expression of each pixel point on the first and second signal basis vectors are calculated in the coordinate system composed of the first and second signal basis vectors and the third direction perpendicular to the plane where they are located. Based on the first and second projection values, the separated first and second signals are obtained. This separation method is simple to operate, does not depend on the nuclear-cytoplasmic distribution of antibody signals, and can prevent interference of human factors on the results.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis and processing, and particularly relates to a method for separating staining signals, an electronic device, and a medium. Background Art

[0002] Immunohistochemistry (IHC) is a commonly used technique for detecting tissue proteins. It can detect the expression of specific proteins or antigens in tissue samples and is widely used in clinical and basic research fields. The basic principle of IHC is to specifically recognize the target antigen through an antibody, and then display the position and expression level of the antigen through the color development of the labeled antibody.

[0003] Typical IHC images contain two parts of information: the target protein signal and the nucleus signal. Among them, the target protein signal reflects the abundance of the target protein, which is usually generated by the color development of diaminobenzidine (DAB) catalyzed by horseradish peroxidase conjugated to an antibody, and it is brown. The nucleus signal serves as a structural marker to assist in signal interpretation and is obtained by hematoxylin staining, which is blue. Before performing bioinformatics analysis or computer-based signal intensity evaluation, it is usually necessary to separate the two signals to accurately achieve the quantitative analysis of the antibody signal.

[0004] Currently, the separation of IHC image signals is usually achieved through color separation. For example, by utilizing the different composition ratios of the two signals in the RGB channels, through certain formulas, the signals in different channels can be subtracted from each other to obtain a separate signal reference value; for example, using bright-field images of the RGB three channels to calculate a standardized "BN" image representing the signal intensity of the antibody. Another example is that the quantitative analysis of the antibody signal can be achieved through deconvolution. Specifically, according to the absorption values of the DAB and hematoxylin single-stained signals in the RGB three channels, the DAB signal value of each pixel is calculated. Some open-source image analysis software, such as ImageJ / Fiji, etc., adopt this method. Another example is that AI-assisted signal splitting can also be used. For example, tools such as QuPath are used to identify DAB and hematoxylin signals through machine learning. Specifically, deep learning is performed based on the labeled staining separation images to perform staining separation on the staining images, or clustering algorithms are used to perform staining separation on the staining images, etc. The foregoing methods usually require the staining images to be in some standard test conditions, with harsh separation conditions, and there are many other types of staining and impurities in the obtained staining channel images, resulting in unclear separation of each staining channel, a large amount of overlapping information, and many artifacts. In addition, the deep learning scheme requires effective annotation of pathological section staining images, increasing the need for manual extraction of staining image features, and has a limited scope of application. Summary of the Invention

[0005] In view of some or all of the problems in the prior art, a first aspect of the present invention provides a method for separating staining signals, including:

[0006] Inverting the image to be separated to obtain a preprocessed image;

[0007] Determining the gray values of each pixel point in the preprocessed image in the red, green, and blue channels to obtain the vector representation of each pixel point in the C_RGB coordinate system, where the X, Y, and Z axes of the C_RGB coordinate system represent the red, green, and blue channels respectively; and

[0008] Calculating the first and second projection values of the vector representation of each pixel point on the first and second signal basis vectors, and obtaining the first and second signals based on the first and second projection values.

[0009] Further, the first signal basis vector is the vector representation of the first single-staining signal in the C_RGB coordinate system, and the second signal basis vector is the vector representation of the second single-staining signal in the C_RGB coordinate system.

[0010] Further, the first and second signal basis vectors are determined according to the average value of the single-staining region of the image to be separated.

[0011] Further, the first and second signal basis vectors are determined according to the average value of the single-staining samples.

[0012] Further, the first and second signal basis vectors are determined according to historical data.

[0013] Further, the separation method further includes:

[0014] Calculating the absolute value of the projection of the vector representation of each pixel point in the third direction to confirm the signal residual, where the third direction is perpendicular to the plane formed by the first and second signal basis vectors.

[0015] Further, the image to be separated is an immunohistochemical image, where the first signal is the target protein signal of brown color produced by the catalysis of diaminobenzidine by horseradish peroxidase conjugated with an antibody, and the second signal is the blue nucleus signal obtained by hematoxylin staining.

[0016] Based on the above-mentioned method for separating staining signals, a second aspect of the present invention provides an electronic device for separating staining signals, which includes a memory and a processor, where the memory is configured to store a computer program, and the computer program executes the above-mentioned separation method when running on the processor.

[0017] The third aspect of the present invention further provides a computer-readable storage medium for separating staining signals, which stores a computer program. When the computer program runs on a processor, it executes the separation method as described above.

[0018] A method for separating staining signals provided by the present invention only requires simple negation and coordinate system transformation operations, and does not depend on the nuclear-cytoplasmic distribution of antibody signals. Therefore, complex parameter settings are not required, the overall process is simple, and at the same time, interference from human factors to the results can be prevented. The coordinate system transformation operation can be realized through matrix operations in a computer, and the calculation efficiency is high. The signals separated by the separation method are grayscale images, and the grayscale values of the pixels in the images directly reflect the strength of the corresponding signals, which can be directly used for subsequent bioinformatics processing flows to achieve quantitative analysis of the signals. In addition, the separation method can directly obtain a residual image to reflect the quality of image signal splitting for quality control. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To further clarify the above and other advantages and features of the embodiments of the present invention, a more specific description of the embodiments of the present invention will be presented with reference to the accompanying drawings. It can be understood that these drawings only depict typical embodiments of the present invention and will not be considered as limiting its scope. In the drawings, for clarity, the same or corresponding components will be denoted by the same or similar reference numerals.

[0020] Figure 1 A schematic flowchart of a method for separating staining signals according to an embodiment of the present invention;

[0021] Figure 2 A schematic diagram showing the principle of coordinate transformation according to an embodiment of the present invention;

[0022] Figure 3 A schematic diagram showing the process of signal separation of an IHC image using a method for separating staining signals according to an embodiment of the present invention;

[0023] Figure 4 and 5 Schematic diagrams showing the results of signal separation of two IHC images with different signal coverage degrees using a method for separating staining signals according to an embodiment of the present invention;

[0024] Figure 6 A schematic flowchart showing the quantitative analysis according to an embodiment of the present invention;

[0025] Figure 7 and 8 Schematic diagrams showing the statistical results of the single-cell expression levels of EGFR and Ki67 in three tissue chip samples respectively; and

[0026] Figure 9The figure shows a schematic diagram of the user interface of the IHC data processing software according to an embodiment of the present invention. Detailed implementation manners

[0027] In the following description, the present invention is described with reference to the embodiments. However, those skilled in the art will recognize that the embodiments can be implemented without one or more specific details or in combination with other alternative and / or additional methods or components. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring the inventive points of the present invention. Similarly, for the purpose of explanation, specific numbers and configurations are set forth to provide a comprehensive understanding of the embodiments of the present invention. However, the present invention is not limited to these specific details.

[0028] In this specification, the reference to "an embodiment" or "the embodiment" means that the specific features, structures, or characteristics described in connection with the embodiment are included in at least one embodiment of the present invention. The phrase "in an embodiment" appearing throughout this specification does not necessarily refer to the same embodiment.

[0029] It should be noted that the embodiments of the present invention describe the method steps in a specific order. However, this is only for the purpose of explaining the specific embodiment and does not limit the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to the actual requirements.

[0030] In order to improve the separation efficiency of the staining signals, the present invention provides a method for separating staining signals, which can achieve the separation of different color signals through simple negation and coordinate system conversion operations.

[0031] The technical solution of the present invention will be further described below with reference to the accompanying drawings of the embodiments.

[0032] Figure 1 The figure shows a schematic flow chart of a method for separating staining signals according to an embodiment of the present invention. As Figure 1 shown, a method for separating staining signals includes:

[0033] First, in step 101, the picture is negated. The picture to be separated is negated to obtain a preprocessed image. After the picture is negated, the gray values of each channel will directly reflect the magnitude of the signal value. The picture negation operation can be understood as replacing the value of each pixel with the corresponding maximum value minus the current value. For example, for an 8-bit image, the pixel value range is between 0 and 255, then the picture negation operation is to replace the current value of each pixel with 255 minus the current value. For an RGB color image, the negation operation can be performed on the red, green, and blue channels respectively;

[0034] Next, in step 102, determine the vector representation of each pixel point in the C_RGB coordinate system. Determine the gray values of each pixel point in the preprocessed image in the three channels of red, green, and blue to obtain the vector representation of each pixel point in the C_RGB coordinate system, where the X, Y, and Z axes of the C_RGB coordinate system represent the red, green, and blue channels respectively. In an embodiment of the present invention, a three-dimensional rectangular coordinate system C_RGB can be first constructed, and its X, Y, and Z axes respectively correspond to the signals of the red, green, and blue channels. Each pixel point on the processed image can be regarded as a vector V signal , and its components V RED , V GREEN , and V Blue on the three coordinate axes are the gray values of the corresponding pixel point in the three channels. The direction of the vector representation is determined by the color of the pixel point, and the length of the vector representation can represent the color depth of the pixel signal and the strength of its signal; and

[0035] Finally, in step 103, signal separation. Calculate the first and second projection values of the vector representation of each pixel point on the first and second signal basis vectors V0 1 , V0 2 . Based on the first and second projection values, the first and second signals can be obtained. The first and second signals are presented in the form of grayscale images and can intuitively reflect the strength of the first and second signals in the image to be separated. In an embodiment of the present invention, the first signal basis vector V0 1 is the vector representation of the first single-stain signal in the C_RGB coordinate system, and the second signal basis vector V0 2 is the vector representation of the second single-stain signal in the C_RGB coordinate system. In an embodiment of the present invention, the first and second signal basis vectors V0 1 , V0 2 are determined according to the average values of the single-stain regions of the image to be separated. Specifically, the average values of the gray values of each pixel point in the red, green, and blue channels of the first and second single-stain regions in the preprocessed image are calculated respectively, and the first and second signal basis vectors V0 1 , V0 2 are obtained according to the calculated average values. In another embodiment of the present invention, the first and second signal basis vectors V0 1 , V0 2 are determined according to the average values of the single-stain samples. Specifically, after performing an inversion operation on the images of the first and second single-stain samples, the average values of the gray values of each pixel point in the red, green, and blue channels are calculated, and the first and second signal basis vectors V0 1 , V0 2 are obtained according to the calculated average values. In yet another embodiment of the present invention, the first and second signal basis vectors V01 and V0 2 Historical data can also be directly adopted.

[0036] This step can be understood as a coordinate system conversion operation. As Figure 2 shown, specifically, taking the first and second signal basis vectors V0 1 and V0 2 as the X and Y axes to establish a new coordinate system C_trans. The Z axis of the new coordinate system is the perpendicular direction V of the plane where the first and second signal basis vectors are located Residual . Then, the vector expression of each pixel point in the preprocessed image in C_RGB is converted to the C_trans coordinate system to obtain a new vector V trans . Then, the component V trans of V on the X axis is the signal value of the first signal, and the component V 1 of V on the Y axis is the signal value of the second signal, thus realizing the separation of the staining signals. At the same time, the component of the vector V 2 on the Z axis is the remaining signal after signal separation. Taking the absolute value is denoted as the residual. The residual can be used to reflect the quality of the picture signal splitting. The smaller the value of the residual, the more complete the signal separation. trans

[0037] The staining signal separation method described above can be applied to signal separation of, for example, IHC pictures. Among them, the IHC pictures need to be separated into the target protein signal of brown color developed by the horseradish peroxidase conjugated with the antibody catalyzing diaminobenzidine (DAB), denoted as the DAB signal, and the blue nucleus signal obtained by hematoxylin staining, denoted as the Hema signal. Figure 3 Fig. shows a schematic diagram of the process of signal separation of IHC pictures using the method described above. As Figure 3 shown, it includes:

[0038] First, invert the picture so that the gray values of each channel can directly reflect the magnitude of the signal value; and

[0039] Next, perform coordinate system conversion to directly obtain the DAB signal, the Hema signal, and the residual. The specific process is as follows:

[0040] Construct a three-dimensional rectangular coordinate system C_RGB, where its X, Y, and Z respectively correspond to the signals of the red, green, and blue channels. In this way, each pixel on the picture can be regarded as a vector V signal in this coordinate system. Its components V RED , V GREEN , and V Blue on the three coordinate axes are the gray values of the pixel in the three channels. In this way, the direction of the vector is determined by the pixel color, and the length of the vector represents the strength of the pixel signal;​

[0041] The average signal value obtained from the single-stained area or single-stained sample, or directly using historical data, determines the two vectors of the single-stained signals of hematoxylin and DAB in the C_RGB coordinate system, which are recorded as V0 and V1 respectively. Hema and V0 DAB . Create a new 3D coordinate system C_trans: X axis is V0 DAB Direction, Y axis is V0 Hema Direction, Z axis is the vertical direction V of the plane where the two are located Residual ;as well as

[0042] Perform coordinate system transformation, convert the original vector into the new coordinate system C_trans, and obtain the new vector V trans , then V trans On the X axis, V0 DAB Direction component V DAB It is the signal value of the DAB signal, on the Y axis, that is, V0 Hema Direction component V Hema It is the signal value of the hematoxylin Hema signal, which completes the signal splitting, the Z axis, that is, V Residual The directional signal is the remaining signal after the split, which is recorded as the residual. Considering that the residual is distributed in both positive and negative directions, the absolute value of the residual is taken.

[0043] In order to verify the applicability of the separation method in IHC images, a variety of IHC images with different signal coverage were selected and the signal separation was performed using the separation method described above. The results are shown in Figure 2. Figure 4 and 5 As shown. Figure 4 The IHC image of EGFR in lung cancer tissue is shown, which is localized in the cytoplasm. Figure 5 The image shown is the IHC image of Ki67 in glioma tissue, which is localized in the cell nucleus. It can be seen that EGFR signals of different intensities can be separated smoothly, but when the DAB signal is too strong, the Hema channel will be affected to a certain extent ( Figure 4 The last line in the figure). Ki67 signals of different intensities can also be separated smoothly. It has been verified that the separation method can be applied to nuclear localization and cytoplasmic localization, and can be applied to the separation of signals of different intensities.

[0044] As mentioned above, after the IHC image is split by the separation method, three pictures can be obtained: a grayscale image of the DAB signal, the Hema signal and the residual, and the grayscale image can be used for quantitative analysis of the IHC results and calculation of the H-Score.

[0045] Figure 6 A schematic diagram of the quantitative analysis process is shown in FIG. Figure 6As shown, after signal separation is completed, first, pixel classification can be performed on the image of the Hema signal to obtain a nuclear probability map. In an embodiment of the present invention, pixel classification can be achieved by using the open-source ilastik. Then, in combination with the image of the DAB signal and the grayscale image of the picture to be separated, single-cell recognition is performed, and the antibody signal of each cell is calculated to obtain the H-Score. For example, cellprofiller can be used to achieve single-cell recognition. Figure 7 and 8 respectively show the statistical results of the single-cell expression levels of EGFR and Ki67 in three tissue microarray samples. Specifically, according to the three set thresholds, the cells are divided into level0, level1, level2, and level3. The distribution of cells at each level is marked on the right tissue picture. According to the percentage of each level, the H-Score result can be calculated, that is, the number in the parentheses in the histogram title in the figure. It should be understood that the embodiments shown in the figure are only for demonstrating the data analysis method. In practical applications, the values of the three thresholds can be adjusted according to specific situations.

[0046] Based on the separation method described above, the present invention further provides an electronic device for separating staining signals, which includes a memory and a processor, wherein the memory is configured to store a computer program, and the computer program executes the separation method described above when running on the processor.

[0047] The present invention further provides a computer-readable storage medium for separating staining signals, which stores a computer program, and the computer program executes the separation method described above when running on a processor. The computer-readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device, such as an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Specifically, the computer-readable storage medium includes, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punched card or a raised structure in a groove storing instructions thereon, and any suitable combination of the above.

[0048] In an embodiment of the present invention, the computer program can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or an external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter or network interface in each computing / processing device receives the computer program from the network and forwards it for storage in the computer-readable storage medium in each computing / processing device.

[0049] In an embodiment of the present invention, the computer program can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages. The programming languages include object-oriented programming languages - such as Smalltalk, C++, etc., and conventional procedural programming languages - such as the "C" language or similar programming languages. The computer program can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or can be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet). In some embodiments, by using the status information of the computer program to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer program.

[0050] In an embodiment of the present invention, the computer program is integrated into the IHC data processing software. The IHC data processing software integrates functions such as whole-slide data preview, region selection, region splitting, IHC signal splitting, and picture export, and provides a user interface for easy user operation. The user interface is as Figure 9As shown, its middle part is the display area, and operation buttons are set on the upper side and the side. When using the IHC data processing software for image processing, first, open the whole-slide scanning data through the "OpenSlide" button. The whole-slide scanning data is in the.mrsx format. Then, manually select the area to be analyzed through the "Manual Gate" button, such as the nucleus area or the cytoplasm area, etc. Next, click the "Split Gate" button to automatically split the selected area into small pieces. Finally, batch complete the signal splitting and image export of each small-piece image through the "Export Images" button. In addition, operations such as selecting image clarity, import and export can also be performed. The IHC data processing software can effectively improve the processing efficiency of IHC whole-slide image data.

[0051] Although the embodiments of the present invention have been described above, it should be understood that they are presented only as examples and not as limitations. It will be apparent to those skilled in the relevant art that various combinations, variations, and changes can be made thereto without departing from the spirit and scope of the present invention. Therefore, the width and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined only by the appended claims and their equivalents.

Claims

1. A method for separating staining signals, characterized in that: Includes steps: Invert the image to be separated to obtain a preprocessed image; Determine the grayscale value of each pixel in the preprocessed image in the red, green and blue channels to obtain a vector expression of each pixel in the C_RGB coordinate system, wherein the X, Y and Z axes of the C_RGB coordinate system represent the red, green and blue channels respectively; A new coordinate system C_trans is established with the first signal basis vector and the second signal basis vector as the X and Y axes, wherein the C_transZ axis of the new coordinate system is the perpendicular direction of the plane where the first signal basis vector and the second signal basis vector are located, wherein the first signal basis vector is the vector expression of the first single-stained signal in the C_RGB coordinate system, and the second signal basis vector is the vector expression of the second single-stained signal in the C_RGB coordinate system; and Convert the vector expression of each pixel in the preprocessed image in the C_RGB coordinate system to the C_trans coordinate system to obtain a new vector V trans , then V trans The X-axis component V of the C_trans coordinate system 1 That is, the signal value of the first signal, the component V of the Y axis in the C_trans coordinate system 2 is the signal value of the second signal.

2. The separation method according to claim 1, characterized in that The first signal basis vector and the second signal basis vector are respectively determined according to average values ​​of a first single-dyed region and a second single-dyed region of the image to be separated.

3. The separation method according to claim 1, characterized in that The first signal basis vector and the second signal basis vector are determined according to the average values ​​of the single-stained samples.

4. The separation method according to claim 1, characterized in that The first signal basis vector and the second signal basis vector are determined according to historical data.

5. The separation method according to claim 1, characterized in that Also includes the steps: Calculate V trans The absolute value of the Z-axis component of the C_trans coordinate system is taken to confirm the residual after signal separation.

6. The separation method according to claim 1, characterized in that The image to be separated is an immunohistochemical image, wherein the first signal is a brown target protein signal generated by antibody-coupled horseradish peroxidase catalyzing diaminobenzidine color development, and the second signal is a blue cell nucleus signal obtained by hematoxylin staining.

7. An electronic device for separating staining signals, characterized in that: The method comprises a memory and a processor, wherein the memory is configured to store a computer program, and the computer program executes the separation method according to any one of claims 1 to 6 when the processor is running.

8. A computer-readable storage medium for separating staining signals, characterized in that: A computer program is stored, and when the computer program is run on a processor, the separation method according to any one of claims 1 to 6 is executed.

Images