Slice modeling method and device, computer equipment and storage medium

Abstract

The invention provides a slice modeling method. The method comprises the following steps: acquiring a bright field image of a target slice; if the target modeling strategy is bright field segmentation, inputting the bright field image into a segmentation model to obtain a first segmentation mask, performing range indentation on the first segmentation mask to obtain a first indentation mask, and determining modeling point distribution according to the first indentation mask; if the target modeling strategy is dark field segmentation, acquiring a plurality of dark field sub-images corresponding to the bright field image, and combining the dark field sub-images to obtain a dark field image; the dark field sub-images are obtained by sequentially scanning slice areas corresponding to the bright field image by using an objective lens with higher power; inputting the dark field image into the segmentation model to obtain a second segmentation mask, and determining modeling point distribution according to the second segmentation mask; and modeling according to the modeling point distribution. On the basis, a user can autonomously select an optimal scheme according to sample complexity, precision requirements and aging requirements, and compared with a traditional modeling method, the modeling quality and efficiency can be taken into consideration more flexibly.

Description

Technical Field

[0001] The present application relates to the field of microscopic experimental technology, and in particular to a slice modeling method, device, computer equipment and storage medium. Background Art

[0002] Efficient modeling and imaging quality improvement of fluorescent stained sections have always been technical difficulties. Using traditional algorithms, it is easy to miss areas of interest or to have extra areas of interest. The segmentation scheme introduced to improve the scanning quality will lead to a decrease in overall efficiency, making it difficult to achieve efficient modeling. Summary of the invention

[0003] The main purpose of the present invention is to overcome the above-mentioned defects of the detection method in the prior art and to propose a slice modeling method, device, computer equipment and storage medium, which can greatly improve the efficiency of slice modeling.

[0004] In a first aspect, the present application provides a slice modeling method, comprising:

[0005] Acquire a bright-field image of the target slice;

[0006] Determine the target modeling strategy;

[0007] If the target modeling strategy is bright field segmentation, the bright field image is input into the segmentation model to obtain a first segmentation mask, the first segmentation mask is range-indented to obtain a first indentation mask, and then the modeling point distribution is determined according to the first indentation mask;

[0008] If the target modeling strategy is dark field segmentation, multiple dark field sub-images corresponding to the bright field image are obtained, and the dark field sub-images are combined to obtain a dark field image; the dark field sub-images are obtained by sequentially scanning the slice areas corresponding to the bright field image using a higher magnification objective lens;

[0009] Inputting the dark field image into the segmentation model to obtain a second segmentation mask, and determining the distribution of modeling points according to the second segmentation mask;

[0010] Modeling is performed based on the distribution of modeling points.

[0011] In one embodiment, determining the distribution of modeling points according to the first indentation mask includes:

[0012] Determine a concentrated distribution area according to the center of the first indentation mask;

[0013] Modeling points with a higher density than other areas are evenly distributed in the concentrated distribution area of ​​the first indentation mask to obtain a modeling point distribution.

[0014] In one embodiment, performing range indentation on the first segmentation mask to obtain a first indentation mask includes:

[0015] Determine the coordinates of the center point of the first segmentation mask;

[0016] Determine the coordinates of the vertices of the rectangle according to the coordinates of the center point and the preset offset; the coordinates of the vertices of the rectangle are located inside the first segmentation mask;

[0017] Get the first indentation mask according to the coordinates of the rectangle vertices.

[0018] In one embodiment, determining the concentrated distribution area according to the center of the first indentation mask includes:

[0019] Determining a target size of the concentrated distribution area according to a diagonal length of the first indentation mask and a first ratio;

[0020] Determine the concentrated distribution area based on the target size and set shape.

[0021] In one embodiment, uniformly distributing modeling points with a higher density than other areas in a concentrated distribution area of ​​the first indentation mask to obtain a modeling point distribution includes:

[0022] Determine the second ratio according to the ratio of the area of ​​the concentrated distribution area to the area of ​​other areas;

[0023] The number of modeling points in the concentrated distribution area and other areas are determined respectively according to the second ratio.

[0024] In one embodiment, obtaining a plurality of dark-field sub-images corresponding to a bright-field image and combining the dark-field sub-images to obtain a dark-field image includes:

[0025] Determine the scanning area based on the bright field image;

[0026] Controlling the microscope to switch to a higher magnification objective lens, and scanning line by line according to the scanning area to obtain multiple dark field sub-images;

[0027] According to the coordinate mapping of each dark field sub-image in the scanning area, each dark field sub-image is filled into the scanning area to obtain a dark field image.

[0028] In one embodiment, before filling each dark field sub-image into the scanning area according to the coordinate mapping of each dark field sub-image in the scanning area to obtain the dark field image, the method further includes:

[0029] determining brightness differences between adjacent dark field sub-images;

[0030] If the brightness difference is greater than the first threshold, brightness compensation is performed on adjacent dark field sub-images respectively to make the brightness difference less than the first threshold.

[0031] In one of the embodiments, the segmentation model is built based on the U-Net architecture.

[0032] In one embodiment, there is a mark on the target slice for highlighting the area where the tissue to be analyzed is located, and the training process of the segmentation model includes:

[0033] Obtaining a bright field training image and a dark field training image respectively to obtain a segmentation training set; the annotation of the bright field training image is used to mark the position of the marker, and the annotation of the dark field training image is used to mark the position of the tissue to be analyzed;

[0034] The initial model is trained using the segmentation training set to obtain a segmentation model.

[0035] In a second aspect, the present application provides a slice modeling device, comprising:

[0036] An image acquisition module, used for acquiring a bright field image of a target slice;

[0037] A strategy determination module, used for determining a target modeling strategy;

[0038] A first segmentation module, for inputting the bright field image into the segmentation model to obtain a first segmentation mask if the target modeling strategy is bright field segmentation, and determining the distribution of modeling points according to the first segmentation mask;

[0039] A dark field image generation module is used to obtain a plurality of dark field sub-images corresponding to the bright field image if the target modeling strategy is dark field segmentation, and combine the dark field sub-images to obtain a dark field image; the dark field sub-images are obtained by sequentially scanning the slice areas corresponding to the bright field image using a higher magnification objective lens;

[0040] A second segmentation module is used to input the dark field image into the segmentation model to obtain a second segmentation mask, and determine the distribution of modeling points according to the second segmentation mask;

[0041] The modeling module is used to perform modeling based on the distribution of modeling points.

[0042] In a third aspect, the present application provides a computer device comprising one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the steps of the slice modeling method in any of the above embodiments are executed.

[0043] In a fourth aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the slice modeling method in any of the above embodiments.

[0044] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0045] The slice modeling method provided in this application achieves a balanced optimization of quality and speed while improving modeling efficiency and accuracy by flexibly selecting the dual strategies of bright field segmentation and dark field segmentation. For the bright field segmentation strategy, after quickly acquiring a large-viewing field bright field image through a low-power objective lens, a segmentation model is used to generate a first segmentation mask, and the invalid area at the edge of the sliced ​​tissue is effectively eliminated by combining the boundary range indentation technology. For scenes with high-precision requirements, the dark field segmentation strategy obtains multiple dark field sub-images through high-power objective scanning and reorganizes them into high-resolution dark field images, and uses the scattered light imaging characteristics to enhance the microstructure characteristics of the sample. Although the dark field image generation time is increased, the efficiency improvement in the modeling stage makes the overall process time change little. Users can independently select the optimal solution according to the sample complexity, accuracy requirements and timeliness requirements. Compared with the traditional modeling method, it can more flexibly take into account the modeling quality and efficiency, and has significant application value in the fields of pathological analysis and cell research. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A schematic diagram of a flow chart of a slice modeling method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a process for obtaining distribution of modeling points in a bright field strategy in one embodiment of the present application;

[0049] Figure 3 A schematic diagram of a process of obtaining a first indentation mask in one embodiment of the present application;

[0050] Figure 4 A schematic diagram of a process for determining a concentrated distribution area in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of a process for determining the number of modeling points in different areas in one embodiment of the present application;

[0052] Figure 6 A schematic diagram of a process for generating a dark field image in one embodiment of the present application;

[0053] Figure 7 This is a schematic diagram of a process for performing brightness compensation on a dark field sub-image in one embodiment of the present application;

[0054] Figure 8This is a comparison diagram of the output results of the segmentation model and the target detection model in one embodiment of the present application;

[0055] Fig. 9 This is a schematic diagram of a process for training a segmentation model in one embodiment of the present application;

[0056] Fig.10 A comparison diagram of a bright field image and a dark field image in one embodiment of the present application;

[0057] Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] The present application provides a slice modeling method, including steps S102 to S110.

[0060] S102, acquiring a bright field image of the target slice.

[0061] It can be understood that the target slice refers to the specific slice sample targeted in the present slice modeling method. It is a sample that has undergone specific processing (such as fixation, slicing, staining, etc.) and is used for subsequent image acquisition and modeling processing. Brightfield images are images acquired through brightfield microscope imaging technology. In brightfield imaging, light directly passes through the sample, and different structures in the sample absorb and scatter light to different degrees, thereby presenting different grayscale or color differences on the image to reflect the structural information of the sample. Specifically, conventional microscope equipment can be used to place the prepared target slice on the stage of the microscope, and adjust the parameters such as the focal length and illumination intensity of the microscope to make the target slice clearly imaged. Then, the image acquisition device equipped with the microscope (such as a CCD camera or a CMOS camera) is used to acquire the image of the target slice, and the optical image is converted into a digital image and stored in the computer.

[0062] S104, determining the target modeling strategy.

[0063] It can be understood that the target modeling strategy refers to the specific way and method used when modeling the target slice, including different strategy selections such as bright field segmentation and dark field segmentation, which determines the subsequent processing method and process of the bright field image. After acquiring the bright field image, it is necessary to select an appropriate modeling strategy based on specific needs and sample characteristics. Specifically, it can be manually selected by the operator based on the initial observation of the target slice and the research purpose.

[0064] S106: If the target modeling strategy is bright field segmentation, the bright field image is input into the segmentation model to obtain a first segmentation mask, the first segmentation mask is range-indented to obtain a first indentation mask, and then the modeling point distribution is determined according to the first indentation mask.

[0065] It can be understood that the microscope uses a low-power objective lens when collecting bright field images, which has a larger field of view, can complete the collection without scanning, has high imaging efficiency, and the bright field segmentation strategy can directly segment the image. When the target modeling strategy is bright field segmentation, the bright field image is input into the segmentation model. The segmentation model is a neural network model trained to segment the stained slice tissue area, which can output a corresponding mask for the input image, but when the input image is a bright field image, the first segmentation mask it outputs corresponds to the slice tissue area marked by a marker (such as a marker) in the image. The first segmentation mask is essentially an image data of the same size as the bright field image, usually presented in the form of a binary image or a multi-channel image. In the case of a binary image, the value of each pixel (usually 0 or 1) represents whether the point belongs to a specific structure or object; in a multi-channel image, the combination of pixel values ​​of different channels can represent different categories. Based on this, after the bright field image of the cell slice is segmented by the segmentation model, the first segmentation mask can clearly identify the cell area and the background area.

[0066] Although the acquisition of bright field images is very efficient, the results of segmentation based on bright field images will contain many areas that are not related to the slice tissue. When laying out the modeling points, some invalid modeling points will be produced, which increases the time consumption of modeling. For this reason, it is necessary to indent the range of the first segmentation mask, which means that at the boundaries of each area identified by the first segmentation mask, the area is contracted inward according to certain rules and degrees, so that the original area range becomes smaller. The purpose of this operation is to further refine the segmented area to highlight the core part of the area. When modeling the slice, all modeling points are distributed in the first indentation mask. Therefore, the distribution of modeling points can be determined according to the first indentation mask and the set distribution rules. Since the first indentation mask is more accurate than the result range obtained by the traditional bright field segmentation modeling method, the number of invalid modeling points can be reduced, and the overall modeling efficiency of bright field segmentation modeling is improved.

[0067] S108, if the target modeling strategy is dark field segmentation, a plurality of dark field sub-images corresponding to the bright field image are obtained, and the dark field sub-images are combined to obtain a dark field image. The dark field sub-images are obtained by sequentially scanning the slice areas corresponding to the bright field image using a higher magnification objective lens.

[0068] S110, inputting the dark field image into the segmentation model to obtain a second segmentation mask, and determining the distribution of modeling points according to the second segmentation mask.

[0069] It can be understood that the dark field image is formed by combining multiple dark field sub-images. The dark field sub-image uses a strong and narrow light beam to illuminate the specimen without allowing the light beam to directly enter the objective lens. However, the particles in the specimen can scatter light, and some of these scattered light rays enter the objective lens, so on a dark background, the particles in the specimen can also be seen at the flash point, and each dark field sub-image is collected using a higher magnification objective lens than when collecting bright field images, which can provide more detailed structural information than bright field images. The dark field image is composed of a combination of multiple dark field sub-images, and the dark field sub-image needs to be scanned and collected multiple times, so its generation takes a certain amount of time, but the dark field segmentation strategy can directly segment the actual position of the sample in the fluorescence imaging, greatly increasing the proportion of effective modeling points, reducing the time-consuming modeling, thereby neutralizing the time cost required to construct the dark field image, thereby improving the overall efficiency. Therefore, the two modeling strategies in this application can take into account both efficiency and quality at the same time, and users can choose bright field or dark field strategies as needed.

[0070] Specifically, after scanning to obtain multiple dark field sub-images, the coordinate mapping relationship between them and the bright field image is combined into a dark field image. The dark field image will be input into the segmentation model for region division. The segmentation model is trained using bright field images and dark field images respectively, and it has the ability to perform segmentation under these two strategies. The difference is that during training, the annotation of the bright field image only annotates its tissue slices.

[0071] S112, modeling is performed according to the distribution of modeling points.

[0072] It can be understood that after the modeling distribution points are determined, feature points are collected according to the corresponding positions of the arranged modeling points, and a model is constructed according to the collected features, so that the structural features of the target slice can be expressed in the form of a mathematical model, a two-dimensional model, or a three-dimensional model. By reasonably utilizing the distribution of modeling points, a model that accurately reflects the sample structure can be constructed to achieve modeling analysis of the target slice.

[0073] The slice modeling method provided in this application achieves a balanced optimization of quality and speed while improving modeling efficiency and accuracy by flexibly selecting the dual strategies of bright field segmentation and dark field segmentation. For the bright field segmentation strategy, after quickly acquiring a large-viewing field bright field image through a low-power objective lens, a segmentation model is used to generate a first segmentation mask, and the invalid area at the edge of the sliced ​​tissue is effectively eliminated by combining the boundary range indentation technology. For scenes with high-precision requirements, the dark field segmentation strategy obtains multiple dark field sub-images through high-power objective scanning and reorganizes them into high-resolution dark field images, and uses the scattered light imaging characteristics to enhance the microstructure characteristics of the sample. Although the dark field image generation time is increased, the efficiency improvement in the modeling stage makes the overall process time change little. Users can independently select the optimal solution according to the sample complexity, accuracy requirements and timeliness requirements. Compared with the traditional modeling method, it can more flexibly take into account the modeling quality and efficiency, and has significant application value in the fields of pathological analysis and cell research.

[0074] In one embodiment, see Figure 2 , determining the distribution of modeling points according to the first segmentation mask, including step S202 and step S204.

[0075] S202: Determine a concentrated distribution area according to the center of the first indentation mask.

[0076] It can be understood that the concentrated distribution area refers to a specific area determined according to the center of the first indentation mask. Considering that the marks drawn by the bright field image using a marker pen are larger than the actual boundary of the slice tissue, and the slice tissue is closer to the center of the mark. Therefore, by setting the center of the concentrated distribution area according to the center of the first indentation mask (such as setting them to be concentric), it can be ensured that more modeling points can fall in the core area of ​​the slice tissue, and the effective proportion of modeling points can be increased, thereby improving accuracy and efficiency. For each area in the first indentation mask, the traditional geometric center calculation method can be used. For example, for a simple rectangular area, its center can be determined by calculating the intersection of the diagonal lines of the rectangle; for an irregularly shaped area, the pixel coordinate information of the image can be used to calculate the average value of the coordinates of all pixel points to obtain the center of the area. In specific implementation, a programming language (such as Python) can be used in combination with an image processing library (such as OpenCV) to achieve it.

[0077] S204: uniformly distribute modeling points with a higher density than other areas in the concentrated distribution area of ​​the first indentation mask to obtain a modeling point distribution.

[0078] It can be understood that the area where the first indentation mask is located is divided into a concentrated distribution area and other areas, and a higher density of modeling points in the concentrated distribution area means that there are more modeling points per unit area in the area. Since the concentrated distribution area has been determined as an area containing important features of the target structure, increasing the density of modeling points in this area can capture and express these features in more detail. Setting relatively few modeling points in other areas can not only ensure basic coverage of the entire target slice structure, but also avoid excessive redundancy of modeling points, thereby improving the efficiency and accuracy of modeling. This distribution of modeling points can make the constructed model have higher accuracy in key parts, while maintaining a reasonable complexity as a whole, and better reflect the structural characteristics of the target slice.

[0079] In one embodiment, see Figure 3 , performing range indentation on the first segmentation mask to obtain a first indentation mask, including steps S302 to S306.

[0080] S302: Determine the coordinates of the center point of the first segmentation mask.

[0081] It can be understood that determining the coordinates of the center point of the first segmentation mask is a basic step for performing the range indentation operation. The center point has certain representativeness and symmetry in the entire mask area, and using the center point as a reference can more conveniently and evenly perform range indentation on the mask.

[0082] S304, determining the coordinates of the vertices of the rectangle according to the coordinates of the center point and the preset offset. The coordinates of the vertices of the rectangle are located inside the first segmentation mask.

[0083] It can be understood that the preset offset is a preset value used to control the size of the rectangular area constructed with the center point as the reference. It represents the distance offset from the center point in all directions (horizontally and vertically). By adjusting the size of the preset offset, the size of the rectangular area can be changed, thereby achieving different degrees of range indentation of the first segmentation mask. When the center point coordinates and the preset offset are determined, the position coordinates of the four vertices of the rectangle in the image coordinate system can be determined based on the preset offset, with the center point of the first segmentation mask as the reference, and usually expressed in the form of (x1, y1), (x2, y2), (x3, y3), (x4, y4). These vertex coordinates determine a rectangular area located inside the first segmentation mask, which will be used to obtain the first indentation mask later. This rectangular area will serve as the boundary of the mask after the range is indented, so that the original first segmentation mask is retained in the rectangular area, and the part beyond the area is removed, achieving the effect of range indentation. In the entire range indentation operation, this step plays a key role in area definition. Reasonable determination of the coordinates of the rectangle vertices can ensure that the mask after range indentation accurately reflects the core part of the target structure while avoiding excessive removal of useful information.

[0084] S306, obtaining a first indentation mask according to the coordinates of the rectangle vertices.

[0085] In one embodiment, see Figure 4 , determining the concentrated distribution area according to the center of the first indentation mask, including step S402 and step S404.

[0086] S402: Determine a target size of the concentrated distribution area according to the diagonal length of the first indentation mask and the first ratio.

[0087] It can be understood that the first indentation mask is a rectangle, and the diagonal length can reflect the size and range of the area. It is an important geometric feature quantity, which is used for the subsequent calculation of the target size of the concentrated distribution area. The first ratio is a preset value, ranging from 0 to 1. It is used to control the degree of scaling of the target size of the concentrated distribution area based on the diagonal length of the first indentation mask. By adjusting the value of the first ratio, the size of the concentrated distribution area can be changed to adapt to different modeling requirements and characteristics of the target slice. The target size refers to the size measurement of the concentrated distribution area calculated based on the diagonal length of the first indentation mask and the first ratio.

[0088] The first indentation mask is already the result of optimizing the original segmentation mask, and its diagonal length reflects the range of the core part of the target structure to a certain extent. By multiplying the diagonal length by the first ratio, a relatively reasonable size value can be obtained as the target size of the concentrated distribution area. The reason for doing this is that the size of the concentrated distribution area is determined by using the inherent characteristics of the first indentation mask, which can make the concentrated distribution area closely related to the core part of the target structure, ensuring that the modeling points set in the area can more accurately reflect the key features of the target structure.

[0089] S404, determining a concentrated distribution area according to the target size and the set shape.

[0090] It can be understood that the set shape is a pre-specified geometric shape of the concentrated distribution area. Common shapes include square, rectangle, regular hexagon, regular octagon, circle, etc. The set shape determines the outline and boundary form of the concentrated distribution area. Different shapes are suitable for different target structure characteristics and modeling requirements. Selecting a suitable set shape helps to more accurately include the key parts of the target structure so that the modeling points can be reasonably distributed in the area later.

[0091] This step is based on the determination of the target size and combined with the set shape to clarify the specific position and outline of the concentrated distribution area. The target size provides information about the size of the area, while the set shape specifies the geometric form of the area. By applying the target size to the set shape, a specific area, namely the concentrated distribution area, can be determined in the first indentation mask image. For example, if the set shape is a square and the target size determines the value of the side length, then a square with a side length of the target size can be drawn in the image as a concentrated distribution area based on the center of the first indentation mask. The concentrated distribution area determined in this way can reasonably delineate the range that needs to be focused on and set the modeling points according to the characteristics of the target structure and the modeling requirements.

[0092] In one embodiment, see Figure 5 , uniformly distributing modeling points with a higher density than other areas in the concentrated distribution area of ​​the first indentation mask to obtain a modeling point distribution, including step S502 and step S504.

[0093] S502: Determine a second ratio according to the ratio of the area of ​​the concentrated distribution area to the area of ​​other areas.

[0094] S504: Determine the number of modeling points in the concentrated distribution area and other areas respectively according to the second ratio.

[0095] It can be understood that the second ratio is a value calculated based on the ratio of the area of ​​the concentrated distribution area to the area of ​​other areas. It is used to reflect the relative relationship between the concentrated distribution area and other areas in terms of area. The concentrated distribution area is set as the part containing the key features of the target structure, and more modeling points are required to accurately describe its features; while other areas contribute less to the key features of the target structure, and the number of modeling points required is relatively small. By calculating the ratio of the area of ​​the concentrated distribution area to the area of ​​other areas to obtain the second ratio, this relative relationship can be quantified. Since the density of modeling points in the concentrated distribution area needs to be greater, the number of modeling points in the concentrated distribution area should be more than that in other areas under the same area. After the second ratio is determined, the number of modeling points in the two parts at the same density can be determined based on the total number of modeling points and the second ratio. At this time, the number of modeling points in the concentrated distribution area obtained is its lower limit value, and more modeling points than this lower limit value need to be allocated to it during allocation.

[0096] In one embodiment, see Figure 6 , obtaining multiple dark-field sub-images corresponding to the bright-field image, and combining the dark-field sub-images to obtain a dark-field image, including steps S602 to S606.

[0097] S602, determining a scanning area according to the bright field image.

[0098] It can be understood that the scanning area is determined based on the boundary of the bright field image, and the dark field image is obtained to obtain more detailed sample information than the bright field image. Determining the scanning area is a key step in the early stage of obtaining the dark field image. The image boundaries of the bright field image and the dark field image are the same, and each dark field sub-image is smaller than the bright field image, so it is necessary to determine the boundary when scanning and collecting the dark field sub-image based on the bright field image, that is, the scanning area.

[0099] S604, controlling the microscope to switch to a higher magnification objective lens, and performing line-by-line scanning according to the scanning area to obtain a plurality of dark field sub-images.

[0100] It can be understood that the microscope here can be the same device as that used to obtain the bright field image, or it can be another one. However, it needs to have a dark field scanning and acquisition function. Compared with the objective lens used for bright field imaging, the objective lens used for collecting dark field sub-images has a higher magnification. Using a higher magnification objective lens can capture more subtle structural features in the sample, thereby obtaining a dark field sub-image that is more refined than the bright field image. However, the field of view of a higher magnification objective lens is relatively small and cannot cover the entire scanning area at one time, so the scanning area needs to be scanned line by line. During the line-by-line scanning process, each scan will acquire a small sub-image until the entire scanning area is covered. The scanning area can be divided into multiple small sub-areas by line-by-line scanning, and the image corresponding to each sub-area is the dark field sub-image. When scanning, it is best to ensure that the collected dark field sub-images have a certain degree of overlap to facilitate subsequent image stitching.

[0101] The switching of the objective lens can be done by the operator manually operating the objective lens switching button of the microscope to switch the objective lens to a higher magnification objective lens. Then, the stage control knob of the microscope is used to move the stage in sequence in the order of line-by-line scanning, so that different sub-areas of the scanning area are in the field of view of the objective lens in turn, and an image acquisition device (such as a CCD camera) is used to capture the image of each sub-area to obtain a dark field sub-image. However, this method requires the operator to have certain operating experience and is relatively inefficient, so it is best to use the automatic control software that comes with the microscope to achieve automatic switching of the objective lens and line-by-line scanning operations by writing scripts or setting parameters. The scanning step size is determined by the size of the scanning area and the dark field sub-image, as long as the obtained dark field sub-image can cover the entire scanning area.

[0102] S606 , filling each dark field sub-image into the scanning area according to the coordinate mapping of each dark field sub-image in the scanning area to obtain a dark field image.

[0103] It can be understood that the coordinate mapping reflects the position information of each dark field sub-image in the scanning area, usually expressed in the form of pixel coordinates. During the line-by-line scanning process, the coordinates of a certain point in the scanning area corresponding to each dark field sub-image are recorded, such as the center point, vertex, etc. This coordinate information is the coordinate mapping. The exact position of each dark field sub-image in the final dark field image can be determined by coordinate mapping. After obtaining multiple dark field sub-images through line-by-line scanning, these sub-images need to be combined into a complete dark field image. The coordinate mapping provides each dark field sub-image with position information in the scanning area. Based on this information, each dark field sub-image can be accurately placed at the corresponding position in the scanning area to achieve image splicing and filling. During the filling process, it is generally necessary to process the overlapping parts between the sub-images to ensure that the spliced ​​dark field image is seamless and continuous. The dark field image obtained in this way can reflect the complete fine structure of the sample in the scanning area.

[0104] Specifically, a blank image of the same size as the scanning area can be created as the basis of the dark field image. Each dark field sub-image is traversed, and the pixel values ​​of the dark field sub-image are copied to the corresponding positions of the dark field image according to its coordinate mapping information. For the overlapping parts between sub-images, a simple covering or averaging processing method can be used. Feature extraction and matching algorithms (such as SIFT, SURF, etc.) can also be used to process the overlapping parts between sub-images. First, feature points are extracted in each dark field sub-image, and then feature matching is performed between adjacent sub-images to find the corresponding relationship between them. Based on the results of feature matching, the transformation matrix (such as affine transformation matrix) between sub-images is calculated, the sub-images are transformed and aligned, and then spliced. This method can more accurately process the overlapping parts between sub-images and improve the accuracy of splicing.

[0105] In one embodiment, see Figure 7 Before filling each dark field sub-image into the scanning area according to the coordinate mapping of each dark field sub-image in the scanning area to obtain the dark field image, it also includes steps S702 to S704.

[0106] S702, determining the brightness difference between adjacent dark field sub-images.

[0107] It can be understood that in the process of acquiring multiple dark field sub-images by scanning line by line according to the scanning area, two dark field sub-images adjacent in position. For example, when scanning line by line, adjacent sub-images in the same row or the same column belong to adjacent dark field sub-images. It is used to measure the degree of brightness difference between adjacent dark field sub-images. In an image, brightness can usually be represented by the grayscale value of a pixel (for a grayscale image), or reflected by a numerical combination of color channels (such as each channel in the RGB color space) (for a color image). The brightness difference can be determined by calculating the difference in brightness values ​​of pixels at corresponding positions of adjacent sub-images. Common calculation methods include average brightness difference, mean square error, etc.

[0108] In the process of acquiring dark field images, due to the influence of various factors that may exist during the scanning process (such as the stability of the microscope light source, the difference in light transmittance at different positions of the sample, etc.), there may be inconsistent brightness between adjacent dark field sub-images. If this brightness difference is not handled, when the dark field sub-images are combined into a dark field image, it will cause obvious seams or uneven brightness in the spliced ​​image, affecting the quality of the dark field image and the subsequent analysis and processing of the image. Therefore, before performing image filling and splicing, first determine the brightness difference between adjacent dark field sub-images so that corresponding measures can be taken to make adjustments based on the difference.

[0109] S704: If the brightness difference is greater than the first threshold, brightness compensation is performed on adjacent dark field sub-images respectively to make the brightness difference less than the first threshold.

[0110] It can be understood that the first threshold is a pre-set value, which is used as a standard to determine whether the brightness difference between adjacent dark field sub-images needs to be adjusted. When the calculated brightness difference is greater than the first threshold, it means that the brightness difference between adjacent sub-images is large and brightness compensation is required; when the brightness difference is less than or equal to the first threshold, it is considered that the brightness difference is within an acceptable range and no compensation is required. Brightness compensation refers to adjusting adjacent dark field sub-images with inconsistent brightness to make their brightness closer. Common brightness compensation methods include linear transformation, histogram equalization, etc., which change the brightness of the image by adjusting the pixel value of the image, thereby reducing the brightness difference between adjacent sub-images. It can be to reduce the brightness of the high-brightness image alone, or to increase the brightness of the low-brightness image alone, or to adjust both at the same time.

[0111] In one embodiment, the segmentation model is built based on the U-Net architecture. The U-Net architecture is a semantic segmentation model architecture based on a convolutional neural network (CNN). It consists of a contraction path (downsampling path) and an expansion path (upsampling path). In the contraction path, the image resolution is gradually reduced through continuous convolution and pooling operations to extract the semantic information of the image; in the expansion path, the image resolution is restored through upsampling and feature fusion operations of the corresponding layers of the contraction path, so that the segmentation result can be accurate to the pixel level. The U-Net architecture is widely used in fields such as medical image segmentation and can effectively segment and classify different areas in an image. Please refer to Figure 8 , the left side of the figure shows the detection result obtained by the target detection model, and the right side shows the result obtained by the segmentation model in this embodiment. Compared with the traditional target detection model, the mask output by the Unet architecture can be irregular, further cropping the area not related to the mark.

[0112] In one embodiment, there is a mark on the target slice for highlighting the area where the tissue to be analyzed is located. For details, please refer to Figure 8 The black ink portion of the image delineates the area where the tissue is located. Fig. 9 , the training process of the segmentation model includes steps S902 to S904.

[0113] S902, respectively obtain a bright field training image and a dark field training image to obtain a segmentation training set. The annotation of the bright field training image is used to mark the position of the mark, and the annotation of the dark field training image is used to mark the position of the tissue to be analyzed.

[0114] It can be understood that the bright field training image is obtained by further annotating the slice image obtained by bright field microscope imaging technology, while the dark field training image is obtained by further annotating the slice image obtained by dark field microscope imaging technology. Fig.10 The difference between the two lies in the different annotation methods. In the bright field image, the marks on the slices are clearly visible, but the difference between the slice tissue and other positions on the slide is not so obvious, so the marks on the slices are used as the basis for segmentation. In the dark field image, the contrast between the slice tissue and the background is higher, and the slice tissue is more prominent. The tissue contour can be used directly as the basis for segmentation. Based on this, the annotation of the bright field training image will identify the location of the above marks, while the annotation of the dark field training image will focus on outlining the contours of the slice tissue. Through this differentiated annotation, the segmentation model can more accurately identify and distinguish slice features under different imaging conditions, thereby learning the ability to segment dark field images and bright field images respectively.

[0115] S904: Train the initial model using the segmentation training set to obtain a segmentation model.

[0116] The initial model is trained with the segmentation training set, which allows the initial model to adjust its own parameters by learning the images and annotation information in the training set, so that it has the ability to accurately segment the target slice image. During the training process, the initial model takes the bright field training image and the dark field training image as input, and extracts the image features through operations such as convolution, pooling, and upsampling within the model. Then, the output of the model is compared with the annotation information, and the loss function (such as the cross entropy loss function, etc.) is calculated to measure the difference between the model prediction result and the true annotation. According to the value of the loss function, the optimization algorithm (such as stochastic gradient descent, Adam optimizer, etc.) is used to adjust the parameters of the model so that the loss function gradually decreases and the model prediction result is closer and closer to the true annotation. Through continuous iterative training, the initial model gradually learns the marked features in the bright field image and the features of the tissue to be analyzed in the dark field image, and finally obtains a segmentation model that can accurately segment the target slice image.

[0117] The present application provides a slice modeling device, including an image acquisition module, a strategy determination module, a first segmentation model and a dark field image generation module.

[0118] The image acquisition module is used to acquire a bright field image of the target slice. The strategy determination module is used to determine the target modeling strategy. The first segmentation module is used to input the bright field image into the segmentation model to obtain a first segmentation mask if the target modeling strategy is bright field segmentation, and determine the distribution of modeling points according to the first segmentation mask. The dark field image generation module is used to acquire multiple dark field sub-images corresponding to the bright field image if the target modeling strategy is dark field segmentation, and combine the dark field sub-images to obtain a dark field image. The dark field sub-image is obtained by sequentially scanning the slice area corresponding to the bright field image using a higher power objective lens. The second segmentation module is used to input the dark field image into the segmentation model to obtain a second segmentation mask, and determine the distribution of modeling points according to the second segmentation mask. The modeling module is used to perform modeling according to the distribution of modeling points.

[0119] For the specific limitations of the slicing modeling device, please refer to the limitations of the slicing modeling method above, which will not be repeated here. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0120] In a third aspect, the present application provides a computer device comprising one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the steps of the slice modeling method in any of the above embodiments are executed.

[0121] Indicatively, Fig.11 As shown, Fig.11 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. Fig.11 The computer device 1100 includes a processing component 1102, which further includes one or more processors, and a memory resource represented by a memory 1101 for storing instructions executable by the processing component 1102, such as an application. The application stored in the memory 1101 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1102 is configured to execute instructions to perform the text recognition method of any of the above embodiments.

[0122] The computer device 1100 may further include a power supply component 1103 configured to perform power management of the computer device 1100, a wired or wireless network interface 1104 configured to connect the computer device 1100 to a network, and an input / output (I / O) interface 1105. The computer device 1100 may operate based on an operating system stored in the memory 1101, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0123] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0124] In a fourth aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the slice modeling method in any of the above embodiments.

[0125] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0126] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0127] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A slice modeling method, characterized in that: include: Acquire a bright-field image of the target slice; Determine the target modeling strategy; If the target modeling strategy is bright field segmentation, the bright field image is input into a segmentation model to obtain a first segmentation mask, the first segmentation mask is range-indented to obtain a first indentation mask, and then the modeling point distribution is determined according to the first indentation mask; If the target modeling strategy is dark field segmentation, a plurality of dark field sub-images corresponding to the bright field image are obtained, and the dark field sub-images are combined to obtain a dark field image; the dark field sub-images are obtained by sequentially scanning the slice areas corresponding to the bright field image using a higher magnification objective lens; Inputting the dark field image into the segmentation model to obtain a second segmentation mask, and determining the modeling point distribution according to the second segmentation mask; Modeling is performed according to the modeling point distribution.

2. The slicing modeling method according to claim 1, characterized in that: The determining of the distribution of modeling points according to the first indentation mask comprises: Determining a concentrated distribution area according to the center of the first indentation mask; Modeling points with a higher density than other areas are evenly distributed in the concentrated distribution area of ​​the first indentation mask to obtain the modeling point distribution.

3. The slicing modeling method according to claim 2, characterized in that: The step of performing range indentation on the first segmentation mask to obtain a first indentation mask includes: Determining the coordinates of the center point of the first segmentation mask; Determine the coordinates of the vertices of the rectangle according to the coordinates of the center point and the preset offset; the coordinates of the vertices of the rectangle are located inside the first segmentation mask; The first indentation mask is obtained according to the coordinates of the rectangle vertices.

4. The slicing modeling method according to claim 3, characterized in that: The determining of the concentrated distribution area according to the center of the first indentation mask includes: Determining a target size of the concentrated distribution area according to a diagonal length of the first indentation mask and a first ratio; The concentrated distribution area is determined according to the target size and the set shape.

5. The slicing modeling method according to claim 4, characterized in that: The step of uniformly distributing modeling points with a higher density than other areas in the concentrated distribution area of ​​the first indentation mask to obtain the modeling point distribution includes: Determining a second ratio according to a ratio of an area of ​​the concentrated distribution region to an area of ​​the other regions; The number of modeling points in the concentrated distribution area and the number of modeling points in the other areas are determined respectively according to the second ratio.

6. The slicing modeling method according to claim 1, characterized in that: The step of acquiring a plurality of dark-field sub-images corresponding to the bright-field image and combining the dark-field sub-images to obtain a dark-field image comprises: determining a scanning area according to the bright field image; Controlling the microscope to switch to a higher magnification objective lens, and performing line-by-line scanning according to the scanning area to obtain a plurality of the dark field sub-images; According to the coordinate mapping of each dark field sub-image in the scanning area, each dark field sub-image is filled into the scanning area to obtain the dark field image.

7. The slicing modeling method according to claim 6, characterized in that: Before filling each dark field sub-image into the scanning area according to the coordinate mapping of each dark field sub-image in the scanning area to obtain the dark field image, the method further includes: Determining the brightness difference between adjacent dark field sub-images; If the brightness difference is greater than a first threshold, brightness compensation is performed on adjacent dark field sub-images respectively to make the brightness difference less than the first threshold.

8. The slicing modeling method according to claim 1, characterized in that: The segmentation model is built based on the U-Net architecture.

9. The slicing modeling method according to claim 8, characterized in that: The target slice has a mark for highlighting the area where the tissue to be analyzed is located. The training process of the segmentation model includes: Acquire a bright field training image and a dark field training image respectively to obtain a segmentation training set; the annotation of the bright field training image is used to mark the position of the mark, and the annotation of the dark field training image is used to mark the position of the tissue to be analyzed; The initial model is trained using the segmentation training set to obtain the segmentation model.

10. A slice modeling device, characterized in that: include: An image acquisition module, used for acquiring a bright field image of a target slice; A strategy determination module, used for determining a target modeling strategy; A first segmentation module, configured to input the bright field image into a segmentation model to obtain a first segmentation mask if the target modeling strategy is bright field segmentation, perform range indentation on the first segmentation mask to obtain a first indentation mask, and then determine a modeling point distribution according to the first indentation mask; A dark field image generation module, for obtaining a plurality of dark field sub-images corresponding to the bright field image and combining the dark field sub-images to obtain a dark field image if the target modeling strategy is dark field segmentation; the dark field sub-images are obtained by sequentially scanning the slice areas corresponding to the bright field image using a higher magnification objective lens; A second segmentation module, used for inputting the dark field image into the segmentation model to obtain a second segmentation mask, and determining the modeling point distribution according to the second segmentation mask; A modeling module is used to perform modeling according to the distribution of modeling points.

11. A computer device, characterized in that: The invention comprises one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the slice modeling method according to any one of claims 1 to 9 are executed.

12. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the slice modeling method according to any one of claims 1 to 9.

Images