A real-time image stitching method and system without lighting influence

Abstract

The present invention provides a real-time image stitching method and system without illumination influence, comprising the following steps: obtaining a first prefabricated template and a second prefabricated template; determining a first correction parameter according to the first prefabricated template; determining a second correction parameter according to the second prefabricated template; scanning a pathological slice using a scanner, during the scanning process, each pathological slice image to be stitched obtained by scanning with the scanner can be directly stitched after being corrected by the first correction parameter and the second correction parameter, so that a complete pathological slice image can be obtained by scanning and stitching at the same time during the scanning process; the beneficial effects of the present invention are as follows: the present invention uses the first correction parameter and the second correction parameter to restore the orthographic projection image of the pathological slice image to be stitched and correct the position of the pathological slice image to be stitched, which can maintain the integrity of the image, stitch images containing blank samples, sparse samples or images with very few features, and can also perform stitching while scanning.

Description

Technical Field

[0001] The present invention relates to the technical field of image stitching, and in particular to a real-time image stitching method and system without illumination influence. Background Art

[0002] In digital pathology, in order to obtain a complete pathology slice image, it is usually necessary to stitch multiple high-resolution images. These images often cover different areas of the entire slice, so they must be combined into a complete image through stitching technology. However, since the resolution of pathology images is very high, usually reaching billions of pixels, the amount of calculation in the stitching process is very large and the processing speed is very slow, which seriously affects the efficiency of real-time diagnosis and analysis.

[0003] Defects and shortcomings of existing technologies: 1. Highly dependent on image matching and unable to cope with poor matching: First, the matching algorithm is sensitive to deformation. Since pathological sections may deform during production and scanning, existing spatial domain jigsaw puzzle methods are less robust to these deformations and prone to calculation errors; Second, for samples lacking feature points, it is impossible to calculate a relatively accurate displacement distance and even produce a large offset error. 2. Unable to scan and stitch at the same time: The digital imaging process of the entire slide requires the calculation of the global position of each image. This process requires waiting for all images to be scanned and the displacement deviation to be calculated before global optimization can be performed. Then the machine scanning motion process cannot be imaged, resulting in too slow imaging time. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a real-time image stitching method and system without lighting influence, so as to solve the problem that the prior art is highly dependent on matching between images and thus cannot cope with poor matching and cannot stitch while scanning.

[0005] To achieve the above objectives and other related objectives, the present invention provides the following technical solutions:

[0006] A real-time image stitching method without illumination influence comprises the following steps: obtaining a first prefabricated template and a second prefabricated template, wherein the first prefabricated template and the second prefabricated template are both templates shaped like a chessboard with evenly distributed black and white chessboards, and a uniform, light-transmitting monochrome coating is provided on the white grids of the first prefabricated template, and the thickness of the monochrome coating is 5 μm, because the conventional white grids of the chessboard are transparent grids, and due to the light transmittance of different glass materials, it cannot be guaranteed that there are different brightness distributions when the glass slide is uneven, so the monochrome coating is set so that the light transmittance at different distances has different attenuation conditions; determining a first correction parameter for calculating an orthographic projection image according to the first prefabricated template; determining a second correction parameter for correcting an image position according to the second prefabricated template;

[0007] The pathological slices are scanned using a scanner, and the pathological slice images to be spliced ​​are obtained according to the scanning results; during the scanning process, the plane projection of the pathological slice images is corrected by the first correction parameter, and the poorly matched pathological slice images are corrected by the second correction parameter, and each of the pathological slice images can be directly spliced ​​after being corrected by the first correction parameter and the second correction parameter, so that the complete pathological slice image can be obtained by scanning and splicing at the same time during the scanning process, wherein the complete pathological slice image includes a plurality of spliced ​​pathological slice images, and the content of each of the pathological slice images includes different areas of the entire pathological slice.

[0008] A real-time image stitching system without illumination influence comprises: an acquisition module, used for acquiring a first prefabricated template and a second prefabricated template, wherein the first prefabricated template and the second prefabricated template are both templates shaped like a chessboard with evenly distributed black and white chessboards, and a uniform, light-transmitting monochrome coating is provided on the white grids of the first prefabricated template, the thickness of the monochrome coating is 5 μm, because the conventional white grids of the chessboard are transparent grids, and due to the light transmittance of different glass materials, it cannot be guaranteed that there are different brightness distributions when the glass slide is uneven, so the setting of the monochrome coating makes the light transmittance at different distances have different attenuation conditions; a determination module, used for determining a first correction parameter for calculating an orthographic projection image according to the first prefabricated template; and determining a second correction parameter for correcting an image position according to the second prefabricated template;

[0009] A scanning and stitching module is used to scan pathological slices using a scanner and obtain pathological slice images to be stitched according to the scanning results; during the scanning process, the plane projection of the pathological slice image is corrected by the first correction parameter, and the poorly matched pathological slice images are corrected by the second correction parameter. Each of the pathological slice images can be directly stitched after being corrected by the first correction parameter and the second correction parameter, so that the complete pathological slice image can be obtained by scanning and stitching during the scanning process, wherein the complete pathological slice image includes a plurality of stitched pathological slice images, and the content of each pathological slice image contains different areas of the entire pathological slice.

[0010] An electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned real-time image stitching method without illumination influence.

[0011] In one embodiment of the present invention, the method of determining the first correction parameter for calculating the orthographic projection image based on the first prefabricated template includes: placing the first prefabricated template under the lens of a scanner for shooting, and obtaining a plane projection image of the first prefabricated template after rotation according to the shooting result; obtaining an equivalent image on the rotation plane corresponding to the plane projection image, and determining the rotation angle of the rotation plane relative to the XOY plane according to the plane projection image and the equivalent image on the rotation plane, wherein two solutions are generated when determining the rotation angle of the rotation plane relative to the XOY plane, and whether the rotation plane is warped or sunken is judged by the different attenuation of the light transmittance at different distances, and the correct direction is obtained according to the judgment result; determining a rotation matrix according to the rotation angle, and inversely transforming the rotation matrix, and obtaining the first correction parameter for calculating the orthographic projection image according to the inverse transformation result.

[0012] In one embodiment of the present invention, determining the rotation angle of the rotation plane relative to the XOY plane according to the planar projection image and the equivalent image on the rotation plane includes: determining the rotation angle of the rotation plane relative to the XOY plane according to the following formula: Wherein, θ is the rotation angle of the rotation plane relative to the XOY plane; the general equation of the rotation plane is αx+βy+γz+λ=0, and α, β and γ are the normal vectors of the rotation plane. And λ is a constant term; is the normal vector to the XOY plane.

[0013] In one embodiment of the present invention, a rotation matrix is ​​determined according to the rotation angle, and an inverse transformation is performed on the rotation matrix, and a first correction parameter for calculating the orthographic projection image is obtained according to the inverse transformation result, including: determining the rotation matrix and obtaining the first correction parameter for calculating the orthographic projection image according to the following formula: Where R is the rotation matrix; I is the identity matrix; θ is the rotation angle of the rotation plane relative to the XOY plane; [v] × is the antisymmetric matrix form of the rotation, R -1 It is the inverse matrix of the rotation matrix and also the transposed matrix, and is also the first correction parameter used to calculate the forward projection image.

[0014] In one embodiment of the present invention, determining the second correction parameter for correcting the image position based on the second prefabricated template includes: scanning the second prefabricated template with a scanner, and obtaining blocks with an ideal grid distribution based on the scanning results; obtaining the coordinates of the upper left corner point of each ideal block, and recording the coordinate set as a first list; recording the coordinate set of the upper left corner point of the block obtained by the SURF alignment algorithm as a second list; and determining the second correction parameter for correcting the image position based on the first list and the second list.

[0015] In one embodiment of the present invention, determining the second correction parameter for correcting the image position according to the first list and the second list includes: performing linear regression with the first list as an independent variable and the second list as a dependent variable, obtaining a coefficient matrix according to the linear regression result, and obtaining the second correction parameter for correcting the image position according to the coefficient matrix; wherein the coefficient matrix is a yy Indicates the influence of the input y variable on the output y′, a yx Indicates the impact of the input variable x on the output y′, a xy Indicates the impact of the input y variable on the output x′, a xx Represents the influence of the input variable x on the output x'; so the linear equation can be expressed as: The linear transformation is a second correction parameter for correcting the image position.

[0016] In one embodiment of the present invention, during the scanning process, the plane projection of the pathological slice image is corrected by the first correction parameter, and the poorly matched pathological slice image is corrected by the second correction parameter. Each of the pathological slice images can be directly spliced ​​after being corrected by the first correction parameter and the second correction parameter, so that the complete pathological slice image can be obtained by scanning and stitching during the scanning process, including: during the scanning process, the positive projection image of the pathological slice image is calculated by the first correction parameter, and then the displacement error caused by mechanical movement is calculated by the second correction parameter, and the poorly matched pathological slice image is corrected according to the displacement error. Each of the pathological slice images can be directly spliced ​​after being corrected by the first correction parameter and the second correction parameter, so that the complete pathological slice image can be obtained by scanning and stitching during the scanning process.

[0017] As described above, the real-time image stitching method and system without illumination influence of the present invention have the following beneficial effects:

[0018] The present invention uses a first prefabricated template to correct the direction of the platform rising or sinking to provide a first correction parameter for the orthographic projection image, and restores the orthographic projection image of the pathological slice image to be spliced ​​obtained by scanning the scanner device through the first correction parameter; the present invention also obtains a second correction parameter through the simultaneous application of a linear fitting and matching algorithm, and corrects the displacement error caused by mechanical movement through the second correction parameter, thereby ensuring the accuracy of the displacement correction, so that the pathological slice images to be spliced ​​can be seamlessly spliced, and a more accurate position correction can be made for the pathological slice images to be spliced ​​with poor matching or excessive matching errors; therefore, the present invention can make the calculation speed fast under the action of the first correction parameter and the second correction parameter, and can enable the scanner device to scan and splice at the same time, thereby solving the problem of being highly dependent on matching between images and being unable to cope with poor matching and being unable to scan and splice at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a real-time image stitching method without illumination influence in a first embodiment of the present invention;

[0020] Figure 2 is a flow chart of a real-time image stitching system without illumination influence in a second embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of an electronic device in a third embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of the attenuation of brightness as the distance from the light source increases in the present invention;

[0023] Figure 5 It is a schematic diagram of the present invention in which the captured image is not an orthographic image due to uneven assembly;

[0024] Figure 6 It is a schematic diagram of the spatial relationship between the image and the projection in the present invention;

[0025] Figure 7 It is a schematic diagram of two situations of plane projection restoration of the S rotation plane in the present invention. DETAILED DESCRIPTION

[0026] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0027] The first embodiment of the present invention relates to a real-time image stitching method without illumination influence, the process is as follows: Figure 1 As shown, the details are as follows:

[0028] S101, obtaining a first prefabricated template and a second prefabricated template.

[0029] Specifically, for the first prefabricated template: design a scanning template for pre-scanning, and select a template shaped like a chessboard for the scanning template, because the chessboard has a high contrast under the microscope and can well detect the projection shape; for the chessboard size, the width and height need to be designed to be 360μm, and the tolerance is less than or equal to 0.2μm; however, the conventional white chessboard grid is a transparent grid, and due to the light transmittance of different glass materials, it cannot be guaranteed that there will be different brightness distributions when the glass slide is uneven, so a uniform, light-transmitting monochrome coating is applied on the transparent grid of the chessboard, wherein the thickness of the monochrome coating is 5μm, to ensure that when the camera is not vertical or the platform is not horizontal, different distances (the distance between the plane and the light source) will result in different attenuation of the light transmittance intensity; such as Figure 4 As shown, there is less light transmission away from the light source, and its light transmission intensity is low; the light transmission intensity is high near the light source, so the light transmission intensity can be used to determine which direction the glass slide is tilted or sunken.

[0030] For the second prefabricated template: customize a scanning template, which presents a black and white checkerboard distribution, the white checkerboard is transparent checkerboard, the checkerboard size is set to 30μm×30μm, the tolerance is ≤0.2μm, and the reason for the tolerance ≤0.2μm is that the error of the captured image can be <1 pixel.

[0031] S102: Determine a first correction parameter for calculating an orthographic projection image according to the first prefabricated template.

[0032] Specifically, the projection calculation platform rotation angle is used to correct the positive projection image: first, the first prefabricated template is placed under the microscope to shoot, and it is necessary to ensure that there is only one complete monochrome quadrilateral (non-black) under the microscope, such as Figure 5 As shown, during the scanning process, the platform may be uneven due to platform assembly, so the image captured by the camera is not an orthographic projection image, but a plane projection after rotation, which leads to errors (it should be noted that in this embodiment, when a scanner is used to scan a pathological section, a slide containing the pathological section is placed on the platform and then scanned using a scanner device); Figure 6 As shown, assuming that pattern A is the positive projection of the image, A′ is its equivalent image on the rotation plane, the rotation plane where A′ is located is denoted as S, and the irregular trapezoid B is the projection of A′ on the XOY plane. It is necessary to solve the rotation angle of plane S relative to the XOY plane through A′ and B, and then the positive projection image A can be obtained through the inverse transformation of the rotation angle;

[0033] Rectangle A coincides with the XOY plane, and its four vertices can be assumed to have the following coordinates: 1 =(x1 ,y 1 ,0),A 2 =(x 2 ,y 2 ,0),A 3 =(x 3 ,y 3 ,0),A 4 =(x 4 ,y 4 ,0), where z=0, because it coincides with the XOY plane; the rotated rectangle A′ (on plane S): After the spatial rotation, each vertex of rectangle A′ is transformed from the XOY plane to another spatial plane S. At this time, the vertex coordinates of rectangle A′ are: A′ 1 =(x′ 1 ,y′ 1 ,z′ 1 ), A 2 ′=(x′ 2 ,y′ 2 ,z′ 2 ), A′ 3 =(x′ 3 ,y′ 3 ,z′ 3 ), a′ 4 =(x′ 4 ,y′ 4 ,z′ 4 ); Irregular quadrilateral B projected onto the XOY plane: In the rotated state, the rectangle A′ projected onto the XOY plane will form an irregular quadrilateral B. When projected onto the XOY plane, the z coordinates of all vertices are ignored, and the projected vertex coordinates are: B′ 1 =(x′ 1 ,y′ 1 ), B′ 2 =(x′ 2 ,y′ 2 ), B′ 3 =(x′ 3 ,y′ 3 ), B′ 4 =(x′ 4 ,y′ 4 );

[0034] Size constraints of rectangle A: The side lengths of rectangle A are known. Assuming that the side lengths of rectangle A are w (width) and h (height), then no matter how it is rotated, the side lengths after rotation (that is, the distance between vertices) should remain unchanged in 3D space. To explain this in detail: the distance between adjacent vertices of the rectangle, such as the distance between A1 and A2, should remain w before and after rotation, that is: ||A′ 1 -A′ 2||=w①. Similarly, for the other direction of the rectangle, the distance between A1 and A4 should be h: ||A′ 1 -A′ 4 ||=h②, after rotation, the vertex A of rectangle A 1 , A 2 , A 3 , A 4 On the new plane S, the distance between their vertices should still satisfy the geometric constraints. Since the x and y coordinates of each vertex on the projected quadrilateral B are known, and the edge length constraint can provide additional geometric conditions, it helps to determine the normal vector of plane S and its information in the Z direction.

[0035] where x′ i , y′ i is the known projection coordinate, z′ i is the unknown quantity that needs to be solved; for the two opposite sides of the rectangle, the following equations can be written respectively: Width constraint equation: Height constraint equation: Combining ④ and ⑤, let: Available Solving for this yields: Since ⑥ provides z′ 1 Relative to z′ 2 and z′ 4 The relationship between the two is to select a value as the benchmark. Usually, we can set the vertex A′ 1 The z coordinate is 0 (z′ 1 = 0), thereby simplifying the solution (setting the z coordinate of any point does not affect the spatial relationship); at this time, two sets of solutions will be generated, and the corresponding situations of the two sets of solutions are as follows Figure 7 As shown, the light transmittance in the first prefabricated template is used to determine whether the plane S is warped or sunken. After the determination, one of the two sets of solutions will be discarded. Then, there is only one set of solutions, and the spatial rotation angle is unique and determined.

[0036] To calculate the angle between plane S and the XOY plane (the angle between plane S and the X, Y, and Z axes), we need to calculate the normal vector of plane S. The general equation of plane S can be written as: αx+βy+λz+λ=0⑦, where α, β, and γ are the normal vectors of the plane. And λ is a constant term; in order to calculate the angle between plane S and the XOY plane, you need to first find the normal vector Then determine λ through a known point on the plane; through the above calculation, there are three vertices of rectangle A on plane S, namely A′ 1 , A′ 2 , A′4 , A′ 1 =(x′ 1 ,y′ 1 ,z′ 1 ), A′ 2 =(x′ 2 ,y′ 2 ,z′ 2 ), A′ 4 =(x′ 4 ,y′ 4 ,z′ 4 ), then we can construct a vector Normal vector

[0037] By normal vector The rotation angles of plane S and the three axes X, Y, and Z can be known, so as to construct the inverse transformation R -1 , then the orthographic image A=R -1 B, to explain this in detail: To construct the inverse transformation matrix R -1 , its meaning and construction steps need to be clarified, inverse transformation R -1 is used to transform the normal vector of plane S Align to a reference plane (such as the XOY plane); the rotation matrix R transforms the normal vector of the XOY plane Normal vector rotated to the target plane S direction, so the inverse transformation R -1 That is to Rotate back The direction of; normal vector normalization: The rotation axis is obtained, And unitize, The rotation matrix R is constructed using the Rodrigues formula. The Rodrigues formula defines the rotation matrix R around the rotation axis v. The rotation angle is so Where I is the identity matrix, which is used to preserve the non-rotated parts, [v] × is the antisymmetric matrix form of the rotation: The inverse of a rotation matrix is ​​its transposed matrix, so R -1 =R T , substituting the numerical value into it, we can get: Orthographic image A = R -1 B, get the coordinates of the positive projection image, and change R -1 as the first correction parameter.

[0038] S103: Determine a second correction parameter for correcting the image position according to the second prefabricated template.

[0039] Specifically, the deviation caused by the motor movement is obtained through linear fitting, and is used as a correction parameter to correct the image position: first, the second prefabricated template is scanned, and a set of very regularly grid-distributed blocks is obtained according to the second prefabricated template. Specifically, the coordinates of the upper left corner point of each ideal block are obtained, and the coordinate set is recorded as a list pos(y,x). The coordinate set of the upper left corner point of the block calculated by the SURF registration algorithm is recorded as a list image_coordinates(y′,x′), and the coordinates in the pos list are used as independent variables, and the coordinates in the image_coordinates list are used as dependent variables for linear regression; that is, the original coordinates pos are used as input (X), and the calculated coordinates image_coordinates are used as output (Y). According to the mapping relationship, we can get: Y = X·coefficient matrix + intercept. The form of this matrix is ​​as follows: The coefficient matrix is a yy Indicates the influence of the input y variable on the output y′, a yx Indicates the impact of the input variable x on the output y′, a xy Indicates the impact of the input y variable on the output x′, a xx Represents the influence of the input variable x on the output x'; so the linear equation can be expressed as: By finding the coefficient matrix, the error of the image is constrained when the matching is poor, and this linear transformation is used as the second correction parameter.

[0040] S104, scanning the pathological slices using a scanner, and obtaining the pathological slice images to be spliced ​​according to the scanning results.

[0041] S105, during the scanning process, the plane projection of the pathological slice image is corrected by the first correction parameter, and the poorly matched pathological slice image is corrected by the second correction parameter. Each pathological slice image can be directly spliced ​​after being corrected by the first correction parameter and the second correction parameter, so that the pathological slice image can be obtained by scanning and splicing at the same time during the scanning process.

[0042] Specifically, a complete pathological slice image includes several spliced ​​pathological slice images, the content of each pathological slice image includes different areas of the entire pathological slice, and the complete pathological slice image can be considered as a grid map composed of several images, each small grid of the grid map is a pathological slice image, and the more important technical point in the present invention is to calculate the first correction parameter and the second correction parameter, and to correct each pathological slice image by the first correction parameter and the second correction parameter; and in the scanning process, the orthographic projection image of the pathological slice image is calculated by the first correction parameter, and then the displacement error caused by mechanical movement is calculated by the second correction parameter, and the poorly matched pathological slice images are corrected according to the displacement error, and each pathological slice image can be directly spliced ​​after being corrected by the first correction parameter and the second correction parameter, so that the complete pathological slice image can be obtained by scanning and splicing during the scanning process.

[0043] Further, the above process can also be specifically as follows: Step 1: Scan the customized pre-scan template (that is, the first prefabricated template); Step 2: Determine the platform warping direction according to the transmittance intensity. When calculating the rotation angle of the rotating plane S relative to the XOY plane, two solutions will be generated. One direction will be discarded through the direction determination, and the correct direction will be retained; Step 3: Restore the orthographic projection image. The purpose is to obtain the orthographic projection coordinates through the current projection. This is described in detail. By shooting the monochrome rectangular block-pattern A′, the coordinates of the orthographic projection image A of A′ are specified as A 1 =(x 1 ,y 1 ,0),A 2 =(x 2 ,y 2 ,0),A 3 =(x 3 ,y 3 ,0),A 4 =(x 4 ,y 4 ,0), determine the coordinates of the projection of image A′, i.e., the irregular quadrilateral B: B′ 1 =(x′ 1 ,y′ 1 ,0),B′ 2 =(x′ 2 ,y′ 2 ,0),B′ 3 =(x′ 3 ,y′ 3 ,0),B′ 4 =(x′ 4 ,y′ 4 ,0); then the direction obtained by the mathematical operation process in step 102 discards a mathematical solution, and finally the inverse transformation matrix R is obtained-1 , then the orthographic projection coordinates of A′ can be calculated by the current projection B of A′. -1 B, thereby obtaining the first correction parameter;

[0044] Step 4: Calculate the displacement error caused by mechanical movement: Here we need to scan the scale (that is, the second prefabricated template). The original grid position of the scanned block is calculated under ideal conditions. The actual scan position is obtained through the SURF matching algorithm, and then the linear conversion relationship of the position is calculated, that is: actual position = transformation matrix * ideal position + deviation, that is The transformation relationship is used as the second correction parameter; the fifth step: use a scanner to scan the pathological slice to obtain the pathological slice image to be spliced; the sixth step: calculate the orthographic projection image of the pathological slice image through the first correction parameter; the seventh step: correct the poor match through the second correction parameter, so after using the scanner to scan a pathological slice image, it can be corrected by the first correction parameter and the second correction parameter, and after correction, it can be directly spliced, so it can be scanned and spliced ​​at the same time during the scanning process.

[0045] The second embodiment of the present invention relates to a real-time image stitching system without illumination influence, see Figure 2 ,include:

[0046] An acquisition module, used for acquiring a first prefabricated template and a second prefabricated template, wherein the first prefabricated template and the second prefabricated template are both templates shaped like a chessboard with evenly distributed black and white chessboards, and a uniform, light-transmitting single-color coating is provided on the white grids of the first prefabricated template, and the thickness of the single-color coating is 5 μm. Because the conventional white grids of the chessboard are transparent grids, and due to the light transmittance of different glass materials, it cannot be guaranteed that there are different brightness distributions when the glass slide is uneven, so the setting of the single-color coating makes the light transmittance at different distances have different attenuation conditions;

[0047] A determination module, configured to determine a first correction parameter for calculating an orthographic projection image according to a first prefabricated template; and to determine a second correction parameter for correcting an image position according to a second prefabricated template;

[0048] The scanning and stitching module is used to scan the pathological slices with a scanner and obtain the pathological slice images to be stitched according to the scanning results; during the scanning process, the plane projection of the pathological slice images is corrected by the first correction parameter, and the poorly matched pathological slice images are corrected by the second correction parameter. Each pathological slice image can be directly stitched after being corrected by the first correction parameter and the second correction parameter, so that the complete pathological slice image can be obtained by scanning and stitching during the scanning process, wherein the complete pathological slice image includes several stitched pathological slice images, and the content of each pathological slice image includes different areas of the entire pathological slice.

[0049] It is not difficult to find that this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0050] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

[0051] The third embodiment of the present invention relates to an electronic device, see Figure 3 ,include:

[0052] At least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned real-time image stitching method without lighting influence, wherein the electronic device can be arranged inside the scanner, or can be an external terminal device such as a computer.

[0053] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0054] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0055] A fourth embodiment of the present invention relates to a computer-readable storage medium storing a computer program, and the computer program implements the above method embodiment when executed by a processor.

[0056] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0057] In summary, the present invention uses the first correction parameter and the second correction parameter to restore the orthographic projection image of the pathological section image to be stitched and correct the position of the pathological section image to be stitched, which can not only maintain the integrity of the image and stitch images containing blank samples, sparse samples or very few features, but also perform stitching while scanning, and finally stitch all the pathological section images to be stitched into a complete digital image, so as to achieve the purpose of real-time stitching of large-scale images and generation of digital pathological sections, and the image stitching result is more accurate by correcting the stitching position of the image.

[0058] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. All equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A real-time image stitching method without illumination influence, characterized in that: The following steps are involved: Obtain a first prefabricated template and a second prefabricated template, wherein both the first prefabricated template and the second prefabricated template are templates shaped like a chessboard with evenly distributed black and white chessboards, and a uniform, light-transmitting single-color coating is provided on the white grids of the first prefabricated template, and the thickness of the single-color coating is 5 μm. Because the conventional white grids of the chessboard are transparent grids, and due to the light transmittance of different glass materials, it cannot be guaranteed that there are different brightness distributions when the glass slide is uneven, so the setting of the single-color coating makes the light transmittance at different distances have different attenuation conditions; Determining a first correction parameter for calculating the orthographic projection image according to the first prefabricated template; Determining a second correction parameter for correcting the image position according to the second prefabricated template; Scanning the pathological sections with a scanner, and obtaining the pathological section images to be spliced ​​according to the scanning results; During the scanning process, the plane projection of the pathological slice image is corrected by the first correction parameter, and the poorly matched pathological slice image is corrected by the second correction parameter. Each of the pathological slice images can be directly spliced ​​after being corrected by the first correction parameter and the second correction parameter, so that the scanning and splicing can be performed simultaneously to obtain a complete pathological slice image, wherein the complete pathological slice image includes several spliced ​​pathological slice images, and the content of each pathological slice image contains different areas of the entire pathological slice.

2. The real-time image stitching method without illumination influence according to claim 1, characterized in that: The step of determining a first correction parameter for calculating the orthographic projection image according to the first prefabricated template comprises: Placing the first prefabricated template under a scanner lens for photographing, and obtaining a plane projection image of the first prefabricated template after rotation according to the photographing result; Obtain an equivalent image on the rotation plane corresponding to the plane projection image, and determine the rotation angle of the rotation plane relative to the XOY plane according to the plane projection image and the equivalent image on the rotation plane, wherein two solutions are generated when determining the rotation angle of the rotation plane relative to the XOY plane, and judge whether the rotation plane is upturned or downturned by different attenuation conditions of the light transmission intensity at different distances, and obtain the correct direction according to the judgment result; A rotation matrix is ​​determined according to the rotation angle, and an inverse transformation is performed on the rotation matrix. A first correction parameter for calculating the orthographic projection image is obtained according to the inverse transformation result.

3. The real-time image stitching method without illumination influence according to claim 2, characterized in that: Determining the rotation angle of the rotation plane relative to the XOY plane based on the plane projection image and the equivalent image on the rotation plane includes: Determine the rotation angle of the rotation plane relative to the XOY plane according to the following formula: Wherein, θ is the rotation angle of the rotation plane relative to the XOY plane; the general equation of the rotation plane is αx+βy+γz+λ=0, and α, β and γ are the normal vectors of the rotation plane. And λ is a constant term; is the normal vector to the XOY plane.

4. The real-time image stitching method without illumination influence according to claim 2, characterized in that: Determining a rotation matrix according to the rotation angle, and performing an inverse transformation on the rotation matrix, and obtaining a first correction parameter for calculating the orthographic projection image according to the inverse transformation result, including: The rotation matrix and the first correction parameters for calculating the orthographic projection image are determined according to the following formula: Where R is the rotation matrix; I is the identity matrix; θ is the rotation angle of the rotation plane relative to the XOY plane; [v] × is the antisymmetric matrix form of the rotation, R -1 It is the inverse matrix of the rotation matrix and also the transposed matrix, and is also the first correction parameter used to calculate the forward projection image.

5. The real-time image stitching method without illumination influence according to claim 1, characterized in that: The step of determining a second correction parameter for correcting the image position according to the second prefabricated template comprises: Scanning the second prefabricated template using a scanner, and obtaining blocks with ideal grid distribution according to the scanning result; Get the coordinates of the upper left corner of each ideal tile, and record the coordinate set as the first list; The coordinate set of the upper left corner point of the tile obtained by the SURF registration algorithm is recorded as the second list; A second correction parameter for correcting the image position is determined based on the first list and the second list.

6. The real-time image stitching method without illumination influence according to claim 5, characterized in that: The determining of the second correction parameter for correcting the image position according to the first list and the second list comprises: Performing linear regression with the first list as an independent variable and the second list as a dependent variable, obtaining a coefficient matrix according to the linear regression result, and obtaining a second correction parameter for correcting the image position according to the coefficient matrix; Among them, the coefficient matrix is a yy Indicates the influence of the input y variable on the output y′, a yx Indicates the impact of the input variable x on the output y′, a xy Indicates the impact of the input y variable on the output x′, a xx Represents the influence of the input variable x on the output x'; so the linear equation can be expressed as: The linear transformation is a second correction parameter for correcting the image position.

7. The real-time image stitching method without illumination influence according to claim 1, characterized in that: During the scanning process, the plane projection of the pathological slice image is corrected by the first correction parameter, and the poorly matched pathological slice image is corrected by the second correction parameter. Each of the pathological slice images can be directly spliced ​​after being corrected by the first correction parameter and the second correction parameter, so that the complete pathological slice image can be obtained by scanning and splicing during the scanning process, including: During the scanning process, the first correction parameter is used to calculate the orthographic projection image of the pathological slice image, and then the displacement error caused by mechanical movement is calculated by the second correction parameter. The poorly matched pathological slice image is corrected according to the displacement error. Each pathological slice image can be directly spliced ​​after being corrected by the first correction parameter and the second correction parameter, so that the pathological slice image can be scanned and spliced ​​at the same time during the scanning process to obtain a complete pathological slice image.

8. A real-time image stitching system without lighting influence, characterized in that: include: An acquisition module, used for acquiring a first prefabricated template and a second prefabricated template, wherein the first prefabricated template and the second prefabricated template are both templates shaped like a chessboard with evenly distributed black and white chessboards, and a uniform, light-transmitting single-color coating is provided on the white grids of the first prefabricated template, and the thickness of the single-color coating is 5 μm. Because the conventional white grids of the chessboard are transparent grids, and due to the light transmittance of different glass materials, it cannot be guaranteed that there are different brightness distributions when the glass slide is uneven, so the setting of the single-color coating makes the light transmittance at different distances have different attenuation conditions; A determination module, configured to determine a first correction parameter for calculating an orthographic projection image according to the first prefabricated template; and to determine a second correction parameter for correcting an image position according to the second prefabricated template; A scanning and stitching module is used to scan pathological slices using a scanner and obtain pathological slice images to be stitched according to the scanning results; during the scanning process, the plane projection of the pathological slice image is corrected by the first correction parameter, and the poorly matched pathological slice images are corrected by the second correction parameter. Each of the pathological slice images can be directly stitched after being corrected by the first correction parameter and the second correction parameter, so that the complete pathological slice image can be obtained by scanning and stitching during the scanning process, wherein the complete pathological slice image includes a plurality of stitched pathological slice images, and the content of each pathological slice image contains different areas of the entire pathological slice.

9. An electronic device, characterized in that: include: at least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the real-time image stitching method without illumination influence as described in any one of claims 1 to 7.

Images