A fast stitching method and system based on bayer images

Abstract

The present invention belongs to the technical field of microscopic image processing, and specifically provides a fast splicing method and system based on Bayer images, wherein the method includes: calculating the relative position relationship between the current input image and the adjacent cell images at different directional positions; calculating the mapping relationship of the current input image in the canvas; based on image fusion; pasting the fused image onto the canvas. This method only needs to calculate 1 / 3 of the image data of the traditional image. In the matching stage, not only the matching relationship between different channels is considered, but also the linear interpolation method is adopted through super-resolution thinking to improve the matching accuracy. At the same time, in the splicing and fusion stage, a fusion algorithm is designed based on the pixel distribution rules of the image, which solves the color cast and seam problems while improving the splicing efficiency. The position attributes of the R, G1, G2, and B channels and the characteristics of the image to RGB image are fully considered, so that the spliced ​​cell image is free of color cast and seam traces.

Description

Technical Field

[0001] The present invention relates to the technical field of microscopic image processing, and more specifically, to a fast splicing method and system based on Bayer images. Background Art

[0002] Microscopic optical imaging technology refers to the technology that can obtain a magnified image of a tiny sample by passing visible light through a sample or reflected from a sample through one or more lenses. The obtained image is recorded using a photosensitive plate or a digital image detector such as CCD or CMOS, and displayed and analyzed on a computer. With the development of digital microscope technology, the existing technology can use sample slide scanning to fully scan sample slides to obtain full-slice digital microscopic images. However, the growing demand for slide scanning urgently requires a scanning device that can quickly output images. Among the factors that affect the scanning imaging speed, the splicing efficiency of cell images is the most critical. Therefore, how to find an ultra-high-speed image splicing method to improve the efficiency of slide scanning is an urgent problem to be solved. Summary of the invention

[0003] The present invention aims to solve the technical problem of low efficiency in splicing cell images in the prior art.

[0004] The present invention provides a fast splicing method based on Bayer images, comprising the following steps:

[0005] S1, calculate the relative position relationship between the current input Bayer image and the adjacent cell images at different directions;

[0006] S2, calculate the mapping relationship of the current input Bayer image in the canvas according to the relative position relationship;

[0007] S3, image fusion based on bayer graph;

[0008] S4, paste the fused Bayer map onto the canvas.

[0009] Preferably, the S1 specifically includes:

[0010] S11, splitting the current input Bayer image and each adjacent cell image into four-channel RGGB images respectively;

[0011] S12, respectively calculating the relative position relationship between the current input Bayer image and the segmented images of each adjacent cell image under the same channel;

[0012] S13, merging the relative positional relationships under each channel to obtain the relative positional relationship between the current input Bayer image and each adjacent cell image in an xy coordinate system.

[0013] Preferably, S12-S13 specifically include:

[0014] The standard squared difference is used to calculate the minimum value of the current input Bayer image and each adjacent cell image in each channel. Each minimum value corresponds to a candidate matching position in the xy coordinate system.

[0015] The corresponding candidate matches are synthesized to obtain the relative position relationship between the previous input Bayer image and the adjacent cell images.

[0016] Preferably, S2 further includes eliminating errors in the relative position relationship, specifically including:

[0017] S21, construct an error hiding tree V_stichs, where V_stichs[i] represents the i-th tree node, and each node contains the x and y coordinate positions of an image;

[0018] S22, calculating the x and / or y coordinate position difference between the tree node corresponding to the nth image and its adjacent tree node. If the difference is greater than a maximum threshold, it means that the position deviation between the nth image structure and the adjacent image structure in a certain direction exceeds the maximum threshold, wherein the nth image corresponds to the hidden tree V_stichs[n];

[0019] S23, diffusing the difference along the certain direction to the image in the global vertical direction;

[0020] Wherein, i and n are both natural numbers.

[0021] Preferably, the S23 specifically includes:

[0022] The average diffusion method is adopted to complete the error diffusion within two scanning lines;

[0023] When the error of the current row is eliminated, the second row replaces the first row as a reference row to participate in the subsequent error diffusion.

[0024] Preferably, the image fusion mode of S3 is feathering, which specifically includes:

[0025] Overlapping pixels are blended using a weighted average color value alpha factor, which has a value of 1 at the center pixel and then becomes 0 as it decreases linearly with the boundary pixels;

[0026] The four channels of RGGB are fused separately, and then the separately fused images are merged into bayer graphs.

[0027] Preferably, the S3 specifically includes:

[0028] S31, setting the image width and height of the bayer images B1 and B2 to be fused;

[0029] S32, decompose the bayer image into four channels: R, G1, G2, and B;

[0030] S33, using the mapping relationship calculated in step S2 as the fusion input position;

[0031] Specifically, if the position of B1 is pox_x1, pos_y1, and the position of B2 is pos_x2, pos_y2, the following position transformation is required before fusion:

[0032] p_x1 = pox_x1 / 2×2

[0033] p_y1 = pox_y1 / 2×2

[0034] p_x2 = pox_x2 / 2×2

[0035] p_y2 = pox_y2 / 2×2

[0036] S34, using the positions calculated in step S33, the four-channel images are fused respectively.

[0037] The present invention also provides a fast splicing system based on Bayer images, the system is used to implement a fast splicing method based on Bayer images, comprising:

[0038] A relative position relationship calculation module is used to calculate the relative position relationship between the current input Bayer image and adjacent cell images at different directions;

[0039] A mapping relationship calculation module is used to calculate the mapping relationship of the current input Bayer image in the canvas according to the relative position relationship;

[0040] Image fusion module, used for image fusion based on Bayer graph;

[0041] The canvas fitting module is used to paste the fused Bayer map onto the canvas.

[0042] The present invention also provides an electronic device, comprising a memory and a processor, wherein the processor is used to implement the steps of a fast splicing method based on bayer images when executing a computer management program stored in the memory.

[0043] The present invention also provides a computer-readable storage medium on which a computer management program is stored. When the computer management program is executed by a processor, the steps of a fast splicing method based on bayer images are implemented.

[0044] Beneficial effects: The present invention provides a fast stitching method and system based on Bayer images, wherein the method comprises: calculating the relative position relationship between the current input Bayer image and adjacent cell images at different directions; calculating the mapping relationship of the current input Bayer image in the canvas based on the relative position relationship; image fusion based on the Bayer graph; and pasting the fused Bayer graph onto the canvas.

[0045] This method only needs to calculate 1 / 3 of the image data of the traditional image. In the matching stage, not only the matching relationship between different channels is considered, but also the linear interpolation method is used through super-resolution thinking to improve the matching accuracy. At the same time, in the splicing and fusion stage, a fusion algorithm is designed based on the pixel distribution rule of the Bayer image, which improves the splicing efficiency and solves the color cast and seam problems. Fully considering the position attributes of the R, G1, G2, and B channels and the characteristics of the Bayer image to RGB image, the spliced ​​cell image not only has no color cast problem, but also has no trace of seams. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flow chart of a fast splicing method based on Bayer images provided by the present invention;

[0047] Figure 2 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0048] Figure 3 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention;

[0049] Figure 4 The error concealment tree diagram provided by the present invention;

[0050] Figure 5 A canvas map provided by the present invention;

[0051] Figure 6 The bayer to RGB image provided by the present invention. DETAILED DESCRIPTION

[0052] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0053] Combination Figure 1 , Figure 4~Figure 6 The present invention provides a fast splicing method based on Bayer images, comprising the following steps:

[0054] S1, calculate the relative position relationship between the current input Bayer image and the adjacent cell images in different directions. Calculate the current input Bayer image (image resolution is 2448 2048) and the relative position relationship of the cell image in four directions: up, down, left, and right.

[0055] Specifically, to calculate the relative position relationship of the Bayer image, it is necessary to calculate the relative position relationship in the four directions of top, bottom, left, and right. Next, the calculation process is explained by taking the calculation of the position relationship between the current image and the image above as an example. Suppose the current image is img_cur, the image above is img_up, and the Bayer format is RGGB. Before calculation, it is necessary to split the Bayer image into four-channel images R, G1, G2, and B (the four channels of the current image are represented by R_cur, G1_cur, G2_cur, and B_cur, and the image above is represented by R_up, G1_up, G2_up, and B_up), and then calculate the relative position relationship between R_cur and R_up, G1_cur and G1_up, G2_cur and G2_up, and B2_cur and B2_up. Considering that the distance between adjacent cell images is extremely short during the actual cell scanning process, and that the mechanical equipment can be tuned to ensure that the cell slide only moves in the horizontal and vertical directions during the translation process, the standard square difference can be used to calculate the relative position relationship based on the characteristics of horizontal and vertical movement. The calculation methods are (1), (2), (3), and (4). The positions R_x, R_y, G1_x, G1_y, G2_x, G2_y, B_x, B_y corresponding to the calculated minimum values ​​S_R, S_G1, S_G2, and S_B are candidate matching positions.

[0056]

[0057] After obtaining the candidate matching positions, the relative position relationship between the two is obtained through equations (5) and (6).

[0058]

[0059] According to (1) to (6) above, we can similarly obtain down, down, left, left、、 right, right.

[0060] S2, based on the relative position relationship, calculate the mapping relationship of the current input Bayer image in the canvas. How to combine and arrange the obtained positions and how to solve the accumulated error is the main work of this part. When the error accumulates to a certain extent, a local misalignment will be very obvious. The present invention proposes an error accumulation diffusion tree method to perform error diffusion. The specific method is: first construct an error hiding tree, and its structure diagram is as follows Figure 4 shown.

[0061] In the diagram, each square represents an image structure, which is as follows:

[0062] typedef struct soft_stich_st{

[0063] int x_id, y_id;

[0064] int score[4];

[0065] int dis_x[4];

[0066] int dis_y[4];

[0067] }stich_st;

[0068] The error hiding tree in the figure above can be represented by V_stichs, where V_stichs[i] represents the i-th tree node. When cell misalignment occurs in a local area, it is usually because the position deviation between the n-th image structure and the adjacent image structure exceeds the set maximum threshold. Assuming that the image block with a large local error is the n-th block, using the current The x_id and y_id of [n] are used to infer the adjacent image blocks just above, below, to the left, and to the right. The corresponding image blocks are numbered as: m1, m2, m3, m4. If the error just below exceeds the threshold, the error between the nth image and the adjacent image m2 just below is expressed as:

[0069]

[0070] The value calculated by it must be greater than the set threshold thr_dis. Then the error value directly below needs to be diffused to the global vertical image. The diffusion amount of a single image is calculated by equations (7) and (8), and the diffusion method is average diffusion.

[0071] Avg_dis_up = (dis_down - thr_dis) / (line_num - [n].y_id)(7)

[0072] Avg_dis_down = (dis_down - thr_dis) / [n].y_id (8)

[0073] In the above formula, line_num represents the number of scanned frames of a line of image. Finally, in order to balance the scanning speed, the error diffusion needs to be completed within two lines of scanning, that is, Figure 4 As shown in the dotted box in , once the error of the current row is eliminated, the second row will replace the first row as the reference row to participate in the subsequent error diffusion. This can eliminate the position error as much as possible, ensure the relative position relationship of each image is accurate, and facilitate the subsequent image fusion.

[0074] S3, image fusion based on Bayer graph. Image fusion is to mix the pixel colors of the images to be fused in the overlapping area to avoid seams. The simplest form of use is feathering, which uses a weighted average color value to fuse overlapping pixels, specifically using an alpha factor, also known as an alpha channel, which has a value of 1 at the center pixel and then becomes 0 when it decreases linearly with the boundary pixels. Bayer image-based fusion requires four channels to be fused separately, and then the separately fused images are merged by bayer graphs, otherwise the fused image will have a color cast problem. The fusion steps used in the present invention are:

[0075] Step 1: Set the bayer images B1 and B2 to be fused, whose image width and height are imgw and imgh respectively;

[0076] Step 2: Decompose the bayer image into four channels: R, G1, G2, and B.

[0077] Step 3: Use the mapping relationship calculated in step S2 as the fusion input position. If the position of B1 is pox_x1, pos_y1, and the position of B2 is pos_x2, pos_y2, the following position transformation is required before fusion:

[0078] p_x1 = pox_x1 / 2 2

[0079] p_y1 = pox_y1 / 2 2

[0080] p_x2 = pox_x2 / 2 2

[0081] p_y2 = pox_y2 / 2 2

[0082] Step 4: Use the positions calculated in the third step to fuse the four-channel images separately. The specific fusion method is shown in equations (9)-(12):

[0083] ref_i = (p_y1 + imgh + p_y2 ) / 2

[0084] ref_j = (p_x1 + imgw + p_x2 ) / 2

[0085] r1=sqrt((ref_i-i) (ref_i-i)+(ref_j-j) (ref_j-j)) / line_blend (9)

[0086] (i,j)= (i-p_y1,j-p_x1) r1+B2(i-p_x2,j-p_y2) (1-r1) (10)

[0087] (i,j+1)= (i-p_y1,j-p_x1+1) r1+B2(i-p_x2,j-p_y2+1) (1-r1) (11)

[0088] (i+1,j)= (i-p_y1+1,j-p_x1) r1+B2(i-p_x2+1,j-p_y2) (1-r1) (12)

[0089] (13)

[0090] Among them, line_blend is a fixed parameter, which is input from the outside and is usually set to line_blend = imgw / 10. For the non-fused area, the original image is pasted into the canvas.

[0091] S4, paste the fused Bayer graph onto the canvas. Paste the fused Bayer graph onto the canvas. Figure 5 The figure shows the situation of pasting the fused image onto the canvas, where each red box represents 2448 A small picture of 2048. Figure 6 It can be seen that relative to Figure 5 After the bayer image in the figure is converted into an RGB image, there is no color cast problem or splicing marks at the seams.

[0092] Compared with the traditional stitching in RGB image domain, the present invention gives full play to the advantages of small data volume and high computational efficiency in Bayer domain. The beneficial effects of the present invention can be specifically defined as:

[0093] 1. The cell image stitching efficiency is high, which can be achieved on a computer with i7-11370H @ 3.30GHz at 4ms / frame, and the resolution of the Bayer image is 2448 2048;

[0094] 2. The cell image stitching effect is good. Based on the distribution characteristics of the Bayer image, the present invention designs a new Bayer image matching and fusion method. In the matching stage, not only the matching relationship between different channels is considered, but also the linear interpolation method is adopted through super-resolution thinking to improve the matching accuracy. At the same time, in the fusion stage, the position attributes of the R, G1, G2, and B channels and the characteristics of the Bayer image to RGB image are fully considered. Finally, the spliced ​​cell image not only has no color cast problem, but also has no stitching traces.

[0095] The embodiment of the present invention further provides a fast stitching system based on Bayer images, the system is used to implement the fast stitching method based on Bayer images as described above, including:

[0096] A relative position relationship calculation module is used to calculate the relative position relationship between the current input Bayer image and adjacent cell images at different directions;

[0097] A mapping relationship calculation module is used to calculate the mapping relationship of the current input Bayer image in the canvas according to the relative position relationship;

[0098] Image fusion module, used for image fusion based on Bayer graph;

[0099] The canvas fitting module is used to paste the fused Bayer map onto the canvas.

[0100] See also Figure 2 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 2 As shown, an embodiment of the present invention provides an electronic device, including a memory 1310, a processor 1320, and a computer program 1311 stored in the memory 1310 and executable on the processor 1320. When the processor 1320 executes the computer program 1311, the following steps are implemented: S1, calculating the relative position relationship between the current input Bayer image and the adjacent cell images at different direction positions;

[0101] S2, calculate the mapping relationship of the current input Bayer image in the canvas according to the relative position relationship;

[0102] S3, image fusion based on bayer graph;

[0103] S4, paste the fused Bayer map onto the canvas.

[0104] See also Figure 3 Schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. Figure 3 As shown, this embodiment provides a computer-readable storage medium 1400 on which a computer program 1411 is stored. When the computer program 1411 is executed by a processor, the following steps are implemented: S1, calculating the relative position relationship between the current input Bayer image and adjacent cell images at different direction positions;

[0105] S2, calculate the mapping relationship of the current input Bayer image in the canvas according to the relative position relationship;

[0106] S3, image fusion based on bayer graph;

[0107] S4, paste the fused Bayer map onto the canvas.

[0108] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0114] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A fast splicing method based on Bayer images, characterized in that: The following steps are involved: S1, calculate the relative position relationship between the current input Bayer image and the adjacent cell images at different directions; S2, calculate the mapping relationship of the current input Bayer image in the canvas according to the relative position relationship; S3, image fusion based on bayer graph; S4, paste the fused Bayer map onto the canvas; The S1 specifically includes: S11, splitting the current input Bayer image and each adjacent cell image into four-channel RGGB images respectively; S12, respectively calculating the relative position relationship between the current input Bayer image and the segmented images of each adjacent cell image under the same channel; S13, merging the relative positional relationships under each channel to obtain the relative positional relationship between the current input Bayer image and each adjacent cell image in an xy coordinate system; The S2 also includes error elimination of the relative position relationship, specifically including: S21, construct an error hiding tree V_stichs, where V_stichs[i] represents the i-th tree node, and each node contains the x and y coordinate positions of an image; S22, calculating the x and / or y coordinate position difference between the tree node corresponding to the nth image and its adjacent tree node. If the difference is greater than a maximum threshold, it means that the position deviation between the nth image structure and the adjacent image structure in a certain direction exceeds the maximum threshold, wherein the nth image corresponds to the hidden tree V_stichs[n]; S23, diffusing the difference along the certain direction to the image in the global vertical direction; Among them, i and n are both natural numbers; The image fusion method of S3 is feathering, which specifically includes: Overlapping pixels are blended using a weighted average color value alpha factor, which has a value of 1 at the center pixel and then becomes 0 as it decreases linearly with the boundary pixels; The four channels of RGGB are fused separately, and then the separately fused images are merged into a Bayer graph; The S3 specifically includes: S31, setting the image width and height of the bayer images B1 and B2 to be fused; S32, decompose the bayer image into four channels: R, G1, G2, and B; S33, using the mapping relationship calculated in step S2 as the fusion input position; Specifically, if the position of B1 is pox_x1, pos_y1, and the position of B2 is pos_x2, pos_y2, the following position transformation is required before fusion: p_x1 = pox_x1 / 2×2 p_y1 = pox_y1 / 2×2 p_x2 = pox_x2 / 2×2 p_y2 = pox_y2 / 2×2 S34, using the positions calculated in step S33, the four-channel images are fused respectively.

2. The fast splicing method based on Bayer images according to claim 1, characterized in that: The S12-S13 specifically include: The standard squared difference is used to calculate the minimum value of the current input Bayer image and each adjacent cell image in each channel. Each minimum value corresponds to a candidate matching position in the xy coordinate system. The corresponding candidate matches are synthesized to obtain the relative position relationship between the previous input Bayer image and the adjacent cell images.

3. The fast splicing method based on bayer images according to claim 1, characterized in that: The S23 specifically includes: The average diffusion method is adopted to complete the error diffusion within two scanning lines; After the error of the current row is eliminated, the second row replaces the first row as a reference row to participate in the subsequent error diffusion.

4. A fast splicing system based on Bayer images, characterized in that: The system is used to implement the fast stitching method based on Bayer images as described in any one of claims 1 to 3, comprising: A relative position relationship calculation module is used to calculate the relative position relationship between the current input Bayer image and adjacent cell images at different directions; A mapping relationship calculation module is used to calculate the mapping relationship of the current input Bayer image in the canvas according to the relative position relationship; Image fusion module, used for image fusion based on Bayer graph; The canvas fitting module is used to paste the fused Bayer map onto the canvas.

5. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the processor is used to implement the steps of the fast splicing method based on bayer images as described in any one of claims 1 to 3 when executing a computer management program stored in the memory.

6. A computer-readable storage medium, characterized in that: A computer management program is stored thereon, and when the computer management program is executed by the processor, the steps of the fast splicing method based on bayer images as described in any one of claims 1 to 3 are implemented.

Images