A large volume three-dimensional image stitching method, device, computer equipment and medium

Abstract

The present invention discloses a large-volume three-dimensional image stitching method, device, computer equipment and medium, the method comprising the following steps: obtaining a stack matrix of three-dimensional images to be stitched, and extracting the to-be-stitched boundaries of the sub-three-dimensional images to be stitched; performing fusion calculation on the adjacent to-be-stitched boundaries of all adjacent sub-three-dimensional images to be stitched by a sigmoid function to obtain a smooth boundary; performing boundary clipping, dimension conversion and shape conversion on the stack matrix of three-dimensional images to be stitched according to the boundary to be stitched to obtain a three-dimensional matrix; covering all smooth boundaries with the boundary junctions in the three-dimensional matrix to obtain a seamlessly stitched three-dimensional image. The present invention relates to the field of computer vision and image processing technology, and the boundary fusion is performed by a sigmoid function and then covered in a shape-transformed three-dimensional matrix, which not only improves the stitching quality but also reduces the stitching time.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and more specifically to a large-volume three-dimensional image splicing technology. Background Art

[0002] With the development of computer vision and image processing technology, 3D image stitching technology plays an increasingly important role in medical imaging, geographic information system (GIS), virtual reality (VR) and augmented reality (AR). Among them, large-volume 3D image stitching has attracted much attention due to its application value in complex scenes.

[0003] Large-volume 3D image stitching refers to the process of synthesizing multiple 3D data sets or images into a complete 3D model through computational processing. This process often involves complex algorithm design and high-performance computing power. Although 3D image stitching technology has made certain progress, there are still many challenges to achieve seamless stitching of large-volume images. First, due to the huge amount of data, how to efficiently store and process this data has become a key issue. Secondly, the image blocks need to be accurately matched during the stitching process, especially at the edges and details, which requires a highly accurate registration algorithm to reduce errors.

[0004] Therefore, achieving rapid and seamless stitching of large-volume three-dimensional images is an urgent problem that needs to be solved by those skilled in the art. Summary of the invention

[0005] In view of this, the present invention provides a large-volume three-dimensional image stitching method, device, computer equipment and medium to achieve seamless stitching of three-dimensional images with higher efficiency and better effect.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a large volume three-dimensional image stitching method, comprising the following steps:

[0008] S1. Obtain the 3D image stack matrix to be stitched , and extracting the to-be-joined boundaries of the to-be-joined sub-3D images;

[0009] S2, using a sigmoid function to perform fusion calculation on the adjacent to-be-joined boundaries of all adjacent to-be-joined sub-3D images to obtain a smooth boundary;

[0010] S3, stacking matrices of the three-dimensional images to be spliced ​​according to the boundaries to be spliced Perform boundary clipping, dimension conversion and shape conversion to obtain a three-dimensional matrix;

[0011] S4. Cover all smooth boundary correspondences at boundary junctions in the three-dimensional matrix to obtain a seamlessly spliced ​​three-dimensional image.

[0012] Furthermore, step S1 specifically includes:

[0013] S11. Obtaining the 3D image stack matrix to be stitched ; Wherein, the three-dimensional image stack matrix to be spliced Including all sub-3D images to be stitched;

[0014] S12: Extracting the first boundaries of all sub-3D images to be spliced , the second boundary , the third boundary , the fourth boundary ;

[0015] The border widths are all overlap, and the first border of each to-be-joined 3D sub-image is adjacent to the second border and the fourth border, and opposite to the third border.

[0016] Furthermore, the sigmoid function in step S2 is:

[0017]

[0018] Among them, x represents the position index of each element in the boundary to be calculated, and y represents the fusion weight of each element in the boundary to be fused after calculation; in the fusion calculation process, the larger the boundary row and column index, that is, the farther the ion image is, the smaller the calculated fusion weight is.

[0019] Furthermore, in step S2, two adjacent boundaries to be spliced ​​of each sub-three-dimensional image to be spliced ​​are fused and calculated to obtain a smooth boundary; specifically, the step includes:

[0020] Splicing the first boundary of the to-be-joined sub-three-dimensional image in the a-th row and the b-th column and the third boundary of the to-be-joined sub-three-dimensional image in the a-th row and the b±1-th column to obtain a corresponding first smooth boundary;

[0021] The calculation formula of the first smooth boundary is:

[0022]

[0023] Splicing the second boundary of the to-be-joined sub-three-dimensional image in the a-th row and the b-th column and the fourth boundary of the to-be-joined sub-three-dimensional image in the a±1-th row and the b-th column to obtain a corresponding second smooth boundary;

[0024] The calculation formula of the second smooth boundary is:

[0025]

[0026] Among them, 1≤a≤num y , 1≤b≤num x , num x is the number of sub-3D images to be stitched in the horizontal direction, num y is the number of sub-3D images to be stitched in the vertical direction; d k represents the k-th layer image of the sub-3D image to be stitched, h i represents the i-th row image of the sub-3D image to be spliced, w j represents the j-th column image of the sub-3D image to be spliced; 1≤k≤d, 1≤i≤h, 1≤j≤w, d is the pixel depth of the sub-3D image to be spliced, h is the pixel height of the sub-3D image to be spliced, and w is the pixel width of the sub-3D image to be spliced.

[0027] Further, step S3 specifically includes:

[0028] S31, cutting off the borders of all the sub-3D images to be spliced pixels, and obtain the cropped 3D image stack;

[0029] S32, performing a transpose transformation on the three-dimensional image stack to change the dimensional order and obtain a high-dimensional array;

[0030] S33, construct an empty array with the same shape as the high-dimensional array; and use multithreading to fill each matrix into the empty array in parallel along the d dimension of the high-dimensional array; reshape the filled empty array to change the number and size of dimensions, and convert it into a three-dimensional matrix with unsmoothed boundaries.

[0031] Further, step S4 specifically includes:

[0032] The first smoothed boundaries and the second smoothed boundaries of all the sub-three-dimensional images to be spliced ​​obtained in step S2 are correspondingly covered at the boundary junctions in the three-dimensional matrix to obtain a seamlessly spliced ​​three-dimensional image.

[0033] In a second aspect, the present invention provides a large-volume three-dimensional image stitching device, comprising the following modules:

[0034] Extraction module: used to obtain the 3D image stack matrix to be stitched , and extracting the to-be-joined boundaries of the to-be-joined sub-3D images;

[0035] Fusion module: used to perform fusion calculation on the adjacent to-be-joined boundaries of all adjacent to-be-joined sub-3D images through the sigmoid function to obtain a smooth boundary;

[0036] Conversion module: used to convert the matrix of the three-dimensional image stack to be spliced ​​according to the boundary to be spliced Perform boundary clipping, dimension conversion and shape conversion to obtain a three-dimensional matrix;

[0037] The stitching module is used to cover all smooth boundary correspondences at the boundary junctions in the three-dimensional matrix to obtain a seamlessly stitched three-dimensional image.

[0038] In a third aspect, the present invention provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0039] Memory, used to store computer programs;

[0040] The processor, when used to execute the program stored in the memory, can implement the large-volume three-dimensional image stitching method as described in any one of the first aspects.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the processor executes the large-volume three-dimensional image stitching method as described in any one of the first aspects.

[0042] The descriptions of the second to fourth aspects of the present invention can refer to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second to fourth aspects can refer to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0043] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a large volume three-dimensional image stitching method, which has the following beneficial effects:

[0044] 1. The present invention only calculates the boundaries to be spliced, which greatly reduces the amount of data involved in the calculation process and avoids the situation where the computer repeatedly applies for a large amount of memory to store intermediate calculation results, resulting in low calculation efficiency and large memory usage.

[0045] 2. The present invention uses sigmoid weighting on the splicing boundary area to achieve a smooth and uniform transition between adjacent blocks.

[0046] 3. The present invention significantly reduces the time required to apply for memory to store new arrays when continuously performing transpose and reshape operations on high-dimensional arrays through multi-threaded copy operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 The present invention provides a flowchart of a large-volume three-dimensional image stitching method.

[0049] Figure 2 A schematic diagram of a large-volume three-dimensional image stitching method provided by the present invention.

[0050] Figure 3 This is a block diagram of the large-volume three-dimensional image stitching device provided by the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Example 1

[0053] The embodiment of the present invention discloses a large volume three-dimensional image stitching method, referring to Figure 1 As shown, the following steps are included:

[0054] S1. Obtain the 3D image stack matrix to be stitched , and extracting the to-be-joined boundaries of the to-be-joined sub-3D images;

[0055] S2, using a sigmoid function to perform fusion calculation on the adjacent to-be-joined boundaries of all adjacent to-be-joined sub-3D images to obtain a smooth boundary;

[0056] S3, stacking matrices of the three-dimensional images to be spliced ​​according to the boundaries to be spliced Perform boundary clipping, dimension conversion and shape conversion to obtain a three-dimensional matrix;

[0057] S4. Cover all smooth boundary correspondences at boundary junctions in the three-dimensional matrix to obtain a seamlessly spliced ​​three-dimensional image.

[0058] In the field of medical imaging, a series of two-dimensional slice images of the brain are generated through CT scanning or MRI; these images are converted into three-dimensional data through surface reconstruction, volume rendering or segmentation technology, and are appropriately cropped and resized; image data from different imaging devices are fused to obtain more comprehensive patient information. This example obtains 35 16-bit three-dimensional images with a shape of 63*2000*2000 for seamless stitching tasks. Figure 2 As shown, first determine the position of the boundaries that need to be spliced, and extract the pixel data of these boundaries; use the Sigmoid function to fuse the boundaries between the two image blocks to be spliced, in order to make the transition more natural, and only the boundaries are involved, operate in situ, avoid repeated memory requests, and reduce resource and time consumption; at the same time, after the stack matrix composed of all sub-three-dimensional images to be spliced ​​is cropped and dimensionally transformed, multi-threaded copy operations and shape conversions are performed to change the number and size of dimensions. At this time, all sub-three-dimensional images are spliced ​​together, but no smoothing processing has been applied; finally, the previously calculated smoothed boundaries are replaced to the positions of the original splicing points to achieve a seamless splicing effect, and a complete, high-resolution three-dimensional brain model is obtained.

[0059] In this embodiment, the brain image stack matrix to be stitched The shape is ; Among them, num y 、num x are the number of large brain 3D images in the vertical and horizontal directions, d is the pixel depth of a single sub-3D image, h and w are the pixel height and width of a single sub-3D image, respectively; in this embodiment, d=63, h=2000, w=2000, In this embodiment, , .

[0060] The following are detailed descriptions of each of the above steps:

[0061] Step S1: First, the upper, lower, left, and right boundaries of all brain sub-3D images ( , , , )overlap pixels are copied for subsequent use, and the right boundary is used as the first boundary, the upper boundary is used as the second boundary, the left boundary is used as the third boundary, and the lower boundary is used as the fourth boundary; where 1≤overlap<2000; their respective shapes are , , , .

[0062] In this embodiment, assuming overlap=100, the respective shapes are 5x7x63x100x2000, 5x7x63x100x2000, 5x7x63x2000x100, and 5x7x63x2000x100; the smaller the overlap value is, the fewer the pixel values ​​for boundary fusion and smoothing calculations are, the less calculations are done, and the shorter the calculation time is.

[0063] In step S2, in order to erase the seams between adjacent 3D images of the brain sub-images, it is necessary to perform smoothing calculations on the corresponding boundaries. The smoothing operation can be achieved by performing sigmoid fusion on the adjacent boundaries. The sigmoid function is:

[0064]

[0065] Among them, x represents the position index of each element in the boundary to be calculated, and y represents the fusion weight of each element in the boundary to be fused after calculation; in the fusion calculation process, the larger the boundary row and column index, that is, the farther the ion image is, the smaller the calculated fusion weight is, and vice versa.

[0066] by Take the two brain sub-three-dimensional images in the first row, first column and second column as an example. Since the sigmoid function is monotonically decreasing, the actual calculation process can be expressed as follows:

[0067]

[0068] Among them, d k represents the k-th layer image, h i represents the i-th row of the image, and traverses k from 1 to 63 and i from 1 to 2000 in sequence to complete a smoothing calculation between adjacent blocks. Then the above calculation process is extended to All adjacent blocks in .

[0069] In this embodiment, the upper boundary smoothing boundary and the right boundary smoothing boundary of all the sub-3D images to be spliced ​​are calculated.

[0070] The calculation formula for the right boundary smoothing boundary is:

[0071]

[0072] The upper boundary smoothing boundary is calculated as:

[0073]

[0074] In this embodiment, 1≤a≤7, 1≤b≤5; d k represents the k-th layer image of the sub-3D image to be stitched, h iRepresents the i-th row of the sub-3D image to be stitched; 1≤k≤63, 1≤i≤2000.

[0075] This embodiment copies the boundaries and only calculates the brain area to be spliced, which greatly reduces the amount of data involved in the calculation process and avoids the situation where the computer repeatedly applies for huge memory to store intermediate calculation results, resulting in low calculation efficiency and large memory usage; this embodiment uses sigmoid weighting for overlapping areas to achieve smooth and uniform transition between adjacent blocks.

[0076] Step S3 specifically includes:

[0077] S31, sub-3D image boundary clipping: In order to make the original The total number of pixels is the same as the number of pixels after stitching. Each sub-3D image is cut off from top to bottom and left to right pixels, that is, cut off 50 pixels, and the shape is 3D image stack , that is, the shape is 5x7x63x(2000-100)x(2000-100).

[0078] S32, 3D image stack dimension conversion: convert the shape to 3D image stack After transpose, the order of dimensions is changed and converted to shape High-dimensional array of , that is, the shape is 63x5x(2000-100)x7x(2000-10).

[0079] S33, multi-threaded replication and shape conversion: after getting After that, create a new shape An empty array , using multithreading along The d dimension is parallelized Each shape of Fill in the matrix After filling in the information, After the reshape operation, the product remains unchanged in terms of the number and size of dimensions, and is converted to a shape of The three-dimensional matrix , this three-dimensional matrix is ​​the three-dimensional image 63x{5x(2000-100)}x{7x(2000-100)} completed by stitching the unsmoothed boundaries between the sub-images.

[0080] Finally, step S4 is performed to cover the upper and right smoothed boundaries of all the sub-three-dimensional images to be spliced ​​obtained in step S2 with the three-dimensional matrix obtained in step S3. The boundary and position pixels in the image can be combined to obtain a seamless three-dimensional image.

[0081] In this embodiment, only the boundaries are involved in the smoothing calculation operation, and the operation is performed in situ, avoiding repeated memory requests and reducing resource and time consumption; transpose and reshape are decoupled and connected with multi-threaded copy operations, which greatly reduces the array dimension conversion and shape transformation time.

[0082] Example 2

[0083] The embodiment of the present invention discloses a large volume three-dimensional image stitching device, referring to Figure 3 As shown, it includes the following modules:

[0084] Extraction module: used to obtain the 3D image stack matrix to be stitched , and extracting the to-be-joined boundaries of the to-be-joined sub-3D images;

[0085] Fusion module: used to perform fusion calculation on the adjacent to-be-joined boundaries of all adjacent to-be-joined sub-3D images through the sigmoid function to obtain a smooth boundary;

[0086] Conversion module: used to convert the matrix of the three-dimensional image stack to be spliced ​​according to the boundary to be spliced Perform boundary clipping, dimension conversion and shape conversion to obtain a three-dimensional matrix;

[0087] The stitching module is used to cover all smooth boundary correspondences at the boundary junctions in the three-dimensional matrix to obtain a seamlessly stitched three-dimensional image.

[0088] In this embodiment, the device realizes fast large-volume three-dimensional image seamless stitching, which is crucial to improving image quality and applicability of application scenarios. Seamless stitching can ensure that the final synthesized three-dimensional model has higher accuracy and completeness, which is particularly important for medical diagnosis, scientific research and other fields. For example, in medicine, high-quality three-dimensional models help doctors understand the location and morphology of lesions more clearly, so as to formulate more accurate treatment plans, and the rapid realization of the above-mentioned seamless stitching process will greatly shorten the data processing time and improve work efficiency, which is especially valuable for time-sensitive applications such as clinical diagnosis.

[0089] Example 3

[0090] The present invention further provides a computer device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0091] When the processor executes the program, the large volume three-dimensional image stitching method in Embodiment 1 is implemented. For the sake of brevity, it will not be described in detail here.

[0092] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0093] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0094] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0095] Example 4

[0096] The present invention further provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the large-volume three-dimensional image stitching method as described in any one of the embodiments in Example 1.

[0097] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0098] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large volume three-dimensional image stitching method, characterized in that: The following steps are involved: S1. Obtain the 3D image stack matrix to be stitched , and extracting the to-be-joined boundaries of the to-be-joined sub-3D images; S2, using a sigmoid function to perform fusion calculation on the adjacent to-be-joined boundaries of all adjacent to-be-joined sub-3D images to obtain a smooth boundary; S3, stacking matrices of the three-dimensional images to be spliced ​​according to the boundaries to be spliced Perform boundary clipping, dimension conversion and shape conversion to obtain a three-dimensional matrix; S4. Cover all smooth boundary correspondences at boundary junctions in the three-dimensional matrix to obtain a seamlessly spliced ​​three-dimensional image.

2. A large volume three-dimensional image stitching method as claimed in claim 1, characterized in that: Step S1 specifically includes: S11. Obtaining the 3D image stack matrix to be stitched ; Wherein, the three-dimensional image stack matrix to be spliced Including all sub-3D images to be stitched; S12: Extracting the first boundaries of all sub-3D images to be spliced , the second boundary , the third boundary , the fourth boundary ; The border widths are all overlap, and the first border of each to-be-joined 3D sub-image is adjacent to the second border and the fourth border, and opposite to the third border.

3. A large volume three-dimensional image stitching method as claimed in claim 2, characterized in that: The sigmoid function in step S2 is: ； Among them, x represents the position index of each element in the boundary to be calculated, and y represents the fusion weight of each element in the boundary to be fused after calculation; in the fusion calculation process, the larger the boundary row and column index, that is, the farther the ion image is, the smaller the calculated fusion weight is.

4. A large volume three-dimensional image stitching method as claimed in claim 3, characterized in that: In step S2, two adjacent boundaries to be spliced ​​of each sub-three-dimensional image to be spliced ​​are fused and calculated to obtain a smooth boundary; specifically, the step includes: Splicing the first boundary of the to-be-joined sub-three-dimensional image in the a-th row and the b-th column and the third boundary of the to-be-joined sub-three-dimensional image in the a-th row and the b±1-th column to obtain a corresponding first smooth boundary; The calculation formula of the first smooth boundary is: ； Splicing the second boundary of the to-be-joined sub-three-dimensional image in the a-th row and the b-th column and the fourth boundary of the to-be-joined sub-three-dimensional image in the a±1-th row and the b-th column to obtain a corresponding second smooth boundary; The calculation formula of the second smooth boundary is: ； Among them, 1≤a≤num y , 1≤b≤num x , num x is the number of sub-3D images to be stitched in the horizontal direction, num y is the number of sub-3D images to be stitched in the vertical direction; d k represents the k-th layer image of the sub-3D image to be stitched, h i represents the i-th row image of the sub-3D image to be spliced, w j represents the j-th column image of the sub-3D image to be spliced; 1≤k≤d, 1≤i≤h, 1≤j≤w, d is the pixel depth of the sub-3D image to be spliced, h is the pixel height of the sub-3D image to be spliced, and w is the pixel width of the sub-3D image to be spliced.

5. A large volume three-dimensional image stitching method as claimed in claim 4, characterized in that: Step S3 specifically includes: S31, cutting off the borders of all the sub-3D images to be spliced pixels, and obtain the cropped 3D image stack; S32, performing a transpose transformation on the three-dimensional image stack to change the dimensional order and obtain a high-dimensional array; S33, construct an empty array with the same shape as the high-dimensional array; and use multithreading to Each matrix is ​​filled into the empty array in parallel in terms of dimension; the filled empty array is reshaped to change the number and size of dimensions and converted into a three-dimensional matrix with unsmoothed boundaries.

6. A large volume three-dimensional image stitching method as claimed in claim 5, characterized in that: Step S4 specifically includes: The first smoothed boundaries and the second smoothed boundaries of all the sub-three-dimensional images to be spliced ​​obtained in step S2 are correspondingly covered at the boundary junctions in the three-dimensional matrix to obtain a seamlessly spliced ​​three-dimensional image.

7. A large volume three-dimensional image stitching device, characterized in that: Includes the following modules: Extraction module: used to obtain the 3D image stack matrix to be stitched , and extracting the to-be-joined boundaries of the to-be-joined sub-3D images; Fusion module: used to perform fusion calculation on the adjacent to-be-joined boundaries of all adjacent to-be-joined sub-3D images through the sigmoid function to obtain a smooth boundary; Conversion module: used to convert the matrix of the three-dimensional image stack to be spliced ​​according to the boundary to be spliced Perform boundary clipping, dimension conversion and shape conversion to obtain a three-dimensional matrix; The stitching module is used to cover all smooth boundary correspondences at the boundary junctions in the three-dimensional matrix to obtain a seamlessly stitched three-dimensional image.

8. A computer device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor, when used to execute the program stored in the memory, can implement the large-volume three-dimensional image stitching method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, enable the processor to execute the large-volume three-dimensional image stitching method according to any one of claims 1 to 6.

Images