Grain image stitching method, device, and electronic device

By constructing a layer-by-layer stitching method of the initial anchor area and peripheral closed-loop images, the misalignment problem caused by the position offset of the highest frequency response in the grain image stitching is solved, and a more accurate and complete image stitching effect is achieved.

CN116051370BActive Publication Date: 2024-11-12창추안 테크놀로지 (수저우) 컴퍼니 리미티드
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202211648944.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-21
Publication Date
2024-11-12
Estimated Expiration
2042-12-21

AI Technical Summary

Technical Problem

In the prior art, when splicing grain images, splicing based on the highest frequency response position of the image to be spliced ​​can easily lead to image misalignment, especially when the CIS type grain overlap area is repeated more, it is difficult to achieve accurate splicing.

Method used

By obtaining each sub-image to be spliced ​​in the grain image, determining its distribution and constructing an initial anchor area, acquiring the peripheral closed-loop images of each layer, and splicing them from the inside to the outside with the initial anchor area as the center, fusion layer by layer to form a target Die image, optimized stitching offset vectors to reduce duplicate features and error rates.

Benefits of technology

It effectively reduces the image stitching error rate, ensures the accuracy and completeness of the stitching results, and avoids the misalignment problem caused by the position offset of the highest frequency response.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116051370B_ABST
    Figure CN116051370B_ABST
Patent Text Reader

Abstract

The present application discloses a grain image stitching method, device, and electronic device. The method includes: obtaining each sub-image to be stitched of the grain image; determining the distribution of each sub-image to be stitched, and determining the initial anchor area according to the distribution; obtaining each layer of peripheral closed-loop images, each layer of peripheral closed-loop images is obtained by stitching the sub-images to be stitched that are radiated from the inside to the outside with the initial anchor area as the center; stitching the initial anchor area with each layer of peripheral closed-loop images to form a target grain image, wherein the initial anchor area is stitched and fused with the adjacent peripheral closed-loop images, and then stitched and fused with the adjacent peripheral closed-loop images, and stitched in sequence until it is stitched with the outermost layer of peripheral closed-loop images to form a target grain image. The present application solves the technical problem that the stitched image is prone to misalignment due to the fact that the position corresponding to the highest frequency response of the image to be stitched is generally used as the stitching offset position for stitching in the related art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image stitching, and in particular to a grain image stitching method, device, and electronic device. Background Art

[0002] For the detection of a single die, the actual field of view images of different sizes of different die under different magnifications of the camera may not be consistent, resulting in the camera being unable to cover the entire die in a single shot. Therefore, it is necessary to divide a single die into multiple fields of view for shooting, and finally, stitch multiple images corresponding to multiple fields of view to obtain a complete die image for detection. The feature areas between multiple images corresponding to multiple fields of view will be highly repeated or have fewer features, which is not conducive to completing the final stitching of complete die.

[0003] In the related art, during stitching, the phase correlation method is generally used based on the images to be stitched, that is, the position corresponding to the highest frequency response of the solved mutual power spectrum image is used as the stitching offset position for stitching. However, for certain types of grains, for example, for images with a large number of repeated overlapping areas of grains such as CIS, the highest frequency response is not unique, which can easily lead to misalignment of the stitched image.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a grain image stitching method, device, and electronic device to at least solve the technical problem that stitched images are easily misaligned due to the fact that stitching in the related art generally uses the position corresponding to the highest frequency response of the image to be stitched as the stitching offset position.

[0006] According to one aspect of an embodiment of the present application, a method for stitching grain images is provided, comprising: acquiring each sub-image to be stitched of the grain image; determining the distribution of each sub-image to be stitched, and determining an initial anchor area according to the distribution; acquiring each layer of peripheral closed-loop images, wherein each layer of peripheral closed-loop images is obtained by stitching sub-images to be stitched that are distributed radiating from the inside to the outside with the initial anchor area as the center; stitching the initial anchor area with each layer of peripheral closed-loop images to form a target grain image, wherein the initial anchor area is stitched and fused with adjacent peripheral closed-loop images, and then stitched and fused with adjacent peripheral closed-loop images, and stitched in sequence until it is stitched with the outermost layer of peripheral closed-loop images to form a target grain image.

[0007] Optionally, obtaining peripheral closed-loop images of each layer includes: splicing adjacent sub-images to be spliced ​​to obtain a closed-loop image based on theoretical overlap amounts of adjacent sub-images to be spliced; obtaining multiple sets of candidate quantities of the closed-loop image, and determining a definite relative offset vector of adjacent overlapping areas based on optimization of each set of candidate quantities; and fusing overlapping areas of adjacent sub-images to be spliced ​​based on the definite relative offset vector of adjacent overlapping areas to obtain a peripheral closed-loop image.

[0008] Optionally, multiple groups of alternative quantities for the closed-loop image are obtained, and based on the optimization of each group of alternative quantities, a definite relative offset vector of the adjacent overlapping area is determined, including: obtaining multiple groups of alternative quantities for the closed-loop image, wherein each group of alternative quantities has N relative offset vectors, and N is a natural number; selecting a target offset vector within each group of alternative quantities so that the sum of the relative offsets of the closed-loop image approaches 0, wherein the target offset vector is the definite relative offset vector of the adjacent overlapping area.

[0009] Optionally, selecting a target offset vector in each group of candidate quantities includes: calling a path planning algorithm to solve a relative offset vector with a minimum path in each group of candidate quantities; and determining the relative offset vector with a minimum path as the target relative offset vector.

[0010] Optionally, the initial anchor area is spliced ​​with the peripheral closed-loop images of each layer to form a target grain image, including: obtaining multiple pairs of boundary overlapping areas between the initial anchor area and adjacent peripheral closed-loop images; obtaining the optimal stitching offset of each overlapping area to minimize the pixel difference of the overlapping images; based on each optimal stitching offset, fusing the multiple pairs of overlapping areas of the peripheral closed-loop image and the initial anchor area as the first anchor area, and the first anchor area is used to splice with its adjacent peripheral closed-loop image.

[0011] Optionally, before obtaining multiple pairs of boundary overlapping areas between the initial anchor area and the adjacent peripheral closed-loop images, it also includes: placing the initial anchor area at the center of the inscribed frame of the adjacent peripheral closed-loop image; controlling the difference in overlapping area or ratio between the initial anchor area and the peripheral closed-loop image in each direction to be within a preset range.

[0012] Optionally, after obtaining the target grain image, it also includes: obtaining a sample template corresponding to the target grain image, wherein two feature templates for positioning the size are arranged on the sample template along the diagonal direction, and the distance between the two feature templates along the horizontal axis is a first distance, and the distance along the vertical axis is a second distance; obtaining the corresponding positions of the two feature templates in the target grain image, determining a third distance between the two feature templates in the target grain image along the horizontal axis, and a fourth distance along the vertical axis; obtaining a first difference between the first distance and the third distance, and optimizing the size of the target grain image along the horizontal axis based on the first difference; obtaining a second difference between the second distance and the fourth distance, and optimizing the size of the target grain image along the vertical axis based on the second difference.

[0013] Optionally, optimizing the size of the target grain image along the horizontal axis based on the first difference includes: taking the positions of the sub-images to be spliced ​​in the first column of the target grain image as the reference position, determining the number of sub-images to be spliced ​​included in each column of the target grain image except the first column; calculating a first ratio of the first difference to the number of sub-images to be spliced ​​included in each column; obtaining the column number of each column, and determining a first offset distance corresponding to each column of sub-images to be spliced ​​in the target grain image according to the first ratio and the column number; and moving each column of sub-images to be spliced ​​along the horizontal axis based on the first offset distance corresponding to each column of sub-images to be spliced ​​to optimize the size of the target grain image along the horizontal axis.

[0014] Optionally, optimizing the size of the target grain image along the longitudinal axis based on the second difference includes: taking the positions of each sub-image in the first row of the target grain image as the reference position, determining the number of sub-images included in each row of the target grain image except the first row; calculating a second ratio between the second difference and the number of sub-images included in each row; obtaining the row number of each row, and determining a second offset distance corresponding to each row of sub-images in the target grain image according to the second ratio and the row number; and moving each row of sub-images along the longitudinal axis based on the second offset distance corresponding to each row of sub-images to optimize the size of the target grain image along the longitudinal axis.

[0015] Optionally, determining the distribution of each sub-image to be spliced, and determining the initial anchor area according to the distribution, includes: obtaining the distribution quantity of each sub-image to be spliced ​​along the horizontal axis direction and the vertical axis direction of the grain image; determining the initial anchor area according to the distribution quantity of the sub-image to be spliced ​​along the horizontal axis direction and the vertical axis direction of the grain image.

[0016] Optionally, the initial anchor area is determined according to the distribution number of sub-images to be spliced ​​along the horizontal and vertical directions of the grain image, including: when the distribution number of sub-images to be spliced ​​along the horizontal and vertical directions of the grain image is an even number, determining not to set the initial anchor area; when the distribution number of sub-images to be spliced ​​along the horizontal and vertical directions of the grain image is an odd number, determining a single sub-image to be spliced ​​corresponding to the center position of the grain image as the initial anchor area; when the distribution number of sub-images to be spliced ​​along the horizontal direction of the grain image is an even number and the distribution number in the vertical direction is an odd number, or when the distribution number of sub-images to be spliced ​​along the horizontal direction of the grain image is an odd number and the distribution number in the vertical direction is an even number, determining multiple sub-images to be spliced ​​corresponding to the center position of the grain image as the initial anchor area.

[0017] Optionally, when the initial anchor region is a plurality of sub-images to be spliced, the initial anchor region splicing is completed according to the relative offset vectors of the plurality of sub-images to be spliced.

[0018] According to another aspect of the embodiment of the present application, a grain image stitching device is provided, comprising: a first acquisition module for acquiring each sub-image to be stitched of the grain image; a determination module for determining the distribution of each sub-image to be stitched, and determining the initial anchor area according to the distribution; a second acquisition module for acquiring each layer of peripheral closed-loop images, each layer of peripheral closed-loop images being obtained by stitching together a plurality of sub-images to be stitched that are radially distributed from the inside to the outside with the initial anchor area as the center; a stitching module for stitching together the initial anchor area with each layer of peripheral closed-loop images to form a target grain image, wherein the initial anchor area is stitched and fused with adjacent peripheral closed-loop images, and then stitched and fused with adjacent peripheral closed-loop images, and stitched in sequence until it is stitched with the outermost peripheral closed-loop image to form a target grain image. According to another aspect of the embodiment of the present application, a non-volatile storage medium is provided, the storage medium comprising a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute any grain image stitching method.

[0019] According to another aspect of an embodiment of the present application, there is further provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement any one of the grain image stitching methods.

[0020] In an embodiment of the present application, an initial anchor area is constructed and continuously updated, by obtaining each sub-image to be spliced ​​of the grain image; determining the distribution of each sub-image to be spliced, and determining the initial anchor area according to the distribution; obtaining each layer of peripheral closed-loop images, each layer of peripheral closed-loop images is obtained by splicing the sub-images to be spliced ​​that are distributed radiating from the inside to the outside with the initial anchor area as the center; and splicing the initial anchor area with each layer of peripheral closed-loop images to form a target grain image; wherein, after the initial anchor area is spliced ​​and fused with the peripheral closed-loop images of the adjacent peripheral closed-loop images, it is spliced ​​and fused with the peripheral closed-loop images of the adjacent peripheral closed-loop images, and they are spliced ​​in sequence until they are spliced ​​with the outermost peripheral image to form the target grain image, thereby achieving the purpose of reducing the repetitive features of the image to be spliced, thereby achieving the technical effect of reducing the error rate of image splicing, and further solving the technical problem that the spliced ​​image is prone to misalignment due to the fact that the position corresponding to the highest frequency response of the image to be spliced ​​is generally used as the splicing offset position for splicing in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 It is a planar schematic diagram of detecting based on a monochrome sensor and detecting a CIS with a Bayer array in the related art;

[0023] Figure 2 It is a schematic diagram of imaging based on AOI system in related technology;

[0024] Figure 3 It is a schematic diagram of the spatial distribution of FOV used for stitching in the related art;

[0025] Figure 4 is a schematic flow chart of a grain image stitching method according to an embodiment of the present application;

[0026] Figure 5 is a schematic diagram of a splicing method of grain images according to the present application;

[0027] Figure 6 is a schematic diagram of a process for obtaining peripheral closed-loop images of each layer in some embodiments of the present application;

[0028] Figure 7 It is a schematic diagram of a process of obtaining multiple sets of candidate quantities of a closed-loop image and determining a relative offset vector of adjacent overlapping areas in an embodiment of the present application;

[0029] Figure 8 is a schematic diagram of a flow chart of selecting a target offset vector in each group of candidate quantities in an embodiment of the present application;

[0030] Fig. 9 It is a schematic diagram of a process of splicing the initial anchor area and the peripheral closed-loop images of each layer to form a target grain image in an embodiment of the present application;

[0031] Fig.10 is a schematic diagram of a general process for optimizing the size of a grain image after obtaining a target grain image in an embodiment of the present application;

[0032] Fig.11 is a schematic diagram of a process of optimizing the size of the target grain image along the horizontal axis direction based on the first difference in some embodiments of the present application;

[0033] Fig.12 is a schematic diagram of a process for optimizing the size of a target grain image along the longitudinal axis based on a second difference in some embodiments of the present application;

[0034] Fig.13 is a schematic diagram of the splicing algorithm flow in one embodiment of the present application;

[0035] Fig.14 is a schematic diagram of the initial anchor zone layout in an embodiment of the present application;

[0036] Fig.15 is a schematic diagram of the layout of a peripheral closed-loop image in an embodiment of the present application;

[0037] Fig.16is a schematic diagram of the splicing direction in an embodiment of the present application;

[0038] Fig.17 is a schematic diagram of a closed-loop overlapping area in an embodiment of the present application;

[0039] Fig.18 It is an optional dynamic programming directed graph in the embodiment of the present application;

[0040] Fig.19 This is a schematic diagram of left and right splicing based on alph fusion in one embodiment of the present application;

[0041] Fig. 20 is a schematic diagram of the fusion of the closed loop and the anchor area in an embodiment of the present application;

[0042] Fig.21 This is a schematic diagram of Die size optimization in one embodiment of the present application;

[0043] Fig. 22 is a structural schematic diagram of a grain image stitching device according to an embodiment of the present application;

[0044] Fig.23 is a schematic block diagram of an exemplary electronic device for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] In order to facilitate those skilled in the art to better understand the relevant embodiments of the present application, the technical terms or some nouns that may be involved in the relevant embodiments of the present application are explained as follows:

[0048] Die: Die refers to the crystal grain of the chip before it is packaged. It is a small piece (Die) cut from the silicon wafer (Wafer) by laser. Each Die is an independent functional chip, which will eventually be packaged as a unit to become our common chip.

[0049] Phase correlation method: Phase correlation can be used to detect the relative displacement between two images with the same content. It can be used to align images, but it does not have illumination invariance. It is based on the displacement theorem of Fourier transform: the Fourier transform of a translated function is simply the product of the Fourier transform of the untranslated function and an exponential factor with a linear phase, that is, the translation in the spatial domain will cause a phase shift in the spectrum in the frequency domain.

[0050] Image stitching is an important part in the field of integrated circuit inspection. In the field of wafer defect inspection, a single die is often used as the inspection unit. Due to the different sizes of different dies and the different sizes of the actual field of view images (Fov) of the camera under different magnifications, the camera cannot cover the entire die in a single shot, that is, the camera's FOV is smaller than the size of the die. At this time, it is necessary to divide a single die into multiple fields of view of the camera for separate shooting, and finally use image stitching technology to stitch multiple images separately to generate a complete die image.

[0051] Figure 1 This is a schematic diagram of a certain die being imaged in an automatic optical inspection (AOI) system. Figure 1 As shown, there are a lot of repetitive features in the form of thin lines. Figure 2 This is another die imaging diagram in the AOI system. Figure 2 As shown, there are a large number of point-like repeated features. For such a spliced ​​image with a large number of repeated features, when the phase correlation method is used for recognition, its highest frequency response is not unique, so it is easy to cause the spliced ​​image to be misaligned. It should be noted that Figure 1 and Figure 2 This is only an exemplary description of the application scenario of the present application (ie, the case where features are highly repeated), and the present application is also applicable to stitched images with fewer features.

[0052] Figure 3 This is a schematic diagram of the FOV spatial distribution used for stitching in related technologies, such as Figure 3As shown, during the grain surface defect detection process, especially under a high-power objective lens, the camera's FOV is smaller than the size of the grain (hereinafter referred to as Die) to be detected. At this time, multiple FOV images containing the Die part need to be stitched.

[0053] During the execution process, adjacent FOVs need to be stitched together first, and then global constraints are processed to prevent the stitching result from being deformed. This is usually a collinearity constraint, but collinearity is obviously not suitable for stitching images with fewer features. It can be seen that the following problems exist in the relevant technology:

[0054] 1) The features and regions between FOV captured images are highly repeated, and there are multiple optimal solutions for translation parameters, which is not conducive to stitching;

[0055] 2) After splicing is completed, the size is difficult to be constrained to be consistent.

[0056] In order to solve the above technical problems, an embodiment of the present application provides an embodiment of a method for stitching grain images. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0057] Figure 4 is a method for splicing grain images according to an embodiment of the present application, such as Figure 4 As shown, the method comprises the following steps:

[0058] Step S402, obtaining each sub-image to be spliced ​​of the grain image;

[0059] As mentioned above, in the field of wafer inspection, a single die is often used as the inspection unit. However, due to the different sizes of different die and the actual field of view image Fov limitation of cameras and other acquisition devices, the camera often cannot cover the entire die in a single shot, and then it is necessary to divide the single die into multiple fields of view of the camera for shooting, obtain multiple images, and stitch the multiple images to obtain the die image. Therefore, the sub-images to be stitched in step S402 are multiple sub-images to be stitched in the image obtained when shooting in one field of view of the camera. Multiple image stitching is the stitching of multiple sub-images to be stitched in multiple images.

[0060] Step S404, determining the distribution of each sub-image to be spliced, and determining the initial anchor area according to the distribution;

[0061] It should be noted that the distribution of each sub-image to be spliced ​​includes but is not limited to: the distribution quantity of each sub-image to be spliced ​​in the horizontal or vertical direction. The initial anchor area is the center position of the first splicing.

[0062] Step S406, obtaining peripheral closed-loop images of each layer, wherein the peripheral closed-loop images of each layer are obtained by stitching together a plurality of sub-images to be stitched that are radially distributed from the inside to the outside with the initial anchor area as the center;

[0063] It can be understood that each layer of peripheral closed-loop images is obtained by stitching multiple sub-images to be stitched within the same predetermined radius with the initial anchor area as the center. Each layer of peripheral closed-loop images is centered on the initial anchor area and is distributed in a circular or elliptical shape from the inside to the outside.

[0064] Step S408, the initial anchor area is spliced ​​with each layer of peripheral closed-loop images to form a target grain image, wherein the initial anchor area is spliced ​​and fused with adjacent peripheral closed-loop images, and then spliced ​​and fused with adjacent peripheral closed-loop images, and spliced ​​in sequence until it is spliced ​​with the outermost layer of peripheral closed-loop images to form a target grain image.

[0065] It should be noted that, during the stitching process, the peripheral closed-loop image adjacent to the initial anchor area needs to be stitched with the initial anchor area as the stitching center.

[0066] In the grain image stitching method, each sub-image to be stitched of the grain image is obtained; the distribution of each sub-image to be stitched is determined, and the initial anchor area is determined according to the distribution; each layer of peripheral closed-loop images is obtained, and each layer of peripheral closed-loop images is stitched by the sub-images to be stitched that are distributed radiating from the inside to the outside with the initial anchor area as the center; the initial anchor area is stitched with each layer of peripheral closed-loop images to form a target grain image; wherein, after the initial anchor area is spliced ​​and fused with the adjacent peripheral closed-loop images, it is spliced ​​and fused with the adjacent peripheral closed-loop images, and they are stitched in sequence until they are spliced ​​with the outermost layer of peripheral closed-loop images to form the target grain image, thereby achieving the purpose of reducing the repetitive features of the image to be stitched, thereby achieving the technical effect of reducing the error rate of image stitching, and further solving the technical problem that the stitched image is prone to misalignment due to the fact that the position corresponding to the highest frequency response of the image to be stitched is generally used as the stitching offset position for stitching in the related art.

[0067] In the above step S408, the target grain image is formed by splicing the initial anchor area with each layer of peripheral closed-loop images, including: splicing and fusing the initial anchor area with adjacent peripheral closed-loop images, updating the center of the anchor area as a new anchor area to be referenced for the next layer of peripheral closed-loop image splicing, and so on, to obtain the target grain image.

[0068] Figure 5 is a schematic diagram of the splicing method of the grain image according to the present application, such as Figure 5As shown in the figure, the fixed center part does not move, which serves as an important constraint to stabilize the global stitching structure; the center 0 is used as the initial anchor area during stitching, and a group of sub-images to be stitched around it can form a closed loop during stitching, that is, a peripheral closed-loop image is obtained; the peripheral closed-loop image is stitched and fused with the initial anchor area, and the fused image is used as the new anchor area for the next stitching; and so on, all the sub-images to be stitched are globally stitched; finally, the Die size is fine-tuned with the help of template matching to generate the final Die image.

[0069] Figure 6 is a schematic diagram of the process of obtaining peripheral closed-loop images of each layer in some embodiments of the present application, such as Figure 6 As shown, the process includes the following steps:

[0070] S602, stitching the adjacent sub-images to be stitched together to obtain a closed-loop image according to the theoretical overlap amount of the adjacent sub-images to be stitched together; wherein the theoretical overlap amount is calculated by subtracting the encoding positions of the two images taken by the camera;

[0071] S604, obtaining multiple groups of candidate quantities of the closed-loop image, and determining relative offset vectors of adjacent overlapping areas based on optimization of each group of candidate quantities;

[0072] It should be noted that the adjacent overlapping area is the overlapping area between adjacent sub-images to be spliced ​​in the closed-loop image, and the alternative quantities are relative offset vectors to be selected, wherein each group of alternative quantities includes N relative offset vectors, and in the process of optimizing each group of alternative quantities, it is necessary to select a relative offset vector from each group of alternative quantities so that the sum of the relative offsets of the entire closed-loop image tends to 0.

[0073] S606 , according to the determined relative offset vectors of the adjacent overlapping areas, the overlapping areas of the adjacent sub-images to be spliced ​​are merged to obtain a peripheral closed-loop image.

[0074] In the above step S604, a phase correlation algorithm may be used to determine the relative offset vector. The above phase correlation algorithm includes but is not limited to: a two-dimensional pulse function method.

[0075] It should also be noted that the stitching overlap of adjacent sub-images to be stitched is equal to the theoretical overlap plus the relative offset vector, wherein the theoretical overlap is calculated by subtracting the encoding positions of the two images taken by the camera. Theoretically, the position of the peak of the current two-dimensional pulse function can be the relative offset vector of the two images. In reality, since the features of the overlapping area between two adjacent images may be highly repetitive, or there are very few features, the two-dimensional pulse function has no obvious peak, and its maximum value may not be the optimal offset. Therefore, it is necessary to select larger values ​​of multiple two-dimensional pulse functions as alternative relative offset vectors. In the embodiments of the present application, the number of retained alternative relative offset vectors is generally required to be: n≥9, where n represents the number of retained alternative relative offset vectors.

[0076] Figure 7 FIG. 1 is a flow chart of determining a relative offset vector of adjacent overlapping areas in an exemplary embodiment of the present application. Figure 7 As shown, the process mainly includes the following steps:

[0077] S702, obtaining multiple groups of candidate quantities of the closed-loop image, wherein each group of candidate quantities has N relative offset vectors, where N is a natural number;

[0078] S704, selecting a target offset vector in each group of candidate quantities so that the sum of the relative offsets of the closed-loop image approaches 0, wherein the target offset vector is a determined relative offset vector of adjacent overlapping regions.

[0079] It can be understood that there are N relative offset vectors in each set of alternative quantities. Therefore, it is necessary to select a relative offset vector from each set of alternative quantities so that the sum of the relative offsets of the entire closed loop tends to 0, that is, to find the relative offset vector with the smallest path from each set of alternative quantities. It is easy to notice that the scheme adopted in the embodiment of the present application is the splicing method of the initial anchor area plus each layer of peripheral closed-loop images. This scheme can make the Die Fov splicing no longer rely solely on the highest frequency response of the adjacent image Fov, but can introduce multiple groups of frequency responses, and then use the constraint that the closed-loop offset sum is 0 for optimization, which greatly reduces the splicing error rate of the Die image.

[0080] Figure 8 is a schematic diagram of a flow chart of selecting a target offset vector in each group of candidate quantities in an optional embodiment of the present application, such as Figure 8 As shown, the process includes:

[0081] S802, calling a path planning algorithm to solve the minimum relative offset vector of the path in each group of candidate quantities;

[0082] S804: Determine the relative offset vector with the smallest path as the target relative offset vector.

[0083] Fig. 9 FIG. 1 is a schematic diagram of a process of splicing the initial anchor area and the peripheral closed-loop images of each layer to form a target grain image in some embodiments of the present application. Fig. 9 As shown, the process includes:

[0084] S902, obtaining multiple pairs of boundary overlapping regions between the initial anchor region and adjacent peripheral closed-loop images;

[0085] S904, obtaining the best stitching offset of each overlapping area to minimize the pixel difference of the overlapping images;

[0086] S906, according to each optimal stitching offset, merging the peripheral closed-loop image with multiple pairs of overlapping areas of the initial anchor area to form a first anchor area, and the first anchor area is used for stitching with its adjacent peripheral closed-loop image.

[0087] Optionally, before obtaining multiple pairs of boundary overlap regions between the initial anchor region and the adjacent peripheral closed-loop images, the initial anchor region may be placed at the center of the inner frame of the adjacent peripheral closed-loop images; and the difference in the overlap area or ratio between the initial anchor region and the peripheral closed-loop images in each direction is controlled to be within a preset range. For example, for the overlap area, the preset range may be between 0 and 4, and for the ratio, the above ratio may be 10% to 20%:

[0088] It should be noted that the algorithms that can be used in the above splicing process include but are not limited to: Alpha fusion algorithm.

[0089] Optionally, the initial anchor area is spliced ​​with each layer of peripheral closed-loop images to form a target grain image, including: the first anchor area is fused with its adjacent peripheral closed-loop image as the second anchor area, and the second anchor area is used to fuse with its adjacent peripheral closed-loop image, and they are spliced ​​in sequence to achieve the splicing of the initial anchor area and each layer of peripheral closed-loop images to form a target grain image.

[0090] As an optional implementation, when the initial anchor region is a plurality of sub-images to be spliced, the initial anchor region splicing may be completed according to the relative offset vectors of the plurality of sub-images to be spliced.

[0091] As an optional implementation, after obtaining the target grain image, in order to constrain the size of the grain image, the size of the spliced ​​target grain image is prevented from fluctuating, which may affect the performance and effect of grain defect detection in the subsequent process flow. Fig.10 As shown, the specific steps include:

[0092] S1001, obtaining a sample template corresponding to the target grain image, the sample template is as follows Fig.18 As shown, two feature templates for positioning dimensions are arranged on the sample template along the diagonal direction, the distance between the two feature templates along the horizontal axis direction is a first distance, and the distance along the vertical axis direction is a second distance;

[0093] For the stitching of grain images, since each grain image has its own defects, in order to facilitate detection while ensuring that all grains are of the same size, multiple grain images can be fused to obtain a fused grain image with no defects or fewer defects. The fused grain image is used as a sample template, which can be used as a standard image used in the grain detection process.

[0094] In some embodiments of the present application, the size of the sample template is the standard size corresponding to the grain, and the above-mentioned two feature templates are two ROI areas selected on the sample template. For example, they can be two rectangular boxes selected on the sample template. The above-mentioned two rectangular boxes are set along the diagonal of the sample template, one rectangular box is placed in the upper left corner of the sample template, and the other rectangular box is placed in the lower right corner of the sample template. The distance between the intersection of the diagonals of the two rectangular boxes along the horizontal axis is the first distance, and the distance along the vertical axis is the second distance.

[0095] S1002, obtaining corresponding positions of two feature templates in the target grain image, and determining a third distance along the horizontal axis and a fourth distance along the vertical axis of the two feature templates in the target grain image;

[0096] S1003, obtaining a first difference between the first distance and the third distance, and optimizing the size of the target grain image along the horizontal axis based on the first difference;

[0097] S1004, obtaining a second difference between the second distance and the fourth distance, and optimizing the size of the target grain image along the longitudinal axis based on the second difference.

[0098] It is easy to notice that the first distance and the third distance are the offsets of the sub-images of each column of the target grain image in the horizontal direction, and the second distance and the fourth distance are the offsets of the sub-images of each row of the target grain image in the vertical direction.

[0099] Fig.11 is a schematic diagram of a process for optimizing the size of the target grain image along the horizontal axis based on the first difference in some embodiments of the present application, such as Fig.11 As shown, the process includes:

[0100] S10031, taking the positions of the sub-images to be spliced ​​in the first column of the target grain image as reference positions, determining the number of sub-images to be spliced ​​in each column of the target grain image except the first column;

[0101] S10032, obtaining a first ratio of the first difference value to the number of sub-images to be spliced ​​included in each column;

[0102] S10033, obtaining the column number of each column, and determining a first offset distance corresponding to each column of the sub-images to be spliced ​​in the target grain image according to the first ratio and the column number;

[0103] S10034: Move each column of sub-images to be spliced ​​along the horizontal axis direction based on the first offset distance corresponding to each column of sub-images to be spliced ​​to optimize the size of the target grain image along the horizontal axis direction.

[0104] Similarly, in some embodiments of the present application, the size of the target grain image along the longitudinal axis may be optimized based on the second difference. Fig.12 is a schematic diagram of a process for optimizing the size of a target grain image along the longitudinal axis based on the second difference in some embodiments of the present application, such as Fig.12 As shown, the process includes:

[0105] S10041, taking the positions of the sub-images in the first row of the target grain image as reference positions, determining the number of sub-images included in each row of the target grain image except the first row;

[0106] S10042, obtaining a second ratio of the second difference to the number of sub-images included in each row;

[0107] S10043, obtaining the row number of each row, and determining a second offset distance corresponding to each row of sub-images in the target grain image according to the second ratio and the row number;

[0108] S10044, moving each row of sub-images along the longitudinal axis direction based on the second offset distance corresponding to each row of sub-images to optimize the size of the target grain image along the longitudinal axis direction.

[0109] It is easy to notice that by comparing the actual target grain size with the sample size, the actual target grain size can be further optimized to ensure that the spliced ​​target grain size is consistent with the sample size.

[0110] As an optional implementation, determining the distribution of each sub-image to be spliced ​​and determining the initial anchor area based on the distribution can be achieved through the following steps. Specifically, the distribution quantity of each sub-image to be spliced ​​along the horizontal axis and the vertical axis of the grain image can be obtained; then, the initial anchor area is determined based on the distribution quantity of the sub-image to be spliced ​​along the horizontal axis and the vertical axis of the grain image.

[0111] It should be noted that the initial anchor area is determined according to the distribution number of the sub-images to be spliced ​​along the horizontal axis direction and the vertical axis direction of the grain image, which specifically includes the following four cases:

[0112] 1. When the number of sub-images to be spliced ​​along the horizontal axis direction and the vertical axis direction of the grain image is even, it is determined that the initial anchor area is not set, that is, when the number of rows and columns is even, the initial anchor area is not set;

[0113] 2. When the number of sub-images to be spliced ​​along the horizontal and vertical directions of the grain image is an odd number, the single sub-image to be spliced ​​corresponding to the center position of the grain image is determined as the initial anchor area, that is, when the number of rows and columns is an odd number, the single sub-image to be spliced ​​at the center position is the initial anchor area;

[0114] 3. When the number of sub-images to be spliced ​​along the horizontal axis of the grain image is an even number, and the number of sub-images to be spliced ​​along the vertical axis is an odd number, the multiple sub-images to be spliced ​​corresponding to the center position of the grain image are determined as the initial anchor area, that is, when the rows are odd and the columns are even, the initial anchor area is determined after splicing the multiple sub-images to be spliced ​​at the center position. It can be understood that, when the rows are odd and the columns are even, the initial anchor area is determined based on the multiple sub-images to be spliced ​​that are adjacent to each other in the horizontal axis at the center position.

[0115] 4. When the number of sub-images to be spliced ​​along the horizontal axis of the grain image is an odd number, and the number of sub-images to be spliced ​​along the vertical axis is an even number, the multiple sub-images to be spliced ​​corresponding to the center position of the grain image are determined as the initial anchor area, that is, when the rows are even and the columns are odd, the initial anchor area is determined based on the multiple sub-images to be spliced ​​at the center position. It can be understood that, when the rows are even and the columns are odd, the initial anchor area is determined after splicing multiple adjacent sub-images to be spliced ​​in the vertical axis direction at the center position.

[0116] Fig.13 : is a schematic diagram of the splicing algorithm flow in an embodiment of the present application, such as Fig.13 As shown, the process mainly includes: S1300: determine the initial anchor area, S1302: determine the peripheral closed-loop image, S1304: stitch the peripheral closed-loop image, S1306: stitch the peripheral closed-loop image and the anchor area, S1308: determine whether it is the outermost closed-loop image, if it is not the outermost closed-loop image, S1310: update the anchor area; if it is the outermost closed-loop image, S1312: perform size optimization, S1314: output the stitching result.

[0117] Specific:

[0118] S1300: Determine the initial anchor area

[0119] Fig.14 is a schematic diagram of determining the initial anchor area in an embodiment of the present application, such as Fig.14 As shown in the figure, a single Die is composed of multiple Fovs. Depending on the number of Fovs that make up the Die, there can be the following four Fov distributions:

[0120] In the first case, the number of rows and columns is even, so there is no need for an initial anchor area, and the internal Fov image is directly used as the initial anchor area, such as Fig.14 (a)

[0121] In the second case, both the rows and columns are odd numbers, then Fig.14 The 0th Fov image in (b) is used as the initial anchor area, and all the Fov images outside form a closed-loop structure;

[0122] In the third case, the number of rows is odd and the number of columns is even, then Fig.14 The two 0th Fov images in (c) are used as the initial anchor area, and all the outer Fov images form a closed-loop structure;

[0123] In the fourth case, the number of rows is even and the number of columns is odd, then Fig.14 The two 0th Fov images in (d) are used as the initial anchor area, and all the Fov images outside form a closed-loop structure; for the case where the initial anchor area has multiple Fov images, such as Fig.14 As shown in (c, d), they are directly spliced ​​using phase correlation, and the splicing result is used as the initial anchor area;

[0124] It can be understood that, for the initial anchor region, when the initial anchor region is a plurality of sub-images to be spliced, the initial anchor region splicing is completed according to the relative offset vectors of the plurality of sub-images to be spliced.

[0125] Specifically, the relative offset vector is calculated by a phase correlation method, for example, the peak position of the two-dimensional pulse function is taken as the relative offset vector of the two images.

[0126] S1302: Determine the peripheral closed-loop image.

[0127] Fig.15 is a schematic diagram of the layout of the peripheral closed-loop image, such as Fig.15 As shown, starting from the initial anchor region, closed-loop divisions are performed on the sub-images to be spliced, so that each sub-image to be spliced ​​except the initial anchor region is subject to a closed-loop constraint.

[0128] S1304: stitching peripheral closed-loop images

[0129] When the sub-images to be stitched of the peripheral closed-loop image are stitched in a closed-loop, the closed-loop stitching direction may be set to be a clockwise direction (it is understandable that the closed-loop stitching direction may also be set to be a counterclockwise direction). Fig.16 is a schematic diagram of the splicing direction, such as Fig.16 As shown, there are four cases of relative offsets in the overlapping areas of adjacent sub-images to be stitched, where R is the reference image, S is the sub-image to be stitched, and the stitching offset is set to be positive upward, negative downward, positive to the left, and negative to the right. Then, the sum of the correct relative offsets of the entire peripheral closed-loop image should approach 0, so as to constrain the stitching offset of the sub-image to be stitched on the peripheral closed-loop image.

[0130] Specifically, the steps of stitching the sub-images to obtain the peripheral closed-loop image can be as follows:

[0131] 1) Stitching adjacent sub-images to be stitched:

[0132] The overlap of adjacent sub-images to be stitched is equal to the theoretical overlap plus the relative offset vector;

[0133] According to the theoretical overlap amount of each adjacent sub-image to be stitched, each adjacent sub-image to be stitched is stitched to obtain a closed-loop image, wherein the theoretical overlap amount can be calculated by subtracting the encoding positions of two Fov images taken by the camera;

[0134] Acquire multiple sets of candidate quantities of the closed-loop image, and determine relative offset vectors of adjacent overlapping areas based on optimization of each set of candidate quantities;

[0135] The relative offset vector can be calculated using the phase correlation method; theoretically, the peak position of the current two-dimensional pulse function can be the relative offset vector of the adjacent sub-images to be spliced: {Δx, Δy}.

[0136] Furthermore, it should be noted that, in reality, since the features of the theoretical overlapping area between two adjacent sub-images to be stitched may be highly repetitive, or have very few features, the two-dimensional pulse function has no obvious peak, and its maximum value may not be the optimal offset. Therefore, it is necessary to select larger values ​​of multiple two-dimensional pulse functions as relative offset vectors of alternative quantities. In some embodiments of the present application, the number of alternative relative offset vectors that can generally be retained is: n≥9.

[0137] 2) Optimize the closed-loop alternative quantity

[0138] Fig.17 is a schematic diagram of the closed loop overlap area in the embodiment of the present application, such as Fig.17 As shown, the adjacent overlapping area is the overlapping area between each adjacent sub-image to be spliced ​​in the closed-loop image, and the candidate amount is the relative offset vector to be selected, wherein each group of candidate amounts includes N relative offset vectors. In the process of optimizing each group of candidate amounts, it is necessary to select a relative offset vector from each group of candidate amounts so that the sum of the relative offsets of the entire closed-loop image tends to 0, that is, it is necessary to select a target offset {Δx, Δy} within each candidate amount in each group of candidate amounts so that the sum of the relative offsets of the entire closed-loop image tends to 0;

[0139]

[0140] Among them, t is the total number of candidate groups, that is, the number of overlapping areas in the closed loop.

[0141] This optimization problem can be solved by using the path dynamic programming method. Fig.18 is an optional dynamic programming directed graph, such as Fig.18 As shown, by calling the path planning method, the minimum path can be solved, that is, the minimum relative offset vector in each group of alternative quantities can be solved as the determined relative offset vector of all adjacent overlapping areas in the closed loop.

[0142] 3) Fusion of peripheral closed-loop images

[0143] According to the relative offset vector of adjacent overlapping areas, the overlapping areas of adjacent sub-images to be stitched are fused to obtain the peripheral closed-loop image.

[0144] There are many methods for fusing the peripheral closed-loop image. This embodiment uses the alph fusion method to fuse the calculated adjacent overlapping areas. For two adjacent sub-images to be spliced ​​R(x, y) and S(x, y), the fusion result F(x, y) is shown in the following formula.

[0145] F(x,y)=(1-α)·R(x,y)+α·S(x,y)

[0146] Where α is a variable coefficient, with the middle line of the overlapping area as the boundary. Fig.16 This is the principle diagram based on alph fusion left and right splicing, such as Fig.19 As shown, the upper and lower splicing are similar, where w is the overlap width of adjacent sub-images to be stitched together, and d is the pixel distance from the current pixel point p to the right edge of the image R(x, y).

[0147] S1306: stitching the peripheral closed-loop image and the anchor area

[0148] The initial anchor area is spliced ​​with the peripheral closed-loop images of each layer. First, the closed-loop fused image of the peripheral closed-loop image can be regarded as a whole, and the initial anchor area is first placed in the center of the inscribed rectangle of the closed-loop fused image to ensure that the overlapping area or overlapping ratio of the initial anchor area and the closed loop in all directions is basically consistent, which meets the actual constraint that the moving step length is basically consistent during the sub-image acquisition process of the device. Fig. 20 is a schematic diagram of the fusion of the peripheral closed-loop image and the initial anchor area, as shown in Fig. 20 As shown, four pairs of overlapping areas where the initial anchor area overlaps with the peripheral closed-loop image can be taken out respectively, and phase-correlated stitching can be performed respectively. Finally, the stitching offset that minimizes the pixel difference of the overlapping images in the stitching results of the four overlapping areas is used as the optimal stitching offset between the current closed loop and the anchor area. Similarly, ALPH fusion can also be used for the fusion of the peripheral closed-loop image and the initial anchor area.

[0149] S1308: Determine whether it is the outermost closed-loop image

[0150] If not, jump to S1310; if so, continue to execute S1312.

[0151] S1310: Update the anchor area

[0152] The result image after the stitching in S1306 is used as the latest anchor area image, and the latest anchor area image is stitched with the next layer of peripheral closed-loop image, and then the process jumps to S1304.

[0153] S1312: Die size optimization, that is, after obtaining the target grain image, the size of the grain image is optimized.

[0154] For die splicing, it is necessary to ensure that the size of all dies is consistent with the sample template size of the die, so the size of the die after splicing needs to be optimized.

[0155] Fig.21 This is the Die size optimization principle diagram in the above optimization process, such as Fig.21 As shown in the left half, two ROI areas are selected diagonally on the Die sample template as feature templates TM1 and TM2 for size positioning. The distances between these two feature templates in the X and Y directions are W and H respectively. W and H can be used to represent the Die size. It should be noted that the above sample templates include but are not limited to: golden templates and other types of standard templates.

[0156] a) Bring TM1 and TM2 into the Die image generated by actual stitching, and use image template matching algorithms, such as correlation or feature point matching, to calculate the positions of the two templates in the stitched Die image. Fig.21 As shown in the right half, the sizes Wr and Hr of the Die image actually stitched together can be obtained.

[0157] b) Taking the first column Fov that constitutes Die as the reference, the Fovs from the second column to the Xm column are offset in the X direction. The offset distance of the Fov in the i-th column is shown as follows:

[0158]

[0159] c) Taking the first row of Fov that constitutes Die as the reference, the Fovs from the second row to the Ym row are offset in the Y direction. The offset distance of the jth row Fov is shown in the following formula:

[0160]

[0161] The distances of the feature templates TM1 and TM2 in the X and Y directions are W and H respectively; the sizes of the Die images actually stitched together are Wr and Hr in the X and Y directions respectively; i and j represent the column and row respectively; Δx i , Δy i They represent the offset distance (or offset) in the X and Y directions respectively; Xm and Ym are the number of Fovs that make up the Die image in the X and Y directions respectively.

[0162] Fig. 22 is a grain image splicing device according to an embodiment of the present application, such as Fig. 22 As shown, the device comprises:

[0163] A first acquisition module 190 is used to acquire each sub-image to be spliced ​​of the grain image;

[0164] A determination module 192, used to determine the distribution of each sub-image to be spliced, and determine the initial anchor area according to the distribution;

[0165] The second acquisition module 194 is used to acquire peripheral closed-loop images of each layer. The peripheral closed-loop images of each layer are obtained by stitching sub-images to be stitched that are radially distributed from the inside to the outside with the initial anchor area as the center.

[0166] The stitching module 196 is used to stitch the initial anchor area with each layer of peripheral closed-loop images to form a target grain image, wherein the initial anchor area is stitched and fused with the adjacent peripheral closed-loop images, and then stitched and fused with the adjacent peripheral closed-loop images, and stitched in sequence until it is stitched with the outermost layer of peripheral closed-loop images to form a target grain image.

[0167] It should be noted that, during the stitching process, the peripheral closed-loop images adjacent to the initial anchor area need to be stitched with the initial anchor area as the stitching center.

[0168] In the stitching device, a first acquisition module 190 is used to acquire each sub-image to be stitched of the grain image; a determination module 192 is used to determine the distribution of each sub-image to be stitched, and determine the initial anchor area according to the distribution; a second acquisition module 194 is used to acquire each layer of peripheral closed-loop images, each layer of peripheral closed-loop images is obtained by stitching the sub-images to be stitched that are distributed radiating from the inside to the outside with the initial anchor area as the center; a stitching module 196 is used to stitch the initial anchor area with each layer of peripheral closed-loop images to form a target grain image, wherein the initial anchor area is stitched and fused with the adjacent peripheral closed-loop images, and then stitched and fused with the adjacent peripheral closed-loop images, and stitched in sequence until it is stitched with the outermost layer of peripheral closed-loop images to form a target grain image, thereby achieving the purpose of reducing the repetitive features of the image to be stitched, thereby achieving the technical effect of reducing the error rate of image stitching, and further solving the technical problem that the stitched image is easily misaligned due to the fact that the position corresponding to the highest frequency response of the image to be stitched is generally used as the stitching offset position for stitching in the related art.

[0169] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, the storage medium including a stored program, wherein when the program is running, a device where the storage medium is located is controlled to execute any one of the grain image stitching methods.

[0170] Specifically, the above storage medium is used to store program instructions for the following functions to implement the following functions:

[0171] Obtain each sub-image to be spliced ​​of the grain image; determine the distribution of each sub-image to be spliced, and determine the initial anchor area according to the distribution; obtain each layer of peripheral closed-loop images, each layer of peripheral closed-loop images is obtained by splicing the sub-images to be spliced ​​that are distributed radiating from the inside to the outside with the initial anchor area as the center; splice the initial anchor area with each layer of peripheral closed-loop images to form a target grain image, wherein the initial anchor area is spliced ​​and fused with adjacent peripheral closed-loop images, and then spliced ​​and fused with adjacent peripheral closed-loop images, and spliced ​​in sequence until it is spliced ​​with the outermost layer of peripheral closed-loop images to form a target grain image.

[0172] Optionally, in this embodiment, the above-mentioned storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above-mentioned contents. More specific examples of the above-mentioned storage medium may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above-mentioned contents.

[0173] In an exemplary embodiment of the present application, a computer program product is further provided, including a computer program, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned grain image stitching methods.

[0174] Optionally, the computer program may implement the following steps when executed by a processor:

[0175] Obtain each sub-image to be spliced ​​of the grain image; determine the distribution of each sub-image to be spliced, and determine the initial anchor area according to the distribution; obtain each layer of peripheral closed-loop images, each layer of peripheral closed-loop images is obtained by splicing the sub-images to be spliced ​​that are distributed radiating from the inside to the outside with the initial anchor area as the center; splice the initial anchor area with each layer of peripheral closed-loop images to form a target grain image, wherein the initial anchor area is spliced ​​and fused with adjacent peripheral closed-loop images, and then spliced ​​and fused with adjacent peripheral closed-loop images, and spliced ​​in sequence until it is spliced ​​with the outermost layer of peripheral closed-loop images to form a target grain image.

[0176] According to an embodiment of the present application, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned grain image stitching methods.

[0177] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0178] Fig.23 A schematic block diagram of an example electronic device 2000 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0179] like Fig.23 As shown, the device 2000 includes a computing unit 2001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 2002 or a computer program loaded from a storage unit 2008 into a random access memory (RAM) 2003. In the RAM 2003, various programs and data required for the operation of the device 2000 can also be stored. The computing unit 2001, the ROM 2002, and the RAM 2003 are connected to each other via a bus 2004. An input / output (I / O) interface 2005 is also connected to the bus 2004.

[0180] A number of components in the device 2000 are connected to the I / O interface 2005, including: an input unit 2006, such as a keyboard, a mouse, etc.; an output unit 2007, such as various types of displays, speakers, etc.; a storage unit 2008, such as a disk, an optical disk, etc.; and a communication unit 2009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 2009 allows the device 2000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0181] The computing unit 2001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 2001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 2001 performs the various methods and processes described above, such as the splicing method of grain images. For example, in some embodiments, the splicing method of grain images may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 2008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 2000 via the ROM 2002 and / or the communication unit 2009. When the computer program is loaded into the RAM 2003 and executed by the computing unit 2001, one or more steps of the splicing method of grain images described above may be performed. Alternatively, in other embodiments, the computing unit 2001 may be configured to execute the grain image stitching method in any other appropriate manner (for example, by means of firmware).

[0182] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0183] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0184] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0186] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0187] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0188] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0189] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0194] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A grain image stitching method, characterized in that: include: Acquire each sub-image to be spliced ​​of the grain image; Determining the distribution of each of the sub-images to be spliced, and determining an initial anchor area according to the distribution; Acquire peripheral closed-loop images of each layer, wherein the peripheral closed-loop images of each layer are obtained by stitching together a plurality of sub-images to be stitched that are radially distributed from the inside to the outside with the initial anchor area as the center; Acquiring peripheral closed-loop images of each layer, including: splicing adjacent sub-images to be spliced ​​to obtain a closed-loop image according to theoretical overlap amounts of adjacent sub-images to be spliced; acquiring multiple sets of candidate quantities of the closed-loop image, and determining a determined relative offset vector of an adjacent overlapping area based on optimization of each set of candidate quantities; fusing the overlapping areas of adjacent sub-images to be spliced ​​according to the determined relative offset vector of the adjacent overlapping area to obtain a peripheral closed-loop image; The target grain image is formed by splicing the initial anchor area with each layer of the peripheral closed-loop image, wherein the initial anchor area is spliced ​​and fused with the adjacent peripheral closed-loop image, and then spliced ​​and fused with the adjacent peripheral closed-loop image, and spliced ​​in sequence until it is spliced ​​with the outermost layer of the peripheral closed-loop image to form the target grain image; The target grain image is formed by splicing the initial anchor area with the peripheral closed-loop images of each layer, including: Acquire multiple pairs of boundary overlapping areas between the initial anchor area and the adjacent peripheral closed-loop image; acquire the optimal stitching offset of each overlapping area to minimize the pixel difference of the overlapping image; and fuse the peripheral closed-loop image and the multiple pairs of overlapping areas of the initial anchor area as the first anchor area based on each optimal stitching offset, wherein the first anchor area is used for stitching with its adjacent peripheral closed-loop image.

2. The method according to claim 1, characterized in that Acquire multiple sets of candidate quantities of the closed-loop image, and determine relative offset vectors of adjacent overlapping areas based on optimization of each set of candidate quantities, including: Acquire multiple sets of candidate quantities of the closed-loop image, wherein each set of candidate quantities has N relative offset vectors, where N is a natural number; The target offset vector in each group of candidate quantities is selected so that the sum of the relative offsets of the closed-loop image approaches 0, wherein the target offset vector is a determined relative offset vector of adjacent overlapping areas.

3. The method according to claim 2, characterized in that Select the target offset vector within each set of candidate quantities, including: Calling a path planning algorithm to solve the relative offset vector of the minimum path in each group of candidate quantities; The relative offset vector with the smallest path is determined as the target relative offset vector.

4. The method according to claim 1, characterized in that: Before acquiring a plurality of pairs of boundary overlapping regions between the initial anchor region and the adjacent peripheral closed-loop image, the method further includes: Placing the initial anchor area at the center of the inner frame of the adjacent peripheral closed-loop image; The difference in overlapping area or ratio between the initial anchor area and the peripheral closed-loop image in each direction is controlled to be within a preset range.

5. The method according to claim 1, characterized in that After obtaining the target grain image, the method further includes: Acquire a sample template corresponding to the target grain image, wherein two feature templates for positioning dimensions are arranged on the sample template along a diagonal direction, a distance between the two feature templates along a horizontal axis direction is a first distance, and a distance along a vertical axis direction is a second distance; Acquire the corresponding positions of the two feature templates in the target grain image, and determine a third distance along the horizontal axis and a fourth distance along the vertical axis of the two feature templates in the target grain image; Obtaining a first difference between the first distance and the third distance, and optimizing the size of the target grain image along the horizontal axis based on the first difference; A second difference between the second distance and the fourth distance is obtained, and a size of the target grain image along the longitudinal axis is optimized based on the second difference.

6. The method according to claim 5, characterized in that Optimizing the size of the target grain image along the horizontal axis based on the first difference includes: Taking the positions of the sub-images to be spliced ​​in the first column of the target grain image as reference positions, determining the number of sub-images to be spliced ​​in each column of the target grain image except the first column; Calculating a first ratio between the first difference and the number of sub-images to be spliced ​​included in each column; Obtaining the column number of each column, and determining a first offset distance corresponding to each column of the sub-images to be spliced ​​in the target grain image according to the first ratio and the column number; Each column of sub-images to be spliced ​​is moved along the horizontal axis direction based on the first offset distance corresponding to each column of sub-images to be spliced ​​to optimize the size of the target grain image along the horizontal axis direction.

7. The method according to claim 5, characterized in that Optimizing the size of the target grain image along the longitudinal axis based on the second difference includes: Taking the positions of the sub-images in the first row of the target grain image as reference positions, determining the number of sub-images included in each row of the target grain image except the first row; Calculating a second ratio of the second difference value to the number of sub-images included in each row; Acquire the row number of each row, and determine the second offset distance corresponding to each row of sub-images in the target grain image according to the second ratio and the row number; Each row of sub-images is moved along the longitudinal axis direction based on the second offset distance corresponding to each row of sub-images to optimize the size of the target grain image along the longitudinal axis direction.

8. The method according to claim 1, characterized in that Determining the distribution of the sub-images to be spliced, and determining an initial anchor area according to the distribution, includes: Obtaining the distribution quantity of each sub-image to be spliced ​​along the horizontal axis direction and the vertical axis direction of the grain image; The initial anchor area is determined according to the distribution quantity of the sub-images to be spliced ​​along the horizontal axis direction and the vertical axis direction of the grain image.

9. The method according to claim 8, characterized in that Determining an initial anchor area according to the distribution quantity of the sub-images to be spliced ​​along the horizontal axis direction and the vertical axis direction of the grain image includes: In a case where the number of sub-images to be spliced ​​distributed along the horizontal axis direction and the vertical axis direction of the grain image is an even number, determining not to set the initial anchor area; In a case where the number of the sub-images to be spliced ​​distributed along the horizontal axis direction and the vertical axis direction of the grain image is an odd number, determining a single sub-image to be spliced ​​corresponding to the center position of the grain image as the initial anchor area; When the number of sub-images to be spliced ​​along the horizontal axis of the grain image is an even number and the number of sub-images to be spliced ​​along the vertical axis is an odd number, or when the number of sub-images to be spliced ​​along the horizontal axis of the grain image is an odd number and the number of sub-images to be spliced ​​along the vertical axis is an even number, it is determined that the multiple sub-images to be spliced ​​corresponding to the center position of the grain image are the initial anchor area after splicing.

10. The method according to claim 8 or 9, characterized in that: When the initial anchor area is a plurality of the sub-images to be spliced, the initial anchor area splicing is completed according to the relative offset vectors of the plurality of sub-images to be spliced.

11. A grain image splicing device, characterized in that: include: A first acquisition module is used to acquire each sub-image to be spliced ​​of the grain image; A determination module, used to determine the distribution of each of the sub-images to be spliced, and determine an initial anchor area according to the distribution; The second acquisition module is used to acquire the peripheral closed-loop images of each layer, wherein the peripheral closed-loop images of each layer are obtained by splicing a plurality of sub-images to be spliced ​​that are sequentially distributed radially from the inside to the outside with the initial anchor area as the center, wherein acquiring the peripheral closed-loop images of each layer includes: splicing the adjacent sub-images to be spliced ​​to obtain the closed-loop image according to the theoretical overlap amount of each adjacent sub-image to be spliced; acquiring multiple groups of candidate quantities of the closed-loop image, and determining the determined relative offset vector of the adjacent overlapping area based on the optimization of each group of candidate quantities; and fusing the overlapping areas of the adjacent sub-images to be spliced ​​according to the determined relative offset vector of the adjacent overlapping area to acquire the peripheral closed-loop image; A splicing module, used for splicing the initial anchor area with each layer of the peripheral closed-loop image to form a target grain image, wherein the initial anchor area is spliced ​​and fused with the adjacent peripheral closed-loop image, and then spliced ​​and fused with the adjacent peripheral closed-loop image, and spliced ​​in sequence until it is spliced ​​with the outermost layer of the peripheral closed-loop image to form a target grain image; The stitching module is also used to form a target grain image by stitching the initial anchor area with each layer of the peripheral closed-loop images in the following manner: obtaining multiple pairs of boundary overlapping areas between the initial anchor area and the adjacent peripheral closed-loop images; obtaining the optimal stitching offset of each overlapping area to minimize the pixel difference of the overlapping images; and based on each optimal stitching offset, fusing the multiple pairs of overlapping areas of the peripheral closed-loop image and the initial anchor area as the first anchor area, and the first anchor area is used to stitch with its adjacent peripheral closed-loop image.

12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the grain image stitching method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Infrastructure mapping system and method

    CA2880229A1

  • Image stitching method based on overlapping region scale-invariant feather transform (SIFT) feature points

    CN102968777A

  • Template-based Poisson fusion image splicing method, system and device, and medium

    CN110390640A

  • Image splicing method capable of being transplanted to ARM end

    CN114202462A

  • Infrastructure mapping system and method

    WO2014031284A1