Sequence image stitching processing method and system

By using Boolean variable information and matching strategies in the image stitching algorithm, combining local and overall matching, and clarity-based fusion processing, the problems of poor universality and low accuracy in the existing technology are solved, and the efficiency of image stitching is improved and quality.

CN118887084APending Publication Date: 2024-11-01MOTIC CHINA GROUP CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411094183.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

The existing image stitching algorithms have problems such as poor generality and low accuracy, especially when the overlap rate range is unknown, it is difficult to achieve efficient stitching, and the blank and blurred areas are poorly processed, which affects the image quality.

Method used

By setting the Boolean variable information to represent the overlap rate range of image sequence data, a fast local and overall matching strategy is used to find the overlap rate range in the global range, and a fast mean matching method is used in the case of known overlap rates to improve Universality and speed; Blank detection and fuzzy detection are introduced at the same time to avoid mismatch and reduce fuzzy areas through a clarity-based fusion algorithm.

Benefits of technology

It realizes high versatility and accuracy of image stitching, improves stitching speed and image quality, and reduces the proportion of blurred areas in the panoramic map.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887084A_ABST
    Figure CN118887084A_ABST
Patent Text Reader

Abstract

The invention provides a sequence image stitching processing method and system, and relates to the technical field of image processing. The processing method comprises the following steps: acquiring image sequence data to be spliced, wherein the image sequence data comprises multi-row and multi-column image data; performing initialization processing according to the to-be-spliced image sequence data to obtain Boolean variable information; traversing the to-be-spliced image sequence data according to a preset sequence, and performing local splicing processing on the traversed two adjacent image data according to the Boolean variable information to obtain connection relation information; performing coordinate distribution processing on the to-be-spliced image sequence data according to the connection relation information to obtain coordinate distribution data; and splicing the multi-row and multi-column image data in the image sequence data according to the coordinate distribution data to obtain spliced image data. According to the sequence image splicing processing method, the technical effects of high universality and high accuracy of image splicing can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a processing method and system for stitching sequential images. Background Art

[0002] At present, due to the limitations of imaging distance, image sensor size, lens viewing angle and magnification, etc., a single imaging of an imaging device can only obtain a partial image corresponding to the current field of view (i.e., a single "field of view image"). To obtain the full view of the photographed scene, it is necessary to take pictures of different parts of the scene (for example, by moving the image acquisition device at a fixed step length) to obtain a series of partial images (i.e., multiple "field of view images"), and then find out the transformation relationship between them through a specific image processing algorithm (i.e., "image stitching" algorithm), and then combine them into a single image (i.e., "panoramic image"). At present, panoramic images have been widely used in many fields including entertainment, advertising, transportation, pathology, remote sensing, military, virtual reality, etc.

[0003] In the prior art, there are many types of image acquisition devices, which makes the characteristics of the acquired images (color gamut, dynamic range, resolution, distortion level, etc.) also diversified. In the prior art, different stitching algorithms have been proposed for different types of images; however, no matter which type of stitching algorithm is used, there are problems such as poor versatility and low accuracy. Summary of the invention

[0004] The purpose of the present application is to provide a processing method, system, electronic device and computer-readable storage medium for sequential image stitching, which can achieve the technical effect of high versatility and high accuracy of image stitching.

[0005] In a first aspect, the present application provides a method for processing sequential image stitching, comprising:

[0006] Acquire image sequence data to be spliced, wherein the image sequence data includes image data of multiple rows and columns;

[0007] Performing initialization processing according to the image sequence data to be spliced ​​to obtain Boolean variable information;

[0008] Traversing the image sequence data to be spliced ​​according to a preset order, and performing local splicing processing on two adjacent image data traversed according to the Boolean variable information to obtain connection relationship information;

[0009] Performing coordinate allocation processing on the image sequence data to be spliced ​​according to the connection relationship information to obtain coordinate allocation data;

[0010] Image data of multiple rows and columns in the image sequence data are spliced ​​according to the coordinate allocation data to obtain spliced ​​image data.

[0011] In the above implementation process, the processing method for stitching sequential images provided in the embodiment of the present application sets Boolean variable information, and the Boolean variable information can indicate whether the overlapping rate range of the image sequence data to be stitched is known; thus, for image sequence data with unknown overlapping rate range, the overlapping rate range can be found out through a fast matching strategy combining local and overall matching, thereby improving the versatility of the algorithm; for the case where the overlapping rate range is known, the stitching speed can be improved through a fast mean matching method; thus, the processing method for stitching sequential images can achieve the technical effect of high versatility and high accuracy of image stitching.

[0012] Furthermore, the step of performing initialization processing according to the image sequence data to be spliced ​​to obtain Boolean variable information includes:

[0013] Performing initialization processing according to the image sequence data to be spliced, obtaining the horizontal overlap rate, vertical overlap rate, and overlap area-downsampling information of adjacent image data;

[0014] Boolean variable information is generated according to the horizontal overlap rate, the vertical overlap rate and the overlap area-downsampling information.

[0015] In the above implementation process, Boolean variable information is used to indicate whether the horizontal overlap rate, vertical overlap rate and overlapping area-downsampling information in the image sequence data to be stitched are known. If known, the image stitching can be directly performed according to the known horizontal overlap rate, vertical overlap rate and overlapping area-downsampling information in the subsequent local stitching processing, and the stitching speed can be improved by the fast mean matching method; if unknown, the preset horizontal overlap rate, vertical overlap rate and overlapping area-downsampling information can be set, and the overlap rate range can be found through a fast local and overall combination matching strategy.

[0016] Furthermore, before the step of traversing the image sequence data to be spliced ​​according to a preset order and performing local splicing processing on two adjacent traversed image data according to the Boolean variable information to obtain connection relationship information, the method further includes:

[0017] Traverse the image sequence data to be spliced, and perform the following processing on the traversed image data:

[0018] Loading the image data into a preset memory;

[0019] Converting the image data into grayscale image data;

[0020] The grayscale image data is intercepted and processed according to the Boolean variable information to obtain a maximum possible overlapping area sub-image corresponding to the image data, and the image data is removed from the preset memory, wherein the maximum possible overlapping area sub-image includes one or more of the maximum possible overlapping area on the upper side, the maximum possible overlapping area on the lower side, the maximum possible overlapping area on the right side, and the maximum possible overlapping area on the left side.

[0021] In the above implementation process, for each image data, the image data above it, the image data below it, the image data to the right, and the image data to the left may all have overlapping areas with the image data; thus, the grayscale image data is intercepted and processed according to the Boolean variable information to obtain the maximum possible overlapping area sub-image corresponding to the image data, the maximum possible overlapping area sub-image is retained for local stitching, and the image data is deleted, which can effectively reduce the memory usage during image stitching processing and further improve the stitching processing speed.

[0022] Furthermore, the step of traversing the image sequence data to be spliced ​​according to a preset order, and performing local splicing processing on two adjacent traversed image data according to the Boolean variable information to obtain connection relationship information includes:

[0023] The image sequence data to be spliced ​​is traversed according to a preset order, and the traversed image data is processed as follows:

[0024] determining an image adjacent to the image data as adjacent image data;

[0025] Using the maximum possible overlapping area sub-image corresponding to the image data as a reference image, and using the maximum possible overlapping area sub-image of the adjacent image data as a target image;

[0026] If the size of the target image is greater than a preset threshold, downsampling the reference image and the target image to obtain the downsampled reference image and the target image;

[0027] Performing blank detection on the target image and the reference image, and stopping the local stitching process if one of the target image and the reference image is detected as blank;

[0028] Performing blur detection on the target image and the reference image, and stopping the local stitching process if one of the target image and the reference image is detected as blurry;

[0029] Performing local splicing processing on the target graph and the reference graph to obtain sub-connection relationship information;

[0030] After the traversal is completed, the connection relationship information is obtained according to all the sub-connection relationship information.

[0031] In the above implementation process, when the size of the target image is larger than a preset threshold, the target image and the reference image are downsampled, thereby halving the resolution of the target image and the reference image, thereby improving the processing speed of subsequent image stitching; by introducing fast blank detection and blur detection, the probability of mismatching is avoided, thereby also improving the processing speed.

[0032] Furthermore, the step of performing local splicing processing on the target graph and the reference graph to obtain sub-connection relationship information includes:

[0033] Determine whether the Boolean variable corresponding to the splicing direction overlap rate is true according to the Boolean variable information;

[0034] If the Boolean variable is not true, performing one or more local splicing processes of a feature matching operation, an overall optimization operation, and an offset optimization operation on the target image and the reference image;

[0035] If the splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained;

[0036] If the splicing fails, one or more local splicing processes in the template matching operation and the overall optimization operation are performed on the target image and the reference image. If the splicing succeeds, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained;

[0037] If the Boolean variable is true, one or more local splicing processes including a splicing direction mean matching operation and an offset optimization operation are performed on the target image and the reference image, and if the splicing is successful, sub-connection relationship information is obtained;

[0038] If the reference image and the target image have been downsampled, an original resolution optimization operation is performed on the reference image and the target image, and an overlap rate update operation is performed on the Boolean variable information.

[0039] In the above implementation process, if the Boolean variable corresponding to the overlap rate in the stitching direction is true, it means that the overlap rate in the stitching direction is known, and the matching between subsequent adjacent field of view images can be carried out in a local range. Then, according to the known overlap rate, the target image and the reference image can be subjected to stitching direction mean matching operations and offset optimization operations to improve the matching speed and reduce the memory usage rate. If the Boolean variable corresponding to the overlap rate in the stitching direction is not true, it means that the overlap rate in the stitching direction is unknown. Through fast and accurate matching, the overlap rate range between adjacent images can be found in a global range, and feature matching operations, template matching operations, overall optimization operations, offset optimization operations, etc. can be carried out on the target image and the reference image.

[0040] Further, the connection relationship information includes a plurality of sub-connection relationship information, and the sub-connection relationship information includes a five-tuple data [iRef, iObj, Succ, Offset, Conf], wherein iRef represents the sequence number of the reference image in the connection relationship, iObj represents the sequence number of the target image in the reorganized connection relationship, Succ is a Boolean variable, Offset is the offset of the target image relative to the reference image, and Conf represents the credibility of the offset; the step of performing coordinate allocation processing on the image sequence data to be spliced ​​according to the connection relationship information to obtain coordinate allocation data includes:

[0041] Traversing the connection relationship information to obtain the sub-connection relationship information corresponding to the maximum credibility;

[0042] Adding the image data corresponding to the sub-connection relationship information to a list of stitched fields of view;

[0043] Centering on the stitched visual field list, matching is performed sequentially according to the connection relationship information and the image sequence data to be stitched to obtain coordinate allocation data.

[0044] In the above implementation process, based on the credibility in the connection relationship information, the coordinates of each image data in the image sequence data to be stitched are assigned, so that the accuracy of image stitching can be guaranteed; in addition, the center of the stitched field of view list is determined by the sub-connection relationship information corresponding to the maximum credibility, and then the image stitching is gradually completed with this as the center, which ensures accuracy and effectively guarantees processing speed.

[0045] Furthermore, after the step of splicing the image data of multiple rows and columns in the image sequence data according to the coordinate allocation data to obtain the spliced ​​image data, the method further comprises:

[0046] Performing definition calculation on the overlapping area of ​​the image data according to a preset difference algorithm or a preset gradient algorithm to obtain definition data;

[0047] The stitched image data is subjected to image fusion processing according to the definition data to obtain stitched image data after image fusion.

[0048] In the above implementation process, by introducing a clarity-based fusion algorithm to obtain stitched image data after image fusion, the proportion of blurred areas in the panoramic image can be reduced, thereby improving image quality.

[0049] In a second aspect, the present application provides a processing system for stitching sequential images, comprising:

[0050] A sequence image unit, used for acquiring image sequence data to be spliced, wherein the image sequence data includes image data of multiple rows and columns;

[0051] An initialization unit, used for performing initialization processing according to the image sequence data to be spliced ​​to obtain Boolean variable information;

[0052] A local splicing unit, used for traversing the image sequence data to be spliced ​​according to a preset order, and performing local splicing processing on two adjacent image data traversed according to the Boolean variable information to obtain connection relationship information;

[0053] A coordinate allocation unit, used for performing coordinate allocation processing on the image sequence data to be spliced ​​according to the connection relationship information to obtain coordinate allocation data;

[0054] An image splicing unit, used for splicing image data of multiple rows and columns in the image sequence data according to the coordinate allocation data to obtain spliced ​​image data;

[0055] Furthermore, the initialization unit is specifically used for:

[0056] Performing initialization processing according to the image sequence data to be spliced, obtaining the horizontal overlap rate, vertical overlap rate, and overlap area-downsampling information of adjacent image data;

[0057] Boolean variable information is generated according to the horizontal overlap rate, the vertical overlap rate and the overlap area-downsampling information.

[0058] Furthermore, the local splicing unit is also used for:

[0059] Traverse the image sequence data to be spliced, and perform the following processing on the traversed image data:

[0060] Loading the image data into a preset memory;

[0061] Converting the image data into grayscale image data;

[0062] The grayscale image data is intercepted and processed according to the Boolean variable information to obtain a maximum possible overlapping area sub-image corresponding to the image data, and the image data is removed from the preset memory, wherein the maximum possible overlapping area sub-image includes one or more of the maximum possible overlapping area on the upper side, the maximum possible overlapping area on the lower side, the maximum possible overlapping area on the right side, and the maximum possible overlapping area on the left side.

[0063] Furthermore, the local splicing unit is specifically used for:

[0064] The image sequence data to be spliced ​​is traversed according to a preset order, and the traversed image data is processed as follows:

[0065] determining an image adjacent to the image data as adjacent image data;

[0066] Using the maximum possible overlapping area sub-image corresponding to the image data as a reference image, and using the maximum possible overlapping area sub-image of the adjacent image data as a target image;

[0067] If the size of the target image is greater than a preset threshold, downsampling the reference image and the target image to obtain the downsampled reference image and the target image;

[0068] Performing blank detection on the target image and the reference image, and stopping the local stitching process if one of the target image and the reference image is detected as blank;

[0069] Performing blur detection on the target image and the reference image, and stopping the local stitching process if one of the target image and the reference image is detected as blurry;

[0070] Performing local splicing processing on the target graph and the reference graph to obtain sub-connection relationship information;

[0071] After the traversal is completed, the connection relationship information is obtained according to all the sub-connection relationship information.

[0072] Furthermore, the local splicing unit is also used for:

[0073] Determine whether the Boolean variable corresponding to the splicing direction overlap rate is true according to the Boolean variable information;

[0074] If the Boolean variable is not true, performing one or more local splicing processes of a feature matching operation, an overall optimization operation, and an offset optimization operation on the target image and the reference image;

[0075] If the splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained;

[0076] If the splicing fails, one or more local splicing processes in the template matching operation and the overall optimization operation are performed on the target image and the reference image. If the splicing succeeds, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained;

[0077] If the Boolean variable is true, one or more local splicing processes including a splicing direction mean matching operation and an offset optimization operation are performed on the target image and the reference image, and if the splicing is successful, sub-connection relationship information is obtained;

[0078] If the reference image and the target image have been downsampled, an original resolution optimization operation is performed on the reference image and the target image, and an overlap rate update operation is performed on the Boolean variable information.

[0079] Furthermore, the coordinate allocation unit is specifically used for:

[0080] Traversing the connection relationship information to obtain the sub-connection relationship information corresponding to the maximum credibility;

[0081] Adding the image data corresponding to the sub-connection relationship information to a list of stitched fields of view;

[0082] Centering on the stitched visual field list, matching is performed sequentially according to the connection relationship information and the image sequence data to be stitched to obtain coordinate allocation data.

[0083] Furthermore, the image stitching unit is also used for:

[0084] Performing definition calculation on the overlapping area of ​​the image data according to a preset difference algorithm or a preset gradient algorithm to obtain definition data;

[0085] The stitched image data is subjected to image fusion processing according to the definition data to obtain stitched image data after image fusion.

[0086] In a third aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0087] In a fourth aspect, the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.

[0088] In a fifth aspect, the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method as described in any one of the first aspects.

[0089] Other features and advantages disclosed in the present application will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned technology disclosed in the present application.

[0090] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0092] Figure 1 A schematic diagram of a global search provided in an embodiment of the present application;

[0093] Figure 2 A schematic diagram of template matching provided in an embodiment of the present application;

[0094] Figure 3 A schematic diagram of feature matching provided in an embodiment of the present application;

[0095] Figure 4 A schematic diagram of a process flow of a sequence image stitching processing method provided in an embodiment of the present application;

[0096] Figure 5 A schematic flow chart of another method for processing sequential image stitching provided in an embodiment of the present application;

[0097] Figure 6 A schematic diagram of the process of local splicing processing provided in an embodiment of the present application;

[0098] Figure 7 A schematic diagram of image sequence data to be spliced ​​provided in an embodiment of the present application;

[0099] Figure 8 A schematic diagram of image offset and overlapping area for overlap rate calculation provided in an embodiment of the present application;

[0100] Fig. 9 A schematic diagram of a clarity-based fusion algorithm provided in an embodiment of the present application;

[0101] Fig.10 A schematic diagram of the flow of the splicing algorithm provided in the embodiment of the present application;

[0102] Fig.11 A structural block diagram of a processing system for sequential image stitching provided in an embodiment of the present application;

[0103] Fig.12 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0105] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0106] Generally speaking, as a means of generating panoramic images, the stitching algorithm includes three key technologies:

[0107] The first is image matching technology:

[0108] The aim is to find the optimal transformation between adjacent field of view images so that the overlapping area between the images obtained according to the transformation is the smallest or most similar at a certain distance;

[0109] T opt = argmax{s(I n ,T(I))|T∈Γ};

[0110] Among them, I n is the neighborhood image of image I, T is the image transformation, which generally only includes geometric transformation (such as translation transformation, rigid transformation, affine transformation, projective transformation, etc.), Γ is the space where the transformation T is located, and s(X,Y) is some similarity measure (such as Pearson correlation coefficient);

[0111] Image matching can be divided into three main methods according to the chronological order of technological development:

[0112] (1) Global search method: This is the earliest method. The specific operation is: take each pixel position in the adjacent image as a candidate position, calculate the similarity between the current image and the adjacent image in the overlapping area when the origin (lower left corner) of the current image coincides with this point, and take the largest one as the optimal transformation. This method can find the global optimum and is therefore the most accurate, but because it involves a large number of large-scale correlation calculations, it is very time-consuming. In addition, this method is only applicable to situations where the adjacent images only exist in translation transformations; e.g. Figure 1 As shown, Figure 1 A schematic diagram of a global search provided in an embodiment of the present application, where A and B are images to be stitched;

[0113] (2) Template matching method; this is a method developed in the middle period. The specific steps are as follows: extract a small local area (template) in the current image, and perform template matching in a specific area (search area) of the adjacent image, that is, calculate the similarity between the template and the small area of ​​the same size at each position in the search area, and take the largest one as the optimal matching position, and calculate the translation relationship between the two images based on the optimal match. This method avoids large-scale correlation calculations, so the computational complexity is relatively low. The disadvantage is that due to the local similarities in the image, the resulting match is only locally optimal rather than globally optimal. Similarly, this method is only applicable to situations where adjacent images only exist in translation transformations; such as Figure 2 As shown, Figure 2 A schematic diagram of template matching provided in an embodiment of the present application, wherein B is the current image, the gray block therein is a template, A on the left is a neighborhood image, the gray area therein is the corresponding search area of ​​the template, and the dotted rectangular area is the area in the search area that is most similar to the template; assuming that the center position of the template is p and the center point position of the red box is q, then the translation transformation of image B relative to image A is T(B)=B+(pq);

[0114] (3) Feature matching method: This is a method developed relatively late. Like the template matching method mentioned above, it is still a local matching method, except that the data used for matching here comes from some special local areas, such as corners, edges, spots, etc. The specific approach is: for each image, first find the special area in it (called interest point, represented by the center point of the area) through the algorithm, and then extract the feature vectors of these areas (called descriptors) to form the feature point set of each image (one feature point = interest point + descriptor), thereby converting the image matching problem into the matching problem between their feature point sets. Depending on the type of descriptor, this type of feature matching algorithm has good adaptability to translation, rotation, scale transformation and uneven illumination between adjacent images. Since its feature vector comes from the special secondary processing of the original image, this method largely avoids the local similarity in template matching, so the matching accuracy is relatively high, but the calculation is relatively complex, and there is a possibility of matching failure due to the lack of features in the image, and there is also the possibility of mismatching due to local similarity in the image. In this type of method, the commonly used feature points include Harris corner points, SUSAN corner points, FAST corner points, DOG corner points, KR corner points, and KLT corner points, and the common descriptors include HOG, LBP, BRIEF, etc. In addition, there are SIFT features and SURF features that combine feature point extraction and feature descriptor extraction into one; Figure 3 As shown, Figure 3A schematic diagram of feature matching provided for an embodiment of the present application, wherein B is the current image, the small block therein is a feature region, the left A is a neighborhood image, the two regions therein are feature regions, and the matching calculation is performed only in the feature region on the right and the two feature regions on the left. Assuming the center position of the right region is p, and the center point position of the left region that is most similar to the right feature is q, then the translation transformation of image B relative to image A is T(B)=B+(pq);

[0115] The second is coordinate allocation technology:

[0116] The purpose is to determine the coordinate position of each field of view image in the panoramic image. After performing the above image matching on all adjacent images, the relative offset and matching degree (such as the size of the correlation) of each image are obtained. Then, according to the spatial (such as row and column) adjacency relationship, a global coordinate system can be established according to a certain algorithm such as the maximum spanning tree, and the position of all field of view images in the coordinate system can be determined. At this time, how to obtain the most reasonable maximum spanning tree is a key factor;

[0117] The third is image fusion technology:

[0118] The aim is to eliminate the inconsistency of illumination and color difference between adjacent images in the overlapping area, so as to achieve a smooth transition between adjacent images in the panoramic image. The most commonly used method is the linear weighted method, which is simple to implement, fast, and can achieve good results in most cases, but when the overlapping area of ​​adjacent images is small or their brightness and color in the overlapping area are quite different, the fusion effect is poor; the best fusion method is Poisson fusion, which can basically be applied to any occasion, but it has a large amount of calculation and is difficult to use in situations with high timeliness.

[0119] For the existing stitching algorithm, the processing process is as follows: according to the preset overlap rate and fluctuation rate between adjacent images, the minimum possible overlap area between each pair of adjacent images is determined, and the area of ​​one of them is subjected to wavelet transform. The transformation result is used to calculate the feature intensity of the area on the one hand, and on the other hand, the local area where the highest frequency position is located is selected as the "matching block". At the same time, the search area of ​​the matching block in the other image pair is determined again according to the overlap rate and fluctuation rate, and then the matching block is template matched in the search area; after completing the pairwise matching of all field of view image sequences, the maximum spanning tree of the image sequence is obtained based on the correlation of template matching and the feature intensity of the corresponding minimum overlap area, so as to calculate the global coordinates of each field of view image;

[0120] However, in the stitching process of the existing stitching algorithm, only a very small local area needs to be reserved for subsequent template matching. Therefore, this method has the advantages of fast stitching speed and low memory usage, but it has the following problems:

[0121] Universality: This method requires the user to input information such as overlap rate and fluctuation rate. For image sequences that cannot know this information in advance, this method cannot be used directly.

[0122] Accuracy; First, this method only uses local template matching. Due to local similarity, it may only obtain the local optimal match between images; Second, this method uses wavelet (high frequency) coefficients as feature description and template selection. This strategy can obtain good matching for strong texture or even weak texture areas, but it is not applicable to the following situations: the overlapping area is blank, the overlapping area is blurred, etc.

[0123] The unilateral blur problem is not considered in the fusion stage; the existing methods do not consider the problem that one side of the overlapping area may be blurred when performing fusion, resulting in a decrease in the final image quality.

[0124] In view of the technical problems raised above, the embodiments of the present application provide a processing method, system, electronic device and computer-readable storage medium for stitching of sequence images, which can be used in the stitching process of image sequences, for example, can be used in any occasion of stitching of sequence images where only translation transformation exists between adjacent images, such as various automatic scanning devices (pathological slice scanners, automatic optical detection equipment, etc.); Figure 4 As shown, Figure 4 A flowchart of a method for stitching a sequence of images provided in an embodiment of the present application is provided. The method for stitching a sequence of images comprises the following steps:

[0125] S100: Acquire image sequence data to be stitched, where the image sequence data includes image data of multiple rows and columns;

[0126] Exemplarily, the image sequence data to be spliced ​​includes a plurality of image data, and the plurality of image data are arranged in a matrix form.

[0127] S200: Performing initialization processing according to the image sequence data to be spliced ​​to obtain Boolean variable information;

[0128] Exemplarily, by setting up Boolean variable information, it is indicated whether the overlap rate of the image sequence data to be spliced ​​in the splicing direction (horizontal direction or vertical direction) is known;

[0129] Optionally, the Boolean variable information may include multiple Boolean variables; for example, a Boolean variable ho_known is set to indicate whether the horizontal overlap rate is known, a Boolean variable vo_known is set to indicate whether the vertical overlap rate is known; and a Boolean variable downsample is set to indicate whether the maximum possible overlap area is downsampled.

[0130] S300: traversing the image sequence data to be spliced ​​according to a preset order, and performing local splicing processing on two adjacent image data traversed according to Boolean variable information to obtain connection relationship information;

[0131] Exemplarily, the preset order may be to perform the local splicing processing in sequence along the horizontal direction, or to perform the local splicing processing in sequence along the vertical direction, or to perform the local splicing processing in both the horizontal direction and the vertical direction simultaneously or alternately.

[0132] S400: performing coordinate allocation processing on the image sequence data to be spliced ​​according to the connection relationship information to obtain coordinate allocation data;

[0133] S500: splicing image data of multiple rows and columns in the image sequence data according to the coordinate allocation data to obtain spliced ​​image data.

[0134] Exemplarily, the processing method for stitching sequential images provided in the embodiment of the present application sets Boolean variable information, which can indicate whether the overlapping rate range of the image sequence data to be stitched is known; thus, for image sequence data with an unknown overlapping rate range, the overlapping rate range can be found out through a fast matching strategy combining local and overall matching, thereby improving the versatility of the algorithm; for the case where the overlapping rate range is known, the stitching speed can be improved through a fast mean matching method; thus, the processing method for stitching sequential images can achieve the technical effect of high versatility and high accuracy of image stitching.

[0135] See also Figure 5 , Figure 5 A schematic flow chart of another method for processing sequential image stitching provided in an embodiment of the present application;

[0136] Exemplarily, S200: performing initialization processing according to the image sequence data to be spliced ​​to obtain Boolean variable information includes:

[0137] S210: Performing initialization processing according to the image sequence data to be spliced, obtaining the horizontal overlap rate, vertical overlap rate, and overlap area-downsampling information of adjacent image data;

[0138] S220: Generate Boolean variable information according to the horizontal overlap rate, the vertical overlap rate and the overlap area-downsampling information.

[0139] Exemplarily, Boolean variable information is used to indicate whether the horizontal overlap rate, vertical overlap rate and overlapping area-downsampling information in the image sequence data to be stitched are known. If known, the image stitching can be directly performed according to the known horizontal overlap rate, vertical overlap rate and overlapping area-downsampling information in the subsequent local stitching process, and the stitching speed can be improved by the fast mean matching method; if unknown, the preset horizontal overlap rate, vertical overlap rate and overlapping area-downsampling information can be set, and the overlap rate range can be found through a fast local and global combination matching strategy.

[0140] Exemplarily, before the step S300 of traversing the image sequence data to be spliced ​​according to a preset order, performing local splicing processing on two adjacent image data traversed according to Boolean variable information, and obtaining connection relationship information, the method further includes:

[0141] Traverse the image sequence data to be spliced, and perform the following processing on the traversed image data:

[0142] Load image data into preset memory;

[0143] Convert image data into grayscale image data;

[0144] The grayscale image data is intercepted and processed according to the Boolean variable information to obtain a maximum possible overlapping area sub-image corresponding to the image data, and the image data is removed from a preset memory, wherein the maximum possible overlapping area sub-image includes one or more of the upper maximum possible overlapping area, the lower maximum possible overlapping area, the right maximum possible overlapping area, and the left maximum possible overlapping area.

[0145] Exemplarily, for each image data, there may be overlapping areas with the image data above, below, right and left adjacent to it; thus, the grayscale image data is intercepted and processed according to the Boolean variable information to obtain the maximum possible overlapping area sub-image corresponding to the image data, the maximum possible overlapping area sub-image is retained for local stitching, and the image data is deleted, which can effectively reduce the memory usage during image stitching processing and further improve the stitching processing speed.

[0146] See also Figure 6 , Figure 6 A schematic diagram of the process of local splicing processing provided in an embodiment of the present application.

[0147] Exemplarily, S300: traversing the image sequence data to be spliced ​​according to a preset order, performing local splicing processing on two adjacent image data traversed according to Boolean variable information, and obtaining the connection relationship information includes:

[0148] The image sequence data to be spliced ​​is traversed according to the preset order, and the traversed image data is processed as follows:

[0149] S311: Determine an image adjacent to the image data as adjacent image data;

[0150] S312: using the maximum possible overlapping area sub-image corresponding to the image data as a reference image, and using the maximum possible overlapping area sub-image of adjacent image data as a target image;

[0151] S313: Determine whether the size of the target image is greater than a preset threshold;

[0152] S314: If yes, downsample the reference image and the target image to obtain the downsampled reference image and the target image;

[0153] S315: Performing blank detection on the target image and the reference image;

[0154] S316: Determine whether one of the target image and the reference image is detected as blank, if so, jump to S310: stop the local splicing process;

[0155] S317: Perform fuzzy detection on the target image and the reference image;

[0156] S318: Determine whether one of the target image and the reference image is detected as blurred, if so, jump to S310: stop the local stitching process;

[0157] S319: Perform local splicing processing on the target graph and the reference graph to obtain sub-connection relationship information;

[0158] After the traversal is completed, the connection relationship information is obtained based on all the sub-connection relationship information.

[0159] Exemplarily, when the size of the target image is larger than a preset threshold, the target image and the reference image are downsampled, thereby halving the resolution of the target image and the reference image, thereby improving the processing speed of subsequent image stitching; by introducing fast blank detection and blur detection, the probability of mismatching is avoided, thereby also improving the processing speed.

[0160] Exemplarily, S319: performing local splicing processing on the target graph and the reference graph to obtain sub-connection relationship information includes:

[0161] According to the Boolean variable information, it is determined whether the Boolean variable corresponding to the overlap rate in the splicing direction is true;

[0162] If the Boolean variable is not true, one or more local splicing processes among feature matching operation, overall optimization operation, and offset optimization operation are performed on the target image and the reference image;

[0163] If the splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained;

[0164] If the splicing fails, the target image and the reference image are subjected to one or more local splicing processes in the template matching operation and the overall optimization operation. If the splicing succeeds, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained;

[0165] If the Boolean variable is true, one or more local splicing processes in the splicing direction mean matching operation and the offset optimization operation are performed on the target image and the reference image. If the splicing is successful, the sub-connection relationship information is obtained;

[0166] If the reference image and the target image have been downsampled, the original resolution optimization operation is performed on the reference image and the target image, and the overlap rate update operation is performed on the Boolean variable information.

[0167] Exemplarily, if the Boolean variable corresponding to the overlap rate in the stitching direction is true, it means that the overlap rate in the stitching direction is known, and the matching between subsequent adjacent field of view images can be performed in a local range. Then, according to the known overlap rate, the target image and the reference image can be subjected to stitching direction mean matching operations and offset optimization operations to improve the matching speed and reduce memory usage; if the Boolean variable corresponding to the overlap rate in the stitching direction is not true, it means that the overlap rate in the stitching direction is unknown. Through fast and accurate matching, the overlap rate range between adjacent images can be found in a global range, and feature matching operations, template matching operations, overall optimization operations, offset optimization operations, etc. can be performed on the target image and the reference image.

[0168] Exemplarily, the connection relationship information includes a plurality of sub-connection relationship information, and the sub-connection relationship information includes a five-tuple data [iRef, iObj, Succ, Offset, Conf], wherein iRef represents the sequence number of the reference image in the connection relationship in the sequence, iObj represents the sequence number of the target image in the reorganized connection relationship in the sequence, Succ is a Boolean variable, Offset is the offset of the target image relative to the reference image, and Conf represents the credibility of the offset; S400: performing coordinate allocation processing on the image sequence data to be spliced ​​according to the connection relationship information to obtain the coordinate allocation data, comprising:

[0169] S410: traversing the connection relationship information to obtain the sub-connection relationship information corresponding to the maximum credibility;

[0170] S420: adding the image data corresponding to the sub-connection relationship information to the stitched view list;

[0171] S430: Centering on the list of stitched fields of view, sequentially matching the connection relationship information and the image sequence data to be stitched is performed to obtain coordinate allocation data.

[0172] Exemplarily, based on the credibility in the connection relationship information, coordinates are assigned to each image data in the image sequence data to be stitched, so that the accuracy of image stitching can be guaranteed; in addition, the center of the stitched field of view list is determined by the sub-connection relationship information corresponding to the maximum credibility, and then the image stitching is gradually completed with this as the center, which ensures accuracy while effectively ensuring processing speed.

[0173] Exemplarily, after the step of S500: splicing the image data of multiple rows and columns in the image sequence data according to the coordinate allocation data to obtain the spliced ​​image data, the method further includes:

[0174] S510: performing clarity calculation on the overlapping area of ​​the image data according to a preset difference algorithm or a preset gradient algorithm to obtain clarity data;

[0175] S520: Perform image fusion processing on the stitched image data according to the definition data to obtain stitched image data after image fusion.

[0176] For example, by introducing a clarity-based fusion algorithm to obtain stitched image data after image fusion, the proportion of blurred areas in the panoramic image can be reduced, thereby improving image quality.

[0177] In some implementation scenarios, the processing method for stitching sequential images provided in the embodiments of the present application has a specific implementation process example as follows:

[0178] There must be overlapping areas between adjacent field of view images in an image sequence, which is a prerequisite for seamless stitching of the image sequence. In order to obtain an image sequence, the image acquisition device is often moved according to a certain expected overlap rate for acquisition. However, affected by the mechanical precision, vibration and other factors of the equipment, the actual overlap rate between adjacent images acquired at different positions will not be strictly equal to the expected value, but will fall within a certain interval centered on the expected value. The core of image stitching is to calculate the actual overlap rate between adjacent images by means of image matching, and based on this calculate the relative positions between each other, and then connect the images together to form a panoramic image;

[0179] Although the matching calculation between two images can be completed without any prior knowledge, it must be slow because it needs to be calculated in the global range of the image. On the contrary, if it is known that the overlap rate between the two images is within a certain range, the matching calculation can be reduced to the local area, thereby greatly reducing the time complexity of the matching. Therefore, the technical solution of the present application includes two matching strategies: one is to find the overlap rate range between adjacent images in the global range through fast and accurate matching at the initial stage of splicing, that is, when the overlap rate range is unknown; the other is to place the subsequent matching between adjacent field of view images in the local range after obtaining the overlap rate range. Among them, in strategy one, in order to retain the speed advantage of local matching while avoiding the local optimal trap and improving the matching accuracy, the present application proposes a matching strategy that combines local and global matching; for strategy two, in order to improve the matching speed and reduce memory usage, the present application proposes a matching method based on the mean; in addition, in order to avoid matching blank and blurred image pairs, the present application has developed special blank detection and blur detection, and these detection modules are applied before all matching calculations. Finally, considering the blurry situation, the present application proposes a new fusion method;

[0180] The technical solution of this application consists of several defined terms, operation modules and algorithms, which are specifically described as follows:

[0181] I. Definition of Terms:

[0182] 1.1. Image sequence to be stitched: The image sequence to be stitched is the image sequence data to be stitched, which refers to the adjacent R rows and C columns of images on the plane, which can be recorded as S = {I r,c |r=0,1,…R-1;c=0,1,…C-1} and have the same size, for example Figure 7 As shown, Figure 7 A schematic diagram of image sequence data to be spliced ​​provided in an embodiment of the present application is a sequence of images to be spliced ​​of 4 rows and 3 columns;

[0183] 1.2. Image stitching order: refers to the order in which the stitching algorithm processes the field of view images, generally starting from image I 0,0 Start, proceed line by line (the splicing direction is horizontal): that is, process I in sequence 0,0 , I 0,1 ,…I 0,C-1 , I 1,0 , I 1,1 ,…I 1,C-1 ,…I R-1, 0I R-1,1 ,…I R-1,C-1 ; or row by row (the splicing direction is vertical): that is, process I in sequence 0,0 , I 1,0 ,…IR-1,0 , I 0,1 , I 1,1 ,…I R-1,1 ,…I 0,C-1 I 1,C-1 ,…I R-1,C-1 ;

[0184] 1.3. Overlap ratio: refers to the overlap ratio of one image relative to the other image in the horizontal or vertical direction on the plane of two images of the same size;

[0185] 1.4. Offset: refers to the relative position of one image (the origin) relative to the other image (the origin) on the plane.

[0186] The offset between two images of the same size can be calculated based on the image size and their overlap ratio in two directions: suppose images A and B have widths W and heights H respectively, and the overlap ratios of image B relative to image A in the horizontal and vertical directions are OvlpX and OvlpY respectively, then the offset of image B relative to image A is o BA =(OffX,OffY):

[0187] OffX = W x (1-OvlpX) (1);

[0188] OffY=H x(1-OvlpY)

[0189] The offset of image A relative to image B is o AB =(-OffX,-OffY);

[0190] Obviously, the overlap ratio can also be calculated from the offset according to (1).

[0191] 1.5. Overlap area: refers to the intersection area of ​​two images on the plane. It can be calculated based on the size of the two images and the offset between them: Let the width and height of image A be W respectively A and H A , the width and height of image B are W B and H B , the offset of image B relative to image A is o BA =(OffX, OffY), then the overlapping area Ω in image A A for:

[0192] Ω A = [Xs, Xe, Ys, Ye] (2);

[0193] Xs=max(0,OffX)

[0194] Xe=min(W A ,OffX+WB )

[0195] Ys=max(0,OffY)

[0196] Ye=min(H A ,OffY+H B )

[0197] The overlapping area in image B is:

[0198] Ω B = Ω A -t, (3);

[0199] Where t = (OffX, OffY) is the translation (offset) vector;

[0200] like Figure 8 As shown, Figure 8 A schematic diagram of image offset and overlapping area for overlap rate calculation provided in an embodiment of the present application;

[0201] Optionally, W=600, H=480, OvlpX=0.1, OvlpY=0.8, OffX=540, OffY=96; R A =[540,600,96,480],R B =[0,120,0,384];

[0202] 1.6. Default value of overlap rate: refers to the expected (theoretical) overlap rate of the two images collected in the direction when the image acquisition device moves in the horizontal or vertical direction during the acquisition of the image sequence;

[0203] 1.7. Offset default value: refers to the expected (theoretical) offset of the two images captured in the horizontal or vertical direction when the image acquisition device moves in the horizontal or vertical direction during the image sequence acquisition process. In practical applications, the offset value is often set according to the default value of the overlap rate. The calculation formula is as follows (1);

[0204] 1.8. Overlap rate fluctuation value: refers to the maximum offset between the actual overlap rate of the two images collected in the horizontal or vertical direction and the theoretical overlap rate (i.e. the default overlap rate value) when the image acquisition device moves in the horizontal or vertical direction according to the default overlap rate during the image sequence acquisition process. This maximum offset is affected by factors such as mechanical precision and vibration. The higher the mechanical precision and the smaller the vibration, the smaller the offset value, and vice versa.

[0205] 1.9 Offset fluctuation value: refers to the maximum offset between the actual offset of the two images captured in the horizontal or vertical direction and the theoretical offset (i.e., the default offset value) when the image acquisition device moves in the horizontal or vertical direction at the default overlap rate during the image sequence acquisition process. The maximum offset is affected by factors such as mechanical accuracy and vibration. The higher the mechanical accuracy and the smaller the vibration, the smaller the offset value, and vice versa. In practical applications, it is often calculated based on the overlap rate fluctuation value:

[0206] OffFluX = W x OvlpFluX, (4);

[0207] OffFluY=H x OvlpFluY,

[0208] Among them, OvlpFluX and OvlpFluY are the overlap rate fluctuation values ​​in the horizontal and vertical directions when moving along a certain direction, respectively, and W and H are the image width and height;

[0209] 1.10. Maximum possible overlapping area: refers to the maximum possible overlapping area between the two images captured by the image acquisition device moving horizontally or vertically during the acquisition of the image sequence. This area is a rectangular area, and its position and size are determined by the moving direction, image size (width and height), overlap rate default value, and overlap rate fluctuation value:

[0210] When moving in the horizontal direction, the default value of the horizontal overlap ratio of the right image relative to the left image is hodx, and its fluctuation value is hofx. The default value of the vertical overlap ratio is hody, and its fluctuation value is hofy. At this time, the maximum overlap ratios between the two in the horizontal and vertical directions are maxOvlpX = hodx + hofx and maxOvlpY = hody + hofy respectively. Let OvlpX = maxOvlpX, OvlpY = maxOvlpY, and then according to formulas (1), (2), and (3) respectively, the maximum possible overlapping area when moving in the horizontal direction can be obtained;

[0211] When moving in the vertical direction, the default value of the vertical overlap ratio of the upper image relative to the lower image is vody, and the fluctuation value is vofy. The default value of the horizontal overlap ratio is vodx, and the fluctuation value is vofx.

[0212] At this time, the maximum overlap ratios between the two in the horizontal and vertical directions are maxOvlpX = vodx + vofx and maxOvlpY = vody + vofy respectively. Let OvlpX = maxOvlpX, OvlpY = maxOvlpY, and then use equations (1), (2), and (3) to get the maximum possible overlap area when moving in the vertical direction;

[0213] 1.11. Minimum possible overlapping area: refers to the minimum possible overlapping area between the two images captured by the image acquisition device moving horizontally or vertically during the image sequence acquisition process. This area is a rectangular area, and its position and size are determined by the moving direction, image size (width and height), overlap rate default value, and overlap rate fluctuation value:

[0214] When moving in the horizontal direction, the default value of the horizontal overlap ratio of the right image relative to the left image is hodx, and its fluctuation value is hofx. The default value of the vertical overlap ratio is hody, and its fluctuation value is hofy. At this time, the minimum overlap ratios between the two in the horizontal and vertical directions are minOvlpX = hodx-hofx and minOvlpY = hody-hofy respectively. Let OvlpX = minOvlpX, OvlpY = minOvlpY, and then according to formulas (1), (2), and (3) respectively, the minimum possible overlapping area when moving in the horizontal direction can be obtained;

[0215] When moving in the vertical direction, the default value of the vertical overlap ratio of the upper image relative to the lower image is vody, and the fluctuation value is vofy. The default value of the horizontal overlap ratio is vodx, and the fluctuation value is vofx. At this time, the minimum overlap ratios between the two in the horizontal and vertical directions are minOvlpX = vodx-vofx and minOvlpY = vody-vofy respectively. Let OvlpX = minOvlpX, OvlpY = minOvlpY, and then according to formulas (1), (2), and (3) respectively, the minimum possible overlapping area when moving in the vertical direction can be obtained;

[0216] 1.12. Interest points: Special points of a specified type in an image, consisting of the following information:

[0217] Position, which describes the coordinates of the point in the image;

[0218] Intensity, which describes how special the image is at that point;

[0219] The commonly used "specified types" mentioned above include Harris corner points, SUSAN corner points, FAST corner points, DOG corner points, KLT corner points, edge points, and spots, etc.

[0220] 1.13. Feature descriptor: A feature vector of a specified type calculated in the neighborhood of a given point in an image, used to describe the brightness / color changes of the image in the neighborhood. Commonly used "specified types" here include HOG, LBP, BRIEF, etc.

[0221] 1.14. Feature point: A special local area in an image, consisting of an interest point and a feature descriptor at that point. Therefore, the feature point type is determined by the interest point type and the descriptor type used;

[0222] 1.15. Feature extraction area: The area in the image used to extract feature points of a specified type.

[0223] 1.16. Template: The sub-region in the image used for template matching: B(p,w,h), where p is the lower left corner of the region, w and h are the width and height of the region respectively;

[0224] 1.17. Template extraction area: the area in the image used to extract the template;

[0225] 1.18. Reference image: When performing horizontal stitching, the reference image refers to the left image, or the maximum possible overlapping area on the right side of the left image; when performing vertical stitching, the reference image refers to the lower image, or the maximum possible overlapping area on the upper side of the lower image.

[0226] 1.19. Target image: When performing horizontal stitching, the target image refers to the right image, or the maximum possible overlapping area on the left side of the right image; when performing vertical stitching, the target image refers to the upper image, or the maximum possible overlapping area on the lower side of the upper image.

[0227] 1.20. Point pairs with the same name: refers to the corresponding positions of the same point in space in two different images (i.e., the reference image and the target image), which is composed of a triple: (c; p r ,p o ), where p r and p o are the positions in the reference image and the target image respectively, and c is the similarity of the two images in the overlapping area (calculated as follows) determined by the corresponding positions.

[0228] Each pair of points with the same name can determine an offset: if the corresponding positions of a spatial point in two images A and B are (X A ,Y A ) and (X B ,Y B ), then the offset of image B relative to image A is:

[0229] OffX=X A -X B ;

[0230] OffY=Y A -Y B ;

[0231] Furthermore, the respective overlapping areas can be obtained according to equations (2) and (3).

[0232] 1.21. Template search area: The range that needs to be searched in the corresponding image to find the best matching template for a given template. Assume that the default values ​​of the horizontal and vertical offsets of the target image relative to the reference image are OffDefX and OffDefY respectively, and the offset fluctuation values ​​are OffFluX and OffFluY respectively. Then the template B(p,w b ,h b )The search area in the reference image is:

[0233] S(p) = S0 + p (5);

[0234] S0={(x,y)||x-OffDefX|≤OffFluX,|y-OffDefY|≤OffFluY};

[0235] 1.22. Connection relationship: It is used to indicate the connection position and connection strength between two images in an image sequence, and is represented by a five-tuple: [iRef, iObj, Succ, Offset, Conf], where iRef indicates the serial number of the reference image in the connection relationship in the sequence, iObj indicates the serial number of the target image in the connection relationship in the sequence, Succ is a Boolean variable used to describe whether the offset between the target image and the reference image is found, Offset is the offset of the target image relative to the reference image, and Conf indicates the credibility of the offset.

[0236] 2. Operation module:

[0237] 2.1, [Matching conversion operation module]:

[0238] For target image I o and reference figure I r The image pairs formed by o max and Ω r max , then the target graph I o and reference figure I r The matching problem between them can be transformed into o max and Ω r max The matching problem between sub-images on:

[0239] (OffX,OffY) = (Xs,Ys) r - (Xs,Ys) o + (OffX,OffY) max (6);

[0240] Among them, (OffX, OffY) is the target image I o relative to the reference image I r offset, (Xs, Ys) r is Ω r max at the lower left corner coordinates of I r is (Xs, Ys) o is Ω o max at the lower left corner coordinates of I o is (OffX, OffY) max is Ω o max relative to Ω r max offset;

[0241] That is, for two images to be matched, their maximum possible overlapping sub-images can be intercepted first, then matched between the two sub-images, and after obtaining the offset between the sub-images, the offset between the original images can be calculated using formula (6);

[0242] 2.2.

Blank Detection Operation Module

[0243] For a given grayscale image I, perform the following process to obtain its foreground mask M and determine whether it is blank:

[0244] Threshold calculation; perform grayscale histogram statistics on the image I and calculate a grayscale threshold Tg accordingly;

[0245] Foreground mask extraction; if the image is a bright-field image (meaning the background is brighter), its foreground mask is M = {p > Tg | p ∈ I}; if the image is a fluorescence image (meaning the background is darker), its foreground mask is M = {p < Tg | p ∈ I}, where p is the grayscale value of each pixel in the image. M is a binary matrix of the same size as I, where the element value of 1 represents a foreground pixel and 0 represents a background pixel;

[0246] Blank judgment; if the ratio of the total number of foreground pixels (i.e., the number of elements with a value of 1 in the mask M) to the total number of pixels in the image exceeds a given threshold Tb, the image is judged to be blank, otherwise it is non-blank. Among them, Tb is a preset threshold, and its value ranges from 0 to 0.05;

[0247] 2.3.

Blur Detection Operation Module

[0248] For the image pair composed of the target image I o and the reference image I r and their respective foreground masks Mo and Mr, perform the following process to obtain their respective gradient maps Go and Gr, and sharpnesses So and Sr, and determine whether one of them is blurred:

[0249] i. For each pixel in the target image, a gradient operator such as Robert operator, Sobel operator or Laplacian operator is used to calculate its gradient, thereby generating a corresponding gradient image Go;

[0250] ii. For each pixel in the reference image, the same gradient operator is used to calculate its gradient, thereby generating the corresponding gradient map Gr;

[0251] iii. According to the foreground mask Mo and the gradient map Go of the target image, the gradient mean of each foreground pixel of the target image is calculated as the clarity So of the target image;

[0252] iv. According to the foreground mask Mr and gradient map Gr of the reference image, the gradient mean of each foreground pixel of the reference image is calculated as the clarity Sr of the reference image;

[0253] v. If So / Sr>Ts, the reference image is judged to be a blurred image; if Sr / So>Ts, the target image is judged to be a blurred image; wherein Ts is a pre-set threshold value, which is a floating point number greater than or equal to 3.

[0254] 2.4, [Template extraction operation module]:

[0255] Given a grayscale image I, a corresponding gradient map G, and a template extraction region E, the following process is performed to obtain a template sequence of a preset size {B k}, k = 0, 1, ... K-1:

[0256] According to the gradient map G and the template extraction region E, an integral map S corresponding to the template extraction region E is generated.

[0257] According to the integral graph S, the gradient with preset size and the largest K templates are obtained, denoted as {B k}, k=0,1,…K-1. The preset size can usually be 7x7, 9x9, 11x11, 13x13…etc. so that the template is a square with an odd side length.

[0258] 2.5. [Template matching operation module]:

[0259] For target image I o and reference figure I r The image pairs formed are subjected to the following process to obtain a set C of point pairs with the same name that meet the template matching requirements:

[0260] If the minimum possible overlapping area Eo in the target image is a blank area, then:

[0261] If the minimum possible overlapping area Er in the reference image is a blank area, then:

[0262] If the definition So of the target image is greater than the definition Sr of the reference image, then:

[0263] i. Take the entire target image as the template extraction area, execute the [Template Extraction Operation Module] in the target image, and obtain the template sequence {B o k (p o k ,w,h)},k=0,1,…K-1;

[0264] ii. For each template B o k (p o k ,w,h), find the best matching template B in the reference image r k (p r k ,w,h)

[0265] and the corresponding similarity c k :

[0266] [B r k (p r k ,w,h),c k ]=argmax{c=s(B o k (p o k ,w,h),B r (p r ,w,h))|p r ∈S(p o k )};

[0267] Among them, s(X,Y) is the similarity measure between variables X and Y (which can be taken as Pearson correlation coefficient, the same below), S(p o k ) is the corresponding search area of ​​the template in the reference image. k ≥T m , then C k =(c; p r k ,p o k ) are added to the set C of pairs of points with the same name. Among them, T m is the preset template matching threshold (range 0.5 to 1.0, the same below).

[0268] otherwise,

[0269] i. Take the entire reference image as the template extraction area, execute the [Template Extraction Operation Module] in the reference image, and obtain the template sequence {B r k (p r k ,w,h)},k=0,1,…K-1;

[0270] ii. For each template B r k (p r k ,w,h), find the best matching template B in the target graph o k (p o k ,w,h)

[0271] and the corresponding similarity c k :

[0272] [B o k (p o k ,w,h),c k ]=argmax{c=s(B r k (p r k ,w,h),B o (p o ,w,h))|p o ∈S(p r k )};

[0273] Among them, S(p r k ) is the corresponding search area of ​​the template in the target graph. k ≥T m , then C k =(c; p r k ,p o k ) are added to the set C of point pairs with the same name.

[0274] otherwise,

[0275] i. Take the minimum possible overlap of the reference image as the template extraction area, execute the [Template Extraction Operation Module], and obtain the template sequence {B r k (p r k ,w,h)},k=0,1,…K-1;

[0276] ii. For each template B rk (p r k ,w,h), find the best matching template B in the target graph o k (p o k ,w,h)

[0277] and the corresponding similarity c k :

[0278] [B o k (p o k ,w,h),c k ]=argmax{c=s(B r k (p r k ,w,h),B o (p o ,w,h))|p o ∈S(p r k )};

[0279] Among them, S(p r k ) is the corresponding search area of ​​the template in the target graph. k ≥T m , then C k =(c; p r k ,p o k ) are added to the set C of point pairs with the same name;

[0280] otherwise,

[0281] If the minimum possible overlapping area Er in the reference image is a blank area, then:

[0282] i. Extract the region using the minimum possible overlapping region of the target image as a template and perform

[0283]

Template extraction module

[0284] ii. For each template B o k (p o k ,w,h), find the best matching template B in the reference image rk (p r k ,w,h) and the corresponding similarity c k :

[0285] [B r k (p r k ,w,h),c k ]=argmax{c=s(B o k (p o k ,w,h),B r (p r ,w,h))|p r ∈S(p o k )}

[0286] Among them, S(p o k ) is the corresponding search area of ​​the template in the reference image. k ≥T m , then C k =(c; p r k ,p o k ) are added to the set C of point pairs with the same name.

[0287] otherwise,

[0288] If the minimum possible overlap area definition So of the target image is greater than the minimum possible overlap area definition Sr of the reference image, then:

[0289] i. Take the minimum possible overlapping area of ​​the target image as the template extraction area, execute the [Template Extraction Operation Module] in the target image, and obtain the template sequence {B o k (p o k ,w,h)},k=0,1,…K-1;

[0290] ii. For each template B o k (p o k ,w,h), find the best matching template B in the reference image r k (p r k ,w,h) and the corresponding similarity c k :

[0291] [B r k(p r k ,w,h),c k ]=argmax{c=s(B o k (p o k ,w,h),B r (p r ,w,h))|p r ∈S(p o k )}

[0292] If c k ≥T m , then C k =(c; p r k ,p o k ) are added to the set C of point pairs with the same name.

[0293] otherwise,

[0294] i. Take the minimum possible overlapping area of ​​the reference image as the template extraction area, execute the [Template Extraction Operation Module] in the reference image, and obtain the template sequence {B r k (p r k ,w,h)},k=0,1,…K-1;

[0295] ii. For each template B r k (p r k ,w,h), find the best matching template B in the target graph o k (p o k ,w,h) and the corresponding similarity c k :

[0296] [B o k (p o k ,w,h),c k ]=argmax{c=s(B r k (p r k ,w,h),B o (p o ,w,h))|p o ∈S(p r k )};

[0297] If c k ≥T m , then C k =(c; p r k ,p o k ) are added to the set C of point pairs with the same name.

[0298] 2.6. [Feature matching operation module]:

[0299] For target image I o and reference figure I r The image pairs formed by the above method perform the following process to obtain their respective feature point sets. and And the set C of point pairs with the same name that meet the feature matching requirements:

[0300] i. Extract feature points from the target image. For example, the feature points can be FAST corner points as interest points and BRIEF descriptors as feature vectors to obtain a feature point set:

[0301] F o ={F o i |i=0,1,…N-1},F o i =(p o i ,v o i ),

[0302] where p o i is the location of the feature point (i.e., the location of the interest point contained in the feature point, the same below), v o i is the descriptor at the feature point, that is, the feature vector.

[0303] ii. Extract feature points from the reference image. The feature point type used should be at least the same as the descriptor type in the target image, and obtain the feature point set:

[0304]

[0305] where p r j is the location of the feature point, v r j is the descriptor at the feature point, that is, the feature vector.

[0306] iii. Find the set of points with the same name that meet the feature matching requirements:

[0307] C={C mn =(c; p rm ,p o n )|c=s(v r m ,v o n )>T f ,0≤m≤M-1,0≤n≤N-1}

[0308] Among them, s(X,Y) is the similarity measure between variables X and Y, which can be taken as the Pearson correlation coefficient, T f It is the preset feature matching threshold, ranging from 0.5 to 1.0.

[0309] 2.7. [Horizontal mean matching operation module]:

[0310] For two horizontally adjacent visual field images, the maximum possible overlapping area of ​​the left image is I L , the maximum possible overlapping area of ​​the right figure is I R . Perform the following process to obtain I R Relative to I L The optimal offset o b =(OffX,OffY):

[0311] i. Calculate I separately L and I R The column mean curve C L ={c L 0,c L 1,…c L w-1} and C R ={c R 0,c R 1,…c R w-1}, where w is I L width; then, with C R For templates in C L The moving correlation calculation is performed on the above, and the moving range is twice the horizontal offset fluctuation value. The maximum correlation position found during the moving process is recorded as k x , the corresponding correlation is c x If c x >T c , then I R Relative to I L The optimal offset in the horizontal direction is OffX = k x , go to the next step, otherwise the matching fails and exit this module. c is the preset correlation threshold, ranging from 0.5 to 1.0, the same below.

[0312] ii. Calculate subgraph I separatelyL [k x ,w;0,h] and I R [0,wk x ; 0,h] row mean curve C L = {r L 0,r L 1,…r L h-1} and C R = {r R 0,r R 1,…r R h-1}, where h is I L The height of C R For templates in C L The moving correlation calculation is performed on the above, and the moving range is twice the vertical offset fluctuation value. The maximum correlation position found during the moving process is recorded as k y , the corresponding correlation is c y If c y >T c , then I R Relative to I L The optimal offset in the vertical direction is OffY = k y , the match succeeds; otherwise, the match fails.

[0313] 2.8. [Vertical mean matching operation module]:

[0314] For two adjacent visual field images, the maximum possible overlapping area of ​​the following images is recorded as I B , the maximum possible overlapping area of ​​the above figure is I T . Perform the following process to obtain I T Relative to I B The optimal offset o b =(OffX,OffY):

[0315] i. Calculate I separately B and I T The row mean curve C B = {r B 0,r B 1,…r B h-1} and C T = {r T 0,r T 1,…r T h-1}, where h is I B The height of C B For templates in C TThe moving correlation calculation is performed on the above, and the moving range is twice the vertical offset fluctuation value. The maximum correlation position found during the moving process is recorded as k y , the corresponding correlation is c y If c y >T c , then I T Relative to I B The optimal offset in the vertical direction is OffY = k y , proceed to the next step, otherwise the matching fails and exit this module.

[0316] ii. Calculate subgraph I separately B [0,w;k y ,h] and I T [0,w; 0,hk y ] of the column mean curve C B ={c B 0,c B 1,…c B w-1} and C T ={c T 0,c T 1,…c T w-1}, where w is I B width; then, with C T For templates in C B The moving correlation calculation is performed on the above, and the moving range is twice the horizontal offset fluctuation value. The maximum correlation position found during the moving process is recorded as k x , the corresponding correlation is c x If c x >T c , then I T Relative to I B The horizontal offset is OffX = k x , the match succeeds; otherwise, the match fails.

[0317] 2.9. [Overall optimization operation module]:

[0318] Target Image I o and reference figure I r The set C of points with the same name between them is solved to find the optimal offset o between them. b and similarity c b :

[0319] [o b ,c b ]=argmax{c=f(C mn ,I o ,I r )|Cmn ∈C};

[0320] f(C mn ,I o ,I r )=s(I o (Ω o ),I r (Ω r ));

[0321] Ω r =Ω∩(Ω+o),;

[0322] Ω o =(Ω∩(Ω+o))-o;

[0323] Ω=[0,W;0,H];

[0324] o=p r m -p o n ;

[0325] Among them, p r m and p o n They are the feature points F r m and F o n The position of the point pair, o is the offset of the target image corresponding to the point pair relative to the reference image, W and H are the width and height of the image respectively, Ω is the rectangular area where the image is located, Ω r is the overlapping area in the reference image, Ω o is the overlapping area in the target graph, and s(X,Y) is the similarity measure between variables X and Y, which can be taken as the Pearson correlation coefficient.

[0326] 2.10, [Offset Optimization Operation Module]:

[0327] Target Image I o and reference figure I r The current optimal offset between b , in a neighborhood with a radius of 1, the best offset o is obtained by the following calculation max and the corresponding similarity c max :

[0328] [o max ,c max ]=argmax{c=g(o,I o ,I r )|∥oo b ∥1≤1};

[0329] g(o,Io ,I r )=s(I o (Ω o ),I r (Ω r ));

[0330] Ω r =Ω∩(Ω+o);

[0331] Ω o =(Ω∩(Ω+o))-o;

[0332] Ω=[0,W;0,H];

[0333] Among them, ∥●∥1 is the 1 norm, o b is the offset of the target image relative to the reference image, W and H are the width and height of the image respectively, Ω is the rectangular area where the image is located, and Ω r is the overlapping area in the reference image, Ω o is the overlapping area in the target graph, and s(X,Y) is the similarity measure between variables X and Y, which can be taken as the Pearson correlation coefficient.

[0334] 2.11. [Original resolution optimization operation module]:

[0335] Let I o and I r The target graph I oc and reference figure I rc A half-resolution version of I o Relative to I r The optimal offset is o b , then I oc Relative to I rc The optimal offset o max and similarity c max It can be obtained through the following process:

[0336] [o max ,c max ]=argmax{c=g(o,I oc ,I rc )|∥o-2o b ∥1≤1};

[0337] g(o,I oc ,I rc )=s(I oc (Ω o ),I rc (Ω r ));

[0338] Ω r =Ω∩(Ω+o),;

[0339] Ω o =(Ω∩(Ω+o))-o;

[0340] Ω=[0,W;0,H];

[0341] Among them, ∥●∥1 is the 1 norm, W and H are image I oc The width and height of the image, Ω is the rectangular area where the image is located, Ω r For reference I rc The overlapping area in Ω o Target Image I oc The overlapping area in , s(X,Y) is the similarity measure between variables X and Y, which can be taken as the Pearson correlation coefficient.

[0342] 2.12. [Overlap rate update operation module]:

[0343] Assume that the target graph I has been found o and reference figure I r The optimal offset between b , then update the overlap rate during the stitching process according to the following process:

[0344] If I o and I r If the two layers are horizontally adjacent, the overlap ratios OffX and OffY are calculated according to (1), and then hodx = OffX, hofx = 0.05, hody = 1.0, hofy = 0.05;

[0345] If I o and I r If the two layers are vertically adjacent to each other, the overlap ratios OffX and OffY are calculated according to (1), and then vodx = 1.0, vofx = 0.05, vody = OffY, vofy = 0.05;

[0346] 2.13. [Image matching operation module 1]:

[0347] For target image I o and reference figure I r The image pairs are constructed and the following process is performed:

[0348] i. Execute the [Feature Matching Operation Module] to obtain the set C of points with the same name between the reference image and the target image. If the set is not empty, proceed to the next step; otherwise, the matching fails and this template is exited;

[0349] ii. Execute the [Overall Optimization Operation Module] to obtain the optimal pair of points with the same name C b =(c b ;

[0350] p rmb ,p o nb ), if c b ≥Tc, then the optimal offset o of the target image relative to the reference image is obtained by this pair of points with the same name b =p r mb -p o nb , and then proceed to the next step; otherwise, the match fails and exit this template;

[0351] iii. Execute the [Offset Optimization Operation Module] to obtain the final offset of the target image relative to the reference image. max , match successful, exit this module.

[0352] 2.14, [Image matching operation module 2]:

[0353] For target image I o and reference figure I r The image pairs formed are executed as follows:

[0354] i. Execute the [Template Matching Operation Module] to obtain the set C of points with the same name between the reference image and the target image. If the set is not empty, proceed to the next step; otherwise, the matching fails and exit this module;

[0355] ii. Execute the [Overall Optimization Operation Module] to obtain the optimal pair of points with the same name C b =(c b ;p r mb ,p o nb ), if c b ≥Tc, the match is successful, and the final offset o of the target image relative to the reference image is obtained from the same-name point pair max =p r mb -p o nb , then exit this module;

[0356] Otherwise, the match fails and the module exits.

[0357] 2.15. [Image matching operation module 3]:

[0358] For left and right adjacent image pairs, perform the following process:

[0359] i. Execute the [horizontal mean matching operation module] to obtain the optimal offset c of the right image relative to the left image b If successful, proceed to the next step; otherwise, the match fails and exit this module;

[0360] ii. Execute the [Offset Optimization Operation Module] to obtain the final offset of the right image relative to the left image. max , match successful, exit this module.

[0361] 2.16, [Image matching operation module 4]:

[0362] For the upper and lower adjacent image pairs, perform the following process:

[0363] i. Execute the [vertical mean matching operation module] to obtain the optimal offset c of the upper image relative to the lower image b If successful, proceed to the next step; otherwise, the match fails and exit this module;

[0364] ii. Execute the [Offset Optimization Operation Module] to obtain the final offset of the upper image relative to the lower image. max If the match is successful, exit this module.

[0365] 3. Algorithm process:

[0366] The splicing algorithm of this application consists of the following stages:

[0367] 3.1, [Initialization]:

[0368] A Boolean variable ho_known is set to indicate whether the horizontal overlap ratio is known, and a Boolean variable vo_known is set to indicate whether the vertical overlap ratio is known. A Boolean variable downsample is set to indicate whether the maximum possible overlap area is downsampled.

[0369] If the horizontal overlap rate information of the image sequence is known, then input the known hodx, hofx, hofy, and set ho_known = true; otherwise, set: hodx = 1.0, hofx = 0.5, hofy = 0.05, and set the Boolean variable ho_known = false; at this time, the maximum possible overlap range in the horizontal direction is the width of the entire image;

[0370] If the vertical overlap rate information of the image sequence is known, input the known vody, vofy, vofx, and set vo_known = true; otherwise, set: vody = 1.0, vofy = 0.5, vofx = 0.05, and set the Boolean variable vo_known = false; at this time, the maximum possible overlap range in the vertical direction is the entire image height.

[0371] 3.2, [Partial splicing]:

[0372] The purpose of this stage is to calculate the connection relationship between every two adjacent images in the image sequence. r,c, for r = 0, 1, …, R - 1 and c = 0, 1, …, C - 1, the following processing is performed in sequence:

[0373] Step1 Image loading. Load the image I r,c into memory;

[0374] Step2 Grayscale conversion. Convert the image to a grayscale image for subsequent calculations;

[0375] Step3 Crop sub - images of the maximum possible overlapping regions:

[0376] If r > 0, crop the upper - side maximum possible overlapping region of the current image, denoted as I r,c [U];

[0377] If r < R - 1, crop the lower - side maximum possible overlapping region of the current image, denoted as I r,c [D];

[0378] If c > 0, crop the left - side maximum possible overlapping region of the current image, denoted as I r,c [L];

[0379] If c < C - 1, crop the right - side maximum possible overlapping region of the current image, denoted as I r,c [R];

[0380] After cropping the sub - images in each direction, remove the current image from memory to save memory.

[0381] Step4 Bidirectional stitching, that is, stitch the current image with the horizontally and vertically adjacent processed images:

[0382] Step4.1 If r > 0, perform vertical stitching:

[0383] Step4.1.0 Data preparation:

[0384] Take out the upper - side sub - image I r,c [U] of the current image as the "reference image", denoted as I r ; take out the lower - side sub - image I r-1,c [D] of the upper - neighborhood image of the current image, called the "target image", denoted as I o .

[0385] Step4.1.1 Downsampling:

[0386] If both the width and height of the target image are greater than the given threshold Ts, back up the target image and the reference image, denoted as I oc and I rc, then downsample the reference image and the target image to halve their resolution, and assign a value to the downsampling flag: downsample = true; otherwise, assign a value to the downsampling flag: downsample = false.

[0387] Step 4.1.2 Blank detection:

[0388] Perform blank detection on the target image and the reference image, and execute the [Blank Detection Module]. If one of them is judged to be blank, terminate the matching and jump to Step 4.2 for horizontal matching;

[0389] Otherwise, proceed to the following fuzzy detection step.

[0390] Step 4.1.3 Fuzzy detection:

[0391] Execute the [Blur Detection Module] on the target image and the reference image. If at least one of them is judged to be blurry, terminate the matching and jump to Step 4.2 for horizontal matching; otherwise, enter the following image matching link.

[0392] Step4.1.4 Image matching:

[0393] If vo_known == false, then:

[0394] i. Execute [Image Matching Module 1]. If successful, assign vo_known = true and go to iii. Otherwise, go to ii.

[0395] ii. Execute [Image Matching Module 2]. If successful, assign vo_known = true, and then go to iii;

[0396] iii. If downsample == true, execute the [original resolution optimization module], and then execute the [overlap rate update module];

[0397] If vo_known == true, then:

[0398] a. Execute [Image Matching Module 4], if successful, go to b.

[0399] b. If downsample == true, execute [Original Resolution Optimization Module]

[0400] Step4.1.5 Record the connection relationship:

[0401] If the above match is successful, then:

[0402] [iRef,iObj,Succ,Offset,Conf]=[r*C+c,(r-1)*C+c,true,o max ,c max ];

[0403] [iRef,iObj,Succ,Offset,Conf]=[(r-1)*C+c,r*C+c,true,-o max ,c max ];

[0404] otherwise:

[0405] [iRef,iObj,Succ,Offset,Conf]=[r*C+c,(r-1)*C+c,false,0,-10];

[0406] [iRef,iObj,Succ,Offset,Conf]=[(r-1)*C+c,r*C+c,false,0,-10];

[0407] Among them, max is the offset of the target image relative to the reference image found in the above matching step, c max is the similarity corresponding to the offset.

[0408] Step 4.2 If c>0, perform horizontal splicing:

[0409] Step4.2.0 Data preparation:

[0410] Take out the right sub-image I of the left neighboring image of the current image r,c-1 [R], as the "reference image",

[0411] For I r , and the left sub-image I of the current image r,c [L], called the "target graph", denoted by I o .

[0412] Step 4.2.1 Downsampling:

[0413] If the width and height of the target image are both greater than the given threshold Ts, the backup target image and the reference image are recorded as I and oc and I rc , then downsample the reference image and the target image to halve their resolution, and assign a value to the downsampling flag: downsample = true; otherwise, assign a value to the downsampling flag: downsample = false.

[0414] Step 4.2.2 Blank detection:

[0415] Perform blank detection on the target image and the reference image, and execute the [Blank Detection Operation Module]. If one of the two is judged to be blank, terminate the match and jump to Step 4.2 for horizontal matching; otherwise, enter the following fuzzy detection link.

[0416] Step 4.2.3 Fuzzy detection:

[0417] Execute the [Blur Detection Operation Module] on the target image and the reference image. If at least one of them is judged to be blurry, terminate the matching and jump to Step 4.2 for horizontal matching.

[0418] Otherwise, proceed to the following image matching step.

[0419] Step4.2.4 Image matching:

[0420] If ho_known == false, then:

[0421] i. Execute [Image Matching Operation Module 1], if successful, assign a value

[0422] ho_known = true, then go to iii, otherwise go to ii,

[0423] ii. Execute [Image Matching Operation Module 2], if successful, assign a value

[0424] ho_known = true, then go to iii;

[0425] iii. If downsample == true, execute the [original resolution optimization module], and then execute the [overlap rate update operation module];

[0426] If vo_known == true, then:

[0427] a. Execute [Image Matching Operation Module 3], if successful, go to b;

[0428] b. If downsample == true, execute [Original Resolution Optimization Operation Module];

[0429] Step4.2.5 Record the connection relationship:

[0430] If the above match is successful, then:

[0431] [iRef,iObj,Succ,Offset,Conf]=[r*C+c-1,r*C+c,true,o max ,c max ];

[0432] [iRef,iObj,Succ,Offset,Conf]=[r*C+c,r*C+c-1,true,-o max ,c max ];

[0433] otherwise:

[0434] [iRef,iObj,Succ,Offset,Conf]=[r*C+c-1,r*C+c,false,0,-10];

[0435] [iRef,iObj,Succ,Offset,Conf]=[r*C+c,r*C+c-1,false,0,-10];

[0436] Among them, max is the offset of the target image relative to the reference image found in the above matching step, c max is the similarity corresponding to the offset.

[0437] 3.3, [Coordinate Allocation]:

[0438] Through the above-mentioned [local stitching] stage, the connection relationship between any two adjacent images in the entire image sequence (if the match is successful, it is expressed by the offset value) is obtained. For an image sequence consisting of R rows and C columns, there are a total of K = R*(C-1) + C*(R-1) connection relationships: {G k}, k = 0, 1, ..., K-1. The present application calculates the position {P[n]}, n = 0, 1, ..., R*C-1 of each field of view image in the entire panoramic image by the following algorithm:

[0439] Step 0: Initialize the stitched view list to be empty;

[0440] Step 1: Find the one with the highest credibility G from all successfully matched connections. max As the starting point, add it to the list of spliced ​​views (at this time, there are only two views in the list), and P[G max (iRef)]=(0,0), P[G max (iObj)]=G max (Offset), where: G max (iRef) means to extract the iRef in the connection relationship, that is, the reference figure number, G max (iObj) means to extract iObj in the connection relationship, that is, the target graph number, G max (Offset) means to extract the Offset in the connection relationship, that is, the offset, the same below;

[0441] Step 2: Find the optimal connection relationship G from the unused connection relationships in the spliced ​​view list. max , the latter satisfies: 1) the match is successful, that is, Succ == true, 2) it is connected to the spliced ​​view, that is, iRef is in the spliced ​​view list, 3) the credibility is the largest, that is, Conf is the largest.

[0442] Step 3 If G max If exists, then G max (iObj) is added to the list of stitched fields of view, and its coordinate position is calculated as follows:

[0443] P[G max (iObj)]=P[G max (iRef)]+G max (Offset),

[0444] Then go to Step 2; otherwise go to the next step.

[0445] Step 4 If the length of the spliced ​​view list is not equal to R*C, then:

[0446] Step 4.1 Count the used connection relationships and obtain the average horizontal and vertical offsets HOffset and VOffset;

[0447] Step 4.2 Find a view iDst that is adjacent to a certain stitched view iSrc from the views that have not been added to the stitched view list. Then add it to the stitched view list and calculate its coordinate position:

[0448] If iSrc and iDst are adjacent horizontally, their coordinate positions are P[iDst]=P[iSrc]+HOffset.

[0449] If they are vertically adjacent, their coordinate positions are P[iDst]=P[iSrc]+VOffset.

[0450] Step 4.3 If the stitched view list is still not equal to R*C, go to Step 4.2, otherwise exit.

[0451] 3.4, [Image Fusion]:

[0452] As the last stage of image stitching, it aims to eliminate the inconsistency of illumination and color difference between adjacent images in the overlapping area, so as to achieve a smooth transition between adjacent images in the panoramic image. Considering the unilateral blur problem in the overlapping area (see Figure 4 ), this application proposes the following unilateral fusion algorithm based on clarity:

[0453] Step 1: Assume that the offset of image B relative to image A is o, then calculate the overlapping area Ω of the two according to formula (2): A and Ω B ;

[0454] Step 2 uses the same difference or gradient calculation subgraph A[Ω A ] and B[Ω B ] A and S B ;

[0455] Step 3 If S A >S B ,but:

[0456] M=A[Ω A1 ]∪F(A[Ω A2 ],B[Ω B2 ]);

[0457] Ω A =Ω A1 ∪Ω A2 ;

[0458] Ω B =Ω B1 ∪Ω B2 ;

[0459] otherwise,

[0460] M=B[Ω B1 ]∪F(A[Ω A2 ],B[Ω B2 ]);

[0461] Where M is the fusion result of the overlapping sub-images, F(I1,I2) is the fusion operator for arbitrary images I1 and I2, such as the most commonly used linear fusion, ∪ is the union operator. The parameter w is the fusion width, which divides the overlapping area into two sub-areas, and can generally be 10 to 30.

[0462] See also Fig. 9 and Fig.10 , Fig. 9 A schematic diagram of a clarity-based fusion algorithm provided in an embodiment of the present application, Fig.10 A schematic diagram of the process flow of the splicing algorithm provided in the embodiment of the present application; Figures 1 to 10The method embodiment shown, the processing method for stitching sequential images provided by the embodiment of the present application includes at least the following beneficial effects: it can be used for stitching sequential images in any occasion where there is only translation transformation between adjacent images, such as various automatic scanning equipment (pathological slice scanners, AOI detection equipment, etc.). By introducing fast blank detection and blur detection, mismatching is avoided and the speed is improved; for image sequences with unknown overlap rate range, the overlap rate range is found through a fast local and overall combination matching strategy, thereby improving the versatility of the algorithm; for situations where the overlap rate range is known, the present invention proposes a fast mean matching method to improve the stitching speed; by introducing a clarity-based fusion algorithm, the proportion of blurred areas in the panorama is reduced, thereby improving the image quality.

[0463] See also Fig.11 , Fig.11 This is a structural block diagram of a processing system for stitching sequential images provided in an embodiment of the present application. The processing system for stitching sequential images includes:

[0464] The sequence image unit 100 is used to obtain the image sequence data to be spliced, and the image sequence data includes multiple rows and columns of image data;

[0465] Initialization unit 200, used for performing initialization processing according to the image sequence data to be spliced ​​to obtain Boolean variable information;

[0466] The local splicing unit 300 is used to traverse the image sequence data to be spliced ​​according to a preset order, and perform local splicing processing on the traversed two adjacent image data according to the Boolean variable information to obtain connection relationship information;

[0467] A coordinate allocation unit 400 is used to perform coordinate allocation processing on the image sequence data to be spliced ​​according to the connection relationship information to obtain coordinate allocation data;

[0468] The image splicing unit 500 is used to splice the image data of multiple rows and columns in the image sequence data according to the coordinate allocation data to obtain spliced ​​image data.

[0469] Exemplarily, the initialization unit 200 is specifically used for:

[0470] Performing initialization processing according to the image sequence data to be spliced, obtaining the horizontal overlap rate, vertical overlap rate, and overlap area-downsampling information of adjacent image data;

[0471] Boolean variable information is generated according to the horizontal overlap rate, the vertical overlap rate and the overlap area-downsampling information.

[0472] Exemplarily, the local splicing unit 300 is further used for:

[0473] Traverse the image sequence data to be spliced, and perform the following processing on the traversed image data:

[0474] Load image data into preset memory;

[0475] Convert image data into grayscale image data;

[0476] The grayscale image data is intercepted and processed according to the Boolean variable information to obtain a maximum possible overlapping area sub-image corresponding to the image data, and the image data is removed from a preset memory, wherein the maximum possible overlapping area sub-image includes one or more of the upper maximum possible overlapping area, the lower maximum possible overlapping area, the right maximum possible overlapping area, and the left maximum possible overlapping area.

[0477] Exemplarily, the local splicing unit 300 is specifically used for:

[0478] The image sequence data to be spliced ​​is traversed according to the preset order, and the traversed image data is processed as follows:

[0479] determining an image adjacent to the image data as adjacent image data;

[0480] The maximum possible overlapping area sub-image corresponding to the image data is used as a reference image, and the maximum possible overlapping area sub-image of the adjacent image data is used as a target image;

[0481] If the size of the target image is larger than a preset threshold, downsampling the reference image and the target image is performed to obtain the downsampled reference image and the target image;

[0482] Perform blank detection on the target image and the reference image. If one of the target image and the reference image is detected as blank, stop the local stitching process.

[0483] Perform blur detection on the target image and the reference image. If one of the target image and the reference image is detected as blurry, stop the local stitching process.

[0484] Perform local splicing processing on the target graph and the reference graph to obtain sub-connection relationship information;

[0485] After the traversal is completed, the connection relationship information is obtained based on all the sub-connection relationship information.

[0486] Exemplarily, the local splicing unit 300 is further used for:

[0487] According to the Boolean variable information, it is determined whether the Boolean variable corresponding to the overlap rate in the splicing direction is true;

[0488] If the Boolean variable is not true, one or more local splicing processes among feature matching operation, overall optimization operation, and offset optimization operation are performed on the target image and the reference image;

[0489] If the splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained;

[0490] If the splicing fails, the target image and the reference image are subjected to one or more local splicing processes in the template matching operation and the overall optimization operation. If the splicing succeeds, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained;

[0491] If the Boolean variable is true, one or more local splicing processes in the splicing direction mean matching operation and the offset optimization operation are performed on the target image and the reference image. If the splicing is successful, the sub-connection relationship information is obtained;

[0492] If the reference image and the target image have been downsampled, the original resolution optimization operation is performed on the reference image and the target image, and the overlap rate update operation is performed on the Boolean variable information.

[0493] Exemplarily, the coordinate allocation unit 400 is specifically used for:

[0494] Traverse the connection relationship information to obtain the sub-connection relationship information corresponding to the maximum credibility;

[0495] Add the image data corresponding to the sub-connection relationship information to the spliced ​​field of view list;

[0496] Centered on the list of stitched fields of view, the coordinate allocation data is obtained by sequentially matching the connection relationship information and the image sequence data to be stitched.

[0497] Exemplarily, the image stitching unit 500 is further used for:

[0498] Performing definition calculation on the overlapping area of ​​the image data according to a preset difference algorithm or a preset gradient algorithm to obtain definition data;

[0499] The stitched image data is subjected to image fusion processing according to the definition data to obtain stitched image data after image fusion.

[0500] It should be noted that the processing system for stitching sequential images provided in the embodiment of the present application is similar to Figures 1 to 10 The method embodiments shown correspond to each other and will not be described again here to avoid repetition.

[0501] This application also provides an electronic device, see Fig.12 , Fig.12A block diagram of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 510, a communication interface 520, a memory 530, and at least one communication bus 540. The communication bus 540 is used to realize direct connection and communication between these components. The communication interface 520 of the electronic device in the embodiment of the present application is used to communicate signaling or data with other node devices. The processor 510 may be an integrated circuit chip with signal processing capabilities.

[0502] The processor 510 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor 510 can also be any conventional processor, etc.

[0503] The memory 530 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 530 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 510, the electronic device can execute the above-mentioned Figures 1 to 10 The method embodiment involves various steps.

[0504] Optionally, the electronic device may further include a storage controller and an input / output unit.

[0505] The memory 530, storage controller, processor 510, peripheral interface, input and output unit components are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 540. The processor 510 is used to execute executable modules stored in the memory 530, such as software function modules or computer programs included in the electronic device.

[0506] The input and output unit is used to provide users with the task creation and to create a start optional time period or preset execution time for the task to realize the interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.

[0507] Understandably, Fig.12 The structure shown is for illustration only, and the electronic device may also include Fig.12 More or fewer components as shown, or with Fig.12 Different configurations are shown. Fig.12 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0508] An embodiment of the present application further provides a storage medium having instructions stored thereon. When the instructions are run on a computer, the computer program is executed by a processor to implement the method described in the method embodiment. To avoid repetition, the method will not be described here.

[0509] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the method embodiment.

[0510] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0511] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0512] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0513] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0514] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0515] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A processing method for stitching sequence images, It is characterized in that include: Acquire image sequence data to be spliced, wherein the image sequence data includes image data of multiple rows and columns; Performing initialization processing according to the image sequence data to be spliced ​​to obtain Boolean variable information; Traversing the image sequence data to be spliced ​​according to a preset order, and performing local splicing processing on two adjacent image data traversed according to the Boolean variable information to obtain connection relationship information; Performing coordinate allocation processing on the image sequence data to be spliced ​​according to the connection relationship information to obtain coordinate allocation data; Image data of multiple rows and columns in the image sequence data are spliced ​​according to the coordinate allocation data to obtain spliced ​​image data.

2. The method for stitching sequential images according to claim 1, It is characterized in that The step of performing initialization processing according to the image sequence data to be spliced ​​to obtain Boolean variable information includes: Performing initialization processing according to the image sequence data to be spliced, obtaining the horizontal overlap rate, vertical overlap rate, and overlap area-downsampling information of adjacent image data; Boolean variable information is generated according to the horizontal overlap rate, the vertical overlap rate and the overlap area-downsampling information.

3. The method for stitching sequential images according to claim 2, It is characterized in that Before the step of traversing the image sequence data to be spliced ​​according to a preset order and performing local splicing processing on two adjacent image data traversed according to the Boolean variable information to obtain connection relationship information, the method further includes: Traverse the image sequence data to be spliced, and perform the following processing on the traversed image data: Loading the image data into a preset memory; Converting the image data into grayscale image data; The grayscale image data is intercepted and processed according to the Boolean variable information to obtain a maximum possible overlapping area sub-image corresponding to the image data, and the image data is removed from the preset memory, wherein the maximum possible overlapping area sub-image includes one or more of the maximum possible overlapping area on the upper side, the maximum possible overlapping area on the lower side, the maximum possible overlapping area on the right side, and the maximum possible overlapping area on the left side.

4. The method for stitching sequential images according to claim 3, It is characterized in that The step of traversing the image sequence data to be spliced ​​according to a preset order, and performing local splicing processing on two adjacent image data traversed according to the Boolean variable information to obtain connection relationship information includes: The image sequence data to be spliced ​​is traversed according to a preset order, and the traversed image data is processed as follows: determining an image adjacent to the image data as adjacent image data; Using the maximum possible overlapping area sub-image corresponding to the image data as a reference image, and using the maximum possible overlapping area sub-image of the adjacent image data as a target image; If the size of the target image is greater than a preset threshold, downsampling the reference image and the target image to obtain the downsampled reference image and the target image; Performing blank detection on the target image and the reference image, and stopping the local stitching process if one of the target image and the reference image is detected as blank; Performing blur detection on the target image and the reference image, and stopping the local stitching process if one of the target image and the reference image is detected as blurry; Performing local splicing processing on the target graph and the reference graph to obtain sub-connection relationship information; After the traversal is completed, the connection relationship information is obtained according to all the sub-connection relationship information.

5. The method for stitching sequential images according to claim 4, It is characterized in that The step of performing local splicing processing on the target graph and the reference graph to obtain sub-connection relationship information includes: Determine whether the Boolean variable corresponding to the splicing direction overlap rate is true according to the Boolean variable information; If the Boolean variable is not true, performing one or more local splicing processes of a feature matching operation, an overall optimization operation, and an offset optimization operation on the target image and the reference image; If the splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained; If the splicing fails, one or more local splicing processes in the template matching operation and the overall optimization operation are performed on the target image and the reference image. If the splicing succeeds, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained; If the Boolean variable is true, one or more local splicing processes including a splicing direction mean matching operation and an offset optimization operation are performed on the target image and the reference image, and if the splicing is successful, sub-connection relationship information is obtained; If the reference image and the target image have been downsampled, an original resolution optimization operation is performed on the reference image and the target image, and an overlap rate update operation is performed on the Boolean variable information.

6. The method for stitching sequential images according to claim 1, It is characterized in that The connection relationship information includes a plurality of sub-connection relationship information, and the sub-connection relationship information includes a five-tuple data [iRef, iObj, Succ, Offset, Conf], wherein iRef represents the sequence number of the reference image in the connection relationship, iObj represents the sequence number of the target image in the reorganized connection relationship, Succ is a Boolean variable, Offset is the offset of the target image relative to the reference image, and Conf represents the credibility of the offset; performing coordinate allocation processing on the image sequence data to be spliced ​​according to the connection relationship information to obtain the coordinate allocation data, comprising: Traversing the connection relationship information to obtain the sub-connection relationship information corresponding to the maximum credibility; Adding the image data corresponding to the sub-connection relationship information to a list of stitched fields of view; Centering on the stitched visual field list, matching is performed sequentially according to the connection relationship information and the image sequence data to be stitched to obtain coordinate allocation data.

7. The method for stitching sequential images according to claim 6, It is characterized in that After the step of splicing the image data of multiple rows and columns in the image sequence data according to the coordinate allocation data to obtain the spliced ​​image data, the method further comprises: Performing definition calculation on the overlapping area of ​​the image data according to a preset difference algorithm or a preset gradient algorithm to obtain definition data; The stitched image data is subjected to image fusion processing according to the definition data to obtain stitched image data after image fusion.

8. A processing system for stitching sequential images. It is characterized in that include: A sequence image unit, used for acquiring image sequence data to be spliced, wherein the image sequence data includes image data of multiple rows and columns; An initialization unit, used for performing initialization processing according to the image sequence data to be spliced ​​to obtain Boolean variable information; A local splicing unit, used for traversing the image sequence data to be spliced ​​according to a preset order, and performing local splicing processing on two adjacent image data traversed according to the Boolean variable information to obtain connection relationship information; A coordinate allocation unit, used for performing coordinate allocation processing on the image sequence data to be spliced ​​according to the connection relationship information to obtain coordinate allocation data; The image splicing unit is used to splice the image data of multiple rows and columns in the image sequence data according to the coordinate allocation data to obtain spliced ​​image data.

9. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the processing method for stitching sequential images as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the processing method for stitching sequential images as claimed in any one of claims 1 to 7.

Citation Information

Cited By

  • Microscope-based digital slice generation system, method and related equipment

    CN119863792A

  • Local minimum spanning tree path planning method supporting global coordinate updating

    CN120746824A