Image sequence stitching processing method and system

Abstract

An image sequence stitching processing method, comprising: acquiring sequential image data to be stitched; performing initialization processing on said sequential image data, so as to obtain Boolean variable information; traversing said sequential image data in a preset order, and on the basis of the Boolean variable information, performing local stitching processing on two adjacent pieces of image data obtained from traversal, so as to obtain connection relationship information; on the basis of the connection relationship information, performing coordinate allocation processing on said sequential image data, so as to obtain coordinate allocation data; and on the basis of the coordinate allocation data, stitching image data in multiple rows and multiple columns in said sequential image data, so as to obtain stitched image data.

Description

A processing method and system for sequence image stitching

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the Chinese patent application No. 202411094183.8, filed on August 9, 2024, and entitled "A processing method and system for sequence image stitching", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of image processing, in particular to a processing method and system for sequence image stitching. BACKGROUND

[0004] At present, due to the reasons such as imaging distance, size of image sensor, angle of view and magnification of lens, etc., the single imaging of the imaging device can only obtain a local image (i.e., a single "field of view image") corresponding to the current field of view. To obtain the overall appearance of the photographed scene, different parts of the scene need to be photographed (for example, the image acquisition device is moved at a certain fixed step to take pictures), so as to obtain a series of local images (i.e., multiple "field of view images"), and then find out the transformation relationship between them through a specific image processing algorithm (i.e., "image stitching" algorithm), and then combine them into a single image (i.e., "panoramic image"). At present, panoramic images have been widely used in many fields including entertainment, advertising, transportation, pathology, remote sensing, military, virtual reality, etc.

[0005] In the prior art, there are various types of image acquisition devices, so that the characteristics (color gamut space, dynamic range, resolution, distortion level, etc.) of the collected images are also diversified. In the prior art, different stitching algorithms have been proposed for different types of images. However, no matter which type of stitching algorithm, there are problems such as poor universality and low accuracy.

[0006] SUMMARY

[0007] The purpose of the present application is to provide a processing method and system for sequence image stitching, an electronic device and a computer readable storage medium, which can realize the technical effects of high universality and high accuracy of image stitching.

[0008] In a first aspect, the present application provides a processing method for sequence image stitching, comprising:

[0009] obtaining image sequence data to be stitched, the image sequence data comprising multiple rows and multiple columns of image data;

[0010] performing initialization processing according to the image sequence data to be stitched to obtain Boolean variable information;

[0011] traverse the image sequence data to be spliced according to a preset order, and perform local splicing processing on two adjacent image data traversed according to the Boolean variable information to obtain connection relationship information;

[0012] perform coordinate allocation processing on the image sequence data to be spliced according to the connection relationship information to obtain coordinate allocation data;

[0013] splice multiple rows and multiple columns of image data in the image sequence data according to the coordinate allocation data to obtain spliced image data.

[0014] In the implementation process, the processing method for splicing sequence images provided in the embodiment sets Boolean variable information, which can represent whether the overlap rate range of the image sequence data to be spliced is known. Therefore, for the image sequence data whose overlap rate range is unknown, the overlap rate range can be found through a fast matching strategy combining local and global matching, thereby improving the universality of the algorithm. For the case where the overlap rate range is known, the splicing speed can be improved through a fast mean value matching method. Therefore, the processing method for splicing sequence images can achieve the technical effects of high universality and high accuracy of image splicing.

[0015] Further, the step of obtaining the Boolean variable information according to the initialization processing of the image sequence data to be spliced comprises:

[0016] performing initialization processing on the image sequence data to be spliced to obtain horizontal direction overlap rate, vertical direction overlap rate, and overlap region-desampling information of the adjacent image data;

[0017] generating the Boolean variable information according to the horizontal direction overlap rate, the vertical direction overlap rate, and the overlap region-desampling information.

[0018] In the implementation process, the Boolean variable information represents whether the horizontal direction overlap rate, the vertical direction overlap rate, and the overlap region-desampling information in the image sequence data to be spliced are known. If they are known, the image splicing can be directly performed according to the known horizontal direction overlap rate, the vertical direction overlap rate, and the overlap region-desampling information in the subsequent local splicing processing, and the splicing speed can be improved through a fast mean value matching method. If they are unknown, preset horizontal direction overlap rate, vertical direction overlap rate, and overlap region-desampling information can be set, and the overlap rate range can be found through a fast matching strategy combining local and global matching.

[0019] Further, before the step of traversing the image sequence data to be spliced according to a preset order, performing local splicing processing on adjacent two image data traversed according to the Boolean variable information, and obtaining connection relationship information, the method further comprises:

[0020] traversing the image sequence data to be spliced, and performing the following processing on the image data traversed:

[0021] loading the image data into a preset memory;

[0022] converting the image data into gray image data;

[0023] performing intercept processing on the gray image data according to the Boolean variable information, obtaining a maximum possible overlapping region subgraph corresponding to the image data, and removing the image data from the preset memory, wherein the maximum possible overlapping region subgraph includes one or more of an uppermost possible overlapping region, a lowermost possible overlapping region, a rightmost possible overlapping region, and a leftmost possible overlapping region.

[0024] In the above implementation process, for each image data, the upper adjacent image data, the lower adjacent image data, the right adjacent image data, and the left adjacent image data can all have overlapping regions with the image data; therefore, according to the Boolean variable information, the intercept processing is performed on the gray image data to obtain the maximum possible overlapping region subgraph corresponding to the image data, the maximum possible overlapping region subgraph is retained for local splicing, and the image data is deleted, which can effectively reduce the memory occupation during image splicing processing and further improve the splicing processing speed.

[0025] Further, the step of traversing the image sequence data to be spliced according to a preset order, performing local splicing processing on adjacent two image data traversed according to the Boolean variable information, and obtaining connection relationship information comprises:

[0026] traversing the image sequence data to be spliced according to a preset order, and performing the following processing on the image data traversed:

[0027] determining an image adjacent to the image data as adjacent image data;

[0028] taking the maximum possible overlapping region subgraph corresponding to the image data as a reference graph, and taking the maximum possible overlapping region subgraph of the adjacent image data as a target graph;

[0029] if the size of the target graph is greater than a preset threshold, performing down-sampling on the reference graph and the target graph to obtain down-sampled reference graph and target graph;

[0030] blank detection is performed on the target image and the reference image, if one of the target image and the reference image is detected as blank, then the local splicing processing is stopped;

[0031] blur detection is performed on the target image and the reference image, if one of the target image and the reference image is detected as blur, then the local splicing processing is stopped;

[0032] the target image and the reference image are subjected to local splicing processing to obtain sub-connection relationship information;

[0033] After the traversal is completed, the connection relationship information is obtained according to all the sub-connection relationship information.

[0034] In the above implementation process, when the size of the target image is greater than a preset threshold, the target image and the reference image are subjected to down-sampling processing, so that the resolution of the target image and the reference image is halved, and the processing speed in subsequent image splicing is improved; by introducing fast blank detection and blur detection, the probability of false matching is avoided, and the processing speed is also improved.

[0035] Further, the step of performing local splicing processing on the target image and the reference image to obtain sub-connection relationship information comprises:

[0036] determining whether the Boolean variable corresponding to the splicing direction overlap rate is true according to the Boolean variable information;

[0037] if the Boolean variable is not true, performing one or more local splicing processes of feature matching operation, overall optimization operation, and offset optimization operation on the target image and the reference image;

[0038] if the splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained;

[0039] if the splicing fails, performing one or more local splicing processes of template matching operation and overall optimization operation on the target image and the reference image, if the splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and the sub-connection relationship information is obtained;

[0040] if the Boolean variable is true, performing one or more local splicing processes of splicing direction mean matching operation and offset optimization operation on the target image and the reference image, if the splicing is successful, the sub-connection relationship information is obtained;

[0041] if the reference image and the target image have been subjected to down-sampling, performing original resolution optimization operation on the reference image and the target image, and performing overlap rate updating operation on the Boolean variable information.

[0042] In the implementation process, if the Boolean variable corresponding to the overlap rate of the splicing direction is true, it indicates that the overlap rate of the splicing direction is known, and the matching between the subsequent adjacent field of view images can be arranged in a local range, then the target image and the reference image can be subjected to the average matching operation, offset optimization operation in the splicing direction according to the known overlap rate, so as to improve the matching speed and reduce the memory usage; if the Boolean variable corresponding to the overlap rate of the splicing direction is not true, it indicates that the overlap rate of the splicing direction is unknown, and the overlap rate range between the adjacent images is found in the global range through fast and accurate matching, and the target image and the reference image can be subjected to the feature matching operation, template matching operation, overall optimization operation, offset optimization operation, etc.

[0043] Further, the connection relationship information includes a plurality of sub-connection relationship information, and the sub-connection relationship information includes five-tuple data [iRef, iObj, Succ, Offset, Conf], wherein iRef represents the serial number of the reference image in the sequence in the connection relationship, iObj represents the serial number of the target image in the sequence in the connection relationship, Succ is a Boolean variable, Offset is the offset of the target image relative to the reference image, and Conf represents the confidence of the offset; the step of performing coordinate allocation processing on the image sequence data to be spliced according to the connection relationship information to obtain coordinate allocation data, comprising:

[0044] Iterating the connection relationship information to obtain the sub-connection relationship information corresponding to the maximum confidence;

[0045] Adding the image data corresponding to the sub-connection relationship information to the spliced field of view list;

[0046] Taking the spliced field of view list as the center, sequentially matching according to the connection relationship information and the image sequence data to be spliced to obtain coordinate allocation data.

[0047] In the implementation process, according to the confidence in the connection relationship information, the coordinate allocation of each image data in the image sequence data to be spliced can be performed, which can ensure the accuracy of image splicing; in addition, the center of the spliced field of view list is determined by the sub-connection relationship information corresponding to the maximum confidence, and then the image splicing is gradually completed with this center, which can effectively ensure the processing speed while ensuring the accuracy.

[0048] Further, after the step of splicing the image data in the image sequence data according to the coordinate allocation data to obtain spliced image data, the method further comprises:

[0049] According to the preset difference algorithm or the preset gradient algorithm, the clarity of the overlapping area of the image data is calculated to obtain clarity data;

[0050] According to the definition data, the spliced image data is subjected to image fusion processing, and spliced image data after image fusion is obtained.

[0051] In the implementation process, by introducing the fusion algorithm based on definition, the spliced image data after image fusion is obtained, the proportion of the fuzzy area in the panoramic image can be reduced, and the image quality is improved.

[0052] In a second aspect, the application provides a processing system for sequence image splicing, comprising:

[0053] A sequence image unit is configured to acquire image sequence data to be spliced, wherein the image sequence data comprises image data in multiple rows and multiple columns.

[0054] An initialization unit is configured to perform initialization processing according to the image sequence data to be spliced, and obtain Boolean variable information.

[0055] A local splicing unit is configured to traverse the image sequence data to be spliced according to a preset order, perform local splicing processing on adjacent two image data in the image sequence data to be spliced according to the Boolean variable information, and obtain connection relationship information.

[0056] A coordinate allocation unit is configured to perform coordinate allocation processing on the image sequence data to be spliced according to the connection relationship information, and obtain coordinate allocation data.

[0057] A spliced image unit is configured to perform splicing on the image data in multiple rows and multiple columns in the image sequence data according to the coordinate allocation data, and obtain spliced image data.

[0058] Further, the initialization unit is specifically configured to:

[0059] Perform initialization processing according to the image sequence data to be spliced, and obtain a horizontal direction overlap rate, a vertical direction overlap rate, and overlap region-down sampling information of adjacent image data.

[0060] Generate Boolean variable information according to the horizontal direction overlap rate, the vertical direction overlap rate, and the overlap region-down sampling information.

[0061] Further, the local splicing unit is further configured to:

[0062] Traverse the image sequence data to be spliced, and perform the following processing on the traversed image data:

[0063] Load the image data into a preset memory.

[0064] Convert the image data into grayscale image data.

[0065] According to the Boolean variable information, the grayscale image data is intercepted to obtain a maximum possible overlapping area subgraph corresponding to the image data, and the image data is removed from the preset memory, wherein the maximum possible overlapping area subgraph includes one or more of an upper maximum possible overlapping area, a lower maximum possible overlapping area, a right maximum possible overlapping area, and a left maximum possible overlapping area.

[0066] Further, the local splicing unit is specifically used for:

[0067] According to a preset order, the image sequence data to be spliced is traversed, and the traversed image data is processed as follows:

[0068] The image adjacent to the image data is determined as adjacent image data;

[0069] The maximum possible overlapping area subgraph corresponding to the image data is taken as a reference graph, and the maximum possible overlapping area subgraph of the adjacent image data is taken as a target graph;

[0070] If the size of the target graph is greater than a preset threshold, the reference graph and the target graph are down-sampled to obtain the down-sampled reference graph and the target graph;

[0071] The target graph and the reference graph are subjected to blank detection, and if one of the target graph and the reference graph is detected as blank, the local splicing processing is stopped;

[0072] The target graph and the reference graph are subjected to blur detection, and if one of the target graph and the reference graph is detected as blur, the local splicing processing is stopped;

[0073] The target graph and the reference graph are subjected to local splicing processing to obtain sub-connection relationship information;

[0074] After the traversal is completed, connection relationship information is obtained according to all the sub-connection relationship information.

[0075] Further, the local splicing unit is further used for:

[0076] According to the Boolean variable information, it is judged whether the Boolean variable corresponding to the splicing direction overlap rate is true;

[0077] If the Boolean variable is not true, one or more local splicing processes of feature matching operation, overall optimization operation, and offset optimization operation are performed on the target graph and the reference graph;

[0078] If the splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and sub-connection relationship information is obtained;

[0079] If the splicing fails, one or more local splicing processes of template matching operation, overall optimization operation are performed on the target image and the reference image, if the splicing succeeds, a Boolean variable corresponding to the splicing direction overlap rate is assigned as true, and sub-connection relationship information is obtained;

[0080] If the Boolean variable is true, one or more local splicing processes of splicing direction mean matching operation, offset optimization operation are performed on the target image and the reference image, if the splicing succeeds, sub-connection relationship information is obtained.

[0081] If the reference image and the target image have been down-sampled, a original resolution optimization operation is performed on the reference image and the target image, and an overlap rate updating operation is performed on the Boolean variable information.

[0082] Further, the coordinate assignment unit is specifically configured to:

[0083] Iterate through the connection relationship information to obtain sub-connection relationship information corresponding to the maximum confidence;

[0084] Add the image data corresponding to the sub-connection relationship information to a spliced field of view list;

[0085] Centering on the spliced field of view list, sequentially match according to the connection relationship information and the image sequence data to be spliced to obtain coordinate assignment data.

[0086] Further, the spliced image unit is further configured to:

[0087] According to a preset difference algorithm or a preset gradient algorithm, clarity calculation is performed on the overlapping area of the image data to obtain clarity data;

[0088] According to the clarity data, image fusion processing is performed on the spliced image data to obtain image fused spliced image data.

[0089] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method according to any one of the first aspect when executing the computer program.

[0090] In a fourth aspect, a computer readable storage medium is provided, and instructions are stored on the computer readable storage medium, and when the instructions are executed on a computer, the computer executes the method according to any one of the first aspect.

[0091] In a fifth aspect, a computer program product is provided, and when the computer program product is executed on a computer, the computer executes the method according to any one of the first aspect.

[0092] Other features and advantages of the present application will be set forth in the following description, in part in terms of the descriptions of the application and of the examples, and in part will become apparent to those skilled in the art upon examination of the application or can be learned from practice of the application. The application can take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application.

[0093] In order to make the above objectives, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to for detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0094] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0095] Fig. 1 is a schematic diagram of global search provided by the embodiments of the present application;

[0096] Fig. 2 is a schematic diagram of template matching provided by the embodiments of the present application;

[0097] Fig. 3 is a schematic diagram of feature matching provided by the embodiments of the present application;

[0098] Fig. 4 is a flowchart of a processing method of sequence image stitching provided by the embodiments of the present application;

[0099] Fig. 5 is a flowchart of another processing method of sequence image stitching provided by the embodiments of the present application;

[0100] Fig. 6 is a flowchart of local stitching processing provided by the embodiments of the present application;

[0101] Fig. 7 is a schematic diagram of image sequence data to be stitched provided by the embodiments of the present application;

[0102] Fig. 8 is a schematic diagram of overlap rate calculation image offset and overlap area provided by the embodiments of the present application;

[0103] Fig. 9 is a schematic diagram of fusion algorithm based on clarity provided by the embodiments of the present application;

[0104] Fig. 10 is a flowchart of stitching algorithm provided by the embodiments of the present application;

[0105] Fig. 11 is a structural block diagram of a processing system of sequence image stitching provided by the embodiments of the present application;

[0106] Fig. 12 is a structural block diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0107] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0108] It should be noted that similar reference numerals and letters indicate similar items in the drawings below, and therefore, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", and the like are merely used for distinguishing description, and cannot be understood as indicating or implying relative importance.

[0109] Generally, as a generation means of panoramic images, a stitching algorithm contains three key technologies:

[0110] One is image matching technology:

[0111] which aims to find the optimal transformation between adjacent view images, so that the overlapping area between the images obtained according to the transformation is minimum or most similar under a certain distance;

[0112] T opt = argmax{s(I n , T(I)) | T E Γ};

[0113] where I n is the neighborhood image of image I, T is an image transformation, generally only includes geometric transformation (such as translation transformation, rigid body transformation, affine transformation, projective transformation, etc.), Γ is the space of transformation T, and s(X, Y) is a certain similarity measure (such as Pearson correlation coefficient);

[0114] According to the time sequence of technical development, image matching can be mainly divided into three methods:

[0115] (1) Global search method; this is the earliest method, and the specific operation is as follows: taking each pixel position in the adjacent image as a candidate position, calculating the similarity of the current image and the adjacent image in the overlapping area when the origin (lower left corner) of the current image coincides with the point, and taking the maximum as the optimal transformation. This method can find the global optimum, and is therefore the most accurate, but involves a large number of large-size correlation calculations, and is therefore very time-consuming. In addition, this method is only suitable for the case where the adjacent image only exists in the translation transformation; as shown in FIG. 1, which is a global search diagram provided by the embodiments of the present application, A and B are images to be stitched;

[0116] (2) Template matching method; this is a method developed in the middle, the specific method as follows: in the current image to extract a local small area (template), in the adjacent image in a particular area (search area) template matching, that is, the similarity between the template and each position in the search area of the same size small area is calculated, the maximum is the optimal matching position, and the optimal matching is calculated according to the translation relationship between the two images. This method avoids large size correlation calculation, so the calculation complexity is relatively low, the disadvantage is that because of the local similarity in the image, the obtained matching is only locally optimal but not globally optimal. Similarly, this method is only suitable for the case where adjacent images only exist in the translation transformation; as shown in Figure 2, Figure 2 is a template matching diagram provided by the embodiment of the application, wherein B is the current image, the gray small block in which is a template, A on the left is the adjacent image, the gray area in which is the corresponding search area of the template, and the rectangular area of the dashed line is the most similar area to the template in the search area; let the center position of the template be p, and the center point position of the red box be q, then the translation transformation of image B relative to image A is T(B) = B + (p-q);

[0117] (3) Feature matching method; this is a method developed later, as with the above template matching, is still a local matching method, but here the data used for matching comes from some special local areas, such as corner points, edges, spot areas, etc. The specific approach is: for each image, first find the special area (called interest point, represented by the center point of the area) in it through the algorithm, and then extract the feature vector (called descriptor) of the area, thus forming the feature point set (a feature point = interest point + descriptor) of each image. Thus, the matching problem of images is converted into the matching problem between their feature point sets. Depending on the type of descriptor, such feature matching algorithms have good adaptability to translation, rotation, scale transformation and uneven illumination between adjacent images. This method, because its feature vector is derived from the special secondary processing of the original image, largely avoids the local similarity in template matching, so the matching accuracy is relatively high, but the calculation is relatively complex, and there is a possibility of matching failure due to lack of features in the image, and there is also a possibility of false matching due to local similarity in the image. In such methods, commonly used feature points include Harris corner points, SUSAN corner points, FAST corner points, DOG corner points, KR corner points, and KLT corner points, and common descriptors include HOG, LBP, BRIEF, etc. In addition, there are SIFT features, SURF features, etc. that combine feature point extraction and feature descriptor extraction. As shown in FIG. 3, which is a feature matching diagram provided by an embodiment of the present application, B is the current image, the small blocks in it are feature regions, the left side A is the adjacent image, and the two regions in it are feature regions, and the matching calculation is only performed in the right feature region and the left two feature regions. Let the center position of the right region be p, and the center point position of the left region most similar to the right feature be q, then the translation transformation of image B relative to image A is T(B) = B + (p-q);

[0118] Second, coordinate assignment technology:

[0119] It aims to determine the coordinate position of each field of view image in the panoramic image. After the above image matching of all adjacent images, the relative offset and matching degree (such as correlation size) are obtained, and then according to the spatial (such as row and column) adjacency relationship, a global coordinate system can be established according to certain algorithms such as maximum spanning tree, and the positions of all field of view images in the coordinate system can be determined. At this time, how to get a most reasonable maximum spanning tree is a key factor;

[0120] Third, image fusion technology:

[0121] The purpose of the present application is to eliminate the inconsistency of illumination and color difference between adjacent images in the overlapping area, so as to realize the smooth transition between adjacent images in the panoramic image. The most commonly used method is linear weighting method, which is simple to implement, fast, and can achieve good results in most cases, but when the size of the overlapping area of adjacent images is small or the brightness and color of the overlapping area of adjacent images are quite different, the fusion effect is poor. The best fusion method at present is Poisson fusion, which can be basically applied to any occasion, but its calculation amount is large and difficult to use in high timeliness situations.

[0122] For the existing stitching algorithm, the processing process is as follows: according to the preset overlap rate and fluctuation rate between adjacent images, the minimum possible overlapping area between each pair of adjacent images is determined, and the wavelet transform of the area of one side is performed. The obtained transform result is used to calculate the feature strength of the area on one hand, and on the other hand, the local area where the highest frequency position is located is selected as the "matching block". At the same time, according to the overlap rate and fluctuation rate, the search area of the matching block in the other side of the image pair is determined, and then the template matching of the matching block in the search area is performed. After the two-by-two matching of all the field of view image sequences is completed, the correlation of the template matching and the feature strength of the corresponding minimum overlapping area are used as the basis to obtain the maximum spanning tree of the image sequence, so as to calculate the global coordinates of each field of view image.

[0123] However, in the stitching process of the existing stitching algorithm, only a very small local area is needed for subsequent template matching, so this method has the advantages of fast stitching speed and low memory occupancy, but has the following problems:

[0124] Universality; this method requires the user to input information such as overlap rate and fluctuation rate, and for image sequences whose information cannot be predicted in advance, this method cannot be used directly;

[0125] Accuracy; first, this method only uses local template matching, which may only achieve local optimal matching between images due to local similarity; second, this method uses wavelet (high frequency) coefficients as the description of features and the selection of templates. This strategy can achieve good matching in strong texture and even weak texture areas, but cannot be applied to the following situations: the overlapping area is blank, the overlapping area is blurred, etc.

[0126] The fusion stage does not consider the single-side blur problem; the existing method does not consider the possibility of one side of the overlapping area being blurred when performing fusion, which leads to the degradation of the final image quality.

[0127] To solve the technical problems above, the embodiment of the present application provides a processing method and system for image sequence splicing, an electronic device and a computer readable storage medium, which can be used in the image sequence splicing process, for example, can be used in the image sequence splicing process in which only translation transformation exists between any adjacent images, such as various automatic scanning devices (pathological section scanner, automatic optical detection device, etc.); as shown in FIG. 4, which is a flowchart of the processing method for image sequence splicing provided by the embodiment of the present application, the processing method for image sequence splicing comprises the following steps:

[0128] S100: obtaining image sequence data to be spliced, the image sequence data comprising image data in multiple rows and multiple columns;

[0129] Exemplarily, the image sequence data to be spliced comprises multiple image data, and the multiple image data is arranged in a matrix form.

[0130] S200: performing initialization processing according to the image sequence data to be spliced to obtain Boolean variable information;

[0131] Exemplarily, the Boolean variable information is set to indicate whether the overlap rate of the image sequence data to be spliced in the splicing direction (horizontal direction or vertical direction) is known.

[0132] Optionally, the Boolean variable information can comprise multiple Boolean variables; for example, a Boolean variable ho_known is set to indicate whether the horizontal direction overlap rate is known, and a Boolean variable vo_known is set to indicate whether the vertical direction overlap rate is known; and a Boolean variable downsample is set to indicate whether the maximum possible overlap region is down-sampled.

[0133] S300: traversing the image sequence data to be spliced according to a preset order, performing local splicing processing on two adjacent image data traversed according to the Boolean variable information to obtain connection relationship information;

[0134] Exemplarily, the preset order can be to perform local splicing processing along the horizontal direction, or to perform local splicing processing along the vertical direction, or to perform local splicing processing along both the horizontal direction and the vertical direction simultaneously or alternately.

[0135] S400: performing coordinate allocation processing on the image sequence data to be spliced according to the connection relationship information to obtain coordinate allocation data;

[0136] S500: splicing the image data in multiple rows and multiple columns in the image sequence data according to the coordinate allocation data to obtain spliced image data.

[0137] Exemplarily, the sequence image splicing processing method provided by the embodiment of the present application sets Boolean variable information, and the Boolean variable information can represent whether the overlap rate range of the image sequence data to be spliced is known; thus, for the image sequence data whose overlap rate range is unknown, the overlap rate range can be found out through the fast local and overall combined matching strategy, so as to improve the universality of the algorithm; for the case where the overlap rate range is known, the splicing speed can be improved through the fast mean value matching method; thus, the sequence image splicing processing method can realize the technical effects of high universality and high accuracy of image splicing.

[0138] Please refer to FIG. 5, which is a flowchart of another sequence image splicing processing method provided by the embodiment of the present application;

[0139] Exemplarily, S200: a step of performing initialization processing according to the image sequence data to be spliced to obtain Boolean variable information, comprising:

[0140] S210: performing initialization processing according to the image sequence data to be spliced to obtain the horizontal direction overlap rate, the vertical direction overlap rate and the overlap region-desampling information of the adjacent image data;

[0141] S220: generating the Boolean variable information according to the horizontal direction overlap rate, the vertical direction overlap rate and the overlap region-desampling information.

[0142] Exemplarily, the Boolean variable information represents whether the horizontal direction overlap rate, the vertical direction overlap rate and the overlap region-desampling information in the image sequence data to be spliced are known; if yes, the image splicing can be directly performed according to the known horizontal direction overlap rate, the vertical direction overlap rate and the overlap region-desampling information in the subsequent local splicing processing process, and the splicing speed can be improved through the fast mean value matching method; if not, the preset horizontal direction overlap rate, the vertical direction overlap rate and the overlap region-desampling information can be set, and the overlap rate range can be found out through the fast local and overall combined matching strategy.

[0143] Exemplarily, before S300: a step of traversing the image sequence data to be spliced according to a preset order and performing local splicing processing on the adjacent two image data traversed according to the Boolean variable information to obtain connection relationship information, the method further comprises:

[0144] The image sequence data to be spliced is traversed, and the image data traversed is processed as follows:

[0145] The image data is loaded into a preset memory;

[0146] The image data is converted into gray image data;

[0147] According to the Boolean variable information, the gray image data is intercepted to obtain a maximum possible overlapping area subgraph corresponding to the image data, and the image data is removed from the preset memory, wherein the maximum possible overlapping area subgraph includes one or more of an uppermost possible overlapping area, a lowermost possible overlapping area, a rightmost possible overlapping area, and a leftmost possible overlapping area.

[0148] Exemplarily, for each image data, the upper adjacent image data, the lower adjacent image data, the right adjacent image data, and the left adjacent image data can all have an overlapping area with the image data. Therefore, according to the Boolean variable information, the gray image data is intercepted to obtain a maximum possible overlapping area subgraph corresponding to the image data, the maximum possible overlapping area subgraph is retained for local stitching, and the image data is deleted, which can effectively reduce the memory occupation during image stitching processing and further improve the stitching processing speed.

[0149] Please refer to FIG. 6, which is a flowchart of the local stitching processing provided by the embodiment of the present application.

[0150] Exemplarily, S300: according to a preset order, the image sequence data to be stitched is traversed, and the adjacent two image data traversed is locally stitched according to the Boolean variable information to obtain the connection relationship information, including:

[0151] According to the preset order, the image sequence data to be stitched is traversed, and the image data traversed is processed as follows:

[0152] S311: determining the image adjacent to the image data as the adjacent image data;

[0153] S312: taking the maximum possible overlapping area subgraph corresponding to the image data as a reference graph, and taking the maximum possible overlapping area subgraph of the adjacent image data as a target graph;

[0154] S313: judging whether the size of the target graph is greater than a preset threshold;

[0155] S314: if yes, down-sampling the reference graph and the target graph to obtain the down-sampled reference graph and the target graph;

[0156] S315: performing blank detection on the target graph and the reference graph;

[0157] S316: judging whether one of the target graph and the reference graph is detected as blank, if yes, jumping to S310: stopping the local stitching processing;

[0158] S317: performing blur detection on the target graph and the reference graph;

[0159] S318: judging whether one of the target image and the reference image is detected as being blurred, and if so, jumping to S310: stopping the local splicing processing;

[0160] S319: performing local splicing processing on the target image and the reference image to obtain sub-connection relationship information;

[0161] After the traversal is completed, the connection relationship information is obtained according to all the sub-connection relationship information.

[0162] Exemplarily, when the size of the target image is greater than a preset threshold, the target image and the reference image are subjected to down-sampling processing, so that the resolution of the target image and the reference image is halved, and the processing speed in subsequent image splicing is improved; by introducing fast blank detection and blur detection, the probability of false matching is avoided, and the processing speed is also improved.

[0163] Exemplarily, S319: the step of performing local splicing processing on the target image and the reference image to obtain sub-connection relationship information, comprises:

[0164] judging whether the Boolean variable corresponding to the splicing direction overlap rate is true according to the Boolean variable information;

[0165] if the Boolean variable is not true, performing one or more local splicing processing of feature matching operation, overall optimization operation, and offset optimization operation on the target image and the reference image;

[0166] if the splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned as true, and the sub-connection relationship information is obtained;

[0167] if the splicing fails, performing one or more local splicing processing of template matching operation and overall optimization operation on the target image and the reference image, if the splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned as true, and the sub-connection relationship information is obtained;

[0168] if the Boolean variable is true, performing one or more local splicing processing of splicing direction mean matching operation and offset optimization operation on the target image and the reference image, if the splicing is successful, the sub-connection relationship information is obtained;

[0169] if the reference image and the target image have been subjected to down-sampling, performing original resolution optimization operation on the reference image and the target image, and performing overlap rate updating operation on the Boolean variable information.

[0170] Exemplarily, if the Boolean variable corresponding to the overlap rate of the splicing direction is true, it indicates that the overlap rate of the splicing direction is known, and the matching between the subsequent adjacent field of view images can be arranged in a local range, then the target image and the reference image can be subjected to the average matching operation, offset optimization operation in the splicing direction according to the known overlap rate, so as to improve the matching speed and reduce the memory usage; if the Boolean variable corresponding to the overlap rate of the splicing direction is not true, it indicates that the overlap rate of the splicing direction is unknown, and the overlap rate range between the adjacent images is found in the global range through fast and accurate matching, and the target image and the reference image can be subjected to the feature matching operation, template matching operation, overall optimization operation, offset optimization operation, etc.

[0171] Exemplarily, the connection relationship information includes a plurality of sub-connection relationship information, and the sub-connection relationship information includes five-tuple data [iRef, iObj, Succ, Offset, Conf], wherein iRef represents the serial number of the reference image in the sequence in the connection relationship, iObj represents the serial number of the target image in the sequence in the connection relationship, Succ is a Boolean variable, Offset is the offset of the target image relative to the reference image, and Conf represents the confidence of the offset; S400: The step of performing coordinate allocation processing on the image sequence data to be spliced according to the connection relationship information to obtain coordinate allocation data, comprising:

[0172] S410: Iterating the connection relationship information to obtain the sub-connection relationship information corresponding to the maximum confidence;

[0173] S420: Adding the image data corresponding to the sub-connection relationship information to the spliced field of view list;

[0174] S430: Taking the spliced field of view list as the center, sequentially matching according to the connection relationship information and the image sequence data to be spliced to obtain coordinate allocation data.

[0175] Exemplarily, according to the confidence in the connection relationship information, the coordinate allocation of each image data in the image sequence data to be spliced can be performed, which can ensure the accuracy of image splicing; in addition, the center of the spliced field of view list is determined by the sub-connection relationship information corresponding to the maximum confidence, and then the image splicing is gradually completed with this center as the center, which can effectively ensure the processing speed while ensuring the accuracy.

[0176] Exemplarily, after the step S500: Splicing the image data in the image sequence data according to the coordinate allocation data to obtain spliced image data, the method further comprises:

[0177] S510: Calculating the definition of the overlapping area of the image data according to a preset difference algorithm or a preset gradient algorithm to obtain definition data;

[0178] S520: Perform image fusion processing on the spliced image data according to the definition data to obtain spliced image data after image fusion.

[0179] Exemplarily, by introducing the definition-based fusion algorithm, the proportion of the fuzzy area in the panoramic image can be reduced, and thus the image quality is improved.

[0180] In some implementation scenarios, the specific implementation process of the sequence image splicing processing method provided by the embodiment of the present application is as follows:

[0181] There must be an overlapping area between adjacent field of view images in the image sequence, which is a prerequisite for seamless splicing of the image sequence. In order to obtain the image sequence, the image acquisition device is often moved according to a certain expected overlap rate for acquisition. However, due to the mechanical precision, vibration and other reasons of the device, the actual overlap rate between adjacent images obtained at different positions will not be strictly equal to the expected value, but will fall within a certain interval range centered on the expected value. The core of image splicing is to calculate the actual overlap rate between each adjacent image through image matching, and then calculate the relative positions of each other, and then connect the images together to form a panoramic image.

[0182] Although the matching calculation between two images can be completed without any prior knowledge, because the calculation needs to be performed in the global range of the image at this time, it will inevitably have a slow speed; on the contrary, if the overlap rate between two images is known within a certain range, the matching calculation can be reduced to a local area, thereby greatly reducing the time complexity of matching. Therefore, the technical solution of the present application includes two matching strategies: one is in the initial stage of splicing, that is, in the case where the overlap rate range is unknown, the overlap rate range between adjacent images is found in the global range through fast and accurate matching; two is to arrange the matching between subsequent adjacent field of view images in a local range after obtaining the overlap rate range. In strategy one, in order to retain the speed advantage of local matching and avoid local optimal trap and improve matching accuracy, the present application proposes a matching strategy combining local and global; for strategy two, in order to improve the matching speed and reduce the memory usage, the present application proposes a matching method based on mean value; in addition, in order to avoid matching of blank and fuzzy images, the present application develops special blank detection and blur detection, which are applied before all matching calculations. Finally, considering the blur situation, the present application proposes a new fusion method;

[0183] The technical solution of the present application is composed of a number of defined terms, operation modules and algorithms, which are specifically stated as follows:

[0184] I. Definition of terms:

[0185] 1.1, image sequence to be stitched: image sequence to be stitched refers to image sequence data to be stitched, which is adjacent R rows and C columns of images on a plane, and can be denoted as S = {I r,c | r = 0, 1, … R-1; c = 0, 1, … C-1}, and has the same size. For example, as shown in FIG. 7, which is a schematic diagram of image sequence data to be stitched provided by an embodiment of the present application, it is a 4-row 3-column image sequence to be stitched;

[0186] 1.2, image stitching sequence: refers to the sequence of the field of view image processed by the stitching algorithm, which generally starts from image I 0,0 , and is processed row by row (the stitching direction is the horizontal direction): that is, I 0,0 , I 0,1 , …I 0,C-1 , I 1,0 , I 1,1 , …I 1,C-1 , …I R-1,0 I R-1,1 , …I R-1,C-1 is processed in turn; or is processed column by column (the stitching direction is the vertical direction): that is, I 0,0 , I 1,0 , …I R-1,0 , I 0,1 , I 1,1 , …I R-1,1 , …I 0,C-1 I 1,C-1 , …I R-1,C-1 ;

[0187] 1.3, overlap rate: refers to the overlap rate of one of two images of the same size relative to the other in the horizontal or vertical direction on the plane;

[0188] 1.4, offset: refers to the relative position of one image (origin) relative to the other image (origin) on the plane;

[0189] The offset between two images of the same size can be calculated according to the image size and their overlap rates in two directions: assuming that images A and B have a width and height of W and H, and the overlap rates of image B relative to image A in the horizontal and vertical directions are OvlpX and OvlpY, respectively, then the offset of image B relative to image A is o BA = (OffX, OffY):

[0190] OffX = W x (1-OvlpX) (1);

[0191] OffY = H x (1-OvlpY)

[0192] And the offset of image A relative to image B is o AB= (-OffX, -OffY);

[0193] Obviously, the overlap ratio can also be calculated from the offset according to (1).

[0194] 1.5, Overlap region: refers to the intersection region of two images in a plane. The overlap region can be calculated according to the size of two images and the offset therebetween. Let the width and height of image A be W A and H A , the width and height of image B be W B and H B , and the offset of image B relative to image A be o BA = (OffX, OffY), then the overlap region Ω A in image A is:

[0195] Ω A = [Xs, Xe, Ys, Ye] (2);

[0196] Xs = max (0, OffX)

[0197] Xe = min (W A , OffX + W B )

[0198] Ys = max (0, OffY)

[0199] Ye = min (H A , OffY + H B )

[0200] The overlap region in image B is:

[0201] Ω B = Ω A -t, (3);

[0202] wherein t = (OffX, OffY) is a translation (offset) vector;

[0203] As shown in FIG. 8, FIG. 8 is a schematic diagram of calculating the offset and the overlap region of an image according to the overlap ratio provided in the embodiments of the present application;

[0204] Optionally, W = 600, H = 480, OvlpX = 0.1, OvlpY = 0.8, OffX = 540, OffY = 96; R A = [540, 600, 96, 480], R B = [0, 120, 0, 384];

[0205] 1.6, Overlap default value: refers to the expected (theoretical) overlap ratio of two images acquired in sequence in the horizontal or vertical direction during the acquisition of the image sequence;

[0206] 1.7, Offset default value: refers to the expected (theoretical) offset of two images acquired in sequence in the horizontal or vertical direction during the acquisition of the image sequence. In practical applications, the offset value is often set according to the overlap default value, and the calculation formula is as follows:

[0207] 1.8, Overlap fluctuation value: refers to the maximum offset between the actual overlap ratio and the theoretical overlap ratio (i.e. the overlap default value) of two images acquired in sequence in the horizontal or vertical direction when the image acquisition device moves in the direction according to the default overlap ratio during the acquisition of the image sequence. The maximum offset is affected by factors such as mechanical precision and vibration. The higher the mechanical precision and the smaller the vibration, the smaller the value of the offset, and vice versa;

[0208] 1.9 Offset fluctuation value: refers to the maximum offset between the actual offset and the theoretical offset (i.e. the offset default value) of two images acquired in sequence in the horizontal or vertical direction when the image acquisition device moves in the direction according to the default overlap ratio during the acquisition of the image sequence. The maximum offset is affected by factors such as mechanical precision and vibration. The higher the mechanical precision and the smaller the vibration, the smaller the value of the offset, and vice versa. In practical applications, the offset is often calculated according to the overlap fluctuation value:

[0209] OffFluX = W x OvlpFluX, (4);

[0210] OffFluY = H x OvlpFluY,

[0211] wherein OvlpFluX and OvlpFluY are the horizontal and vertical overlap fluctuation values when moving in a certain direction, and W and H are the image width and height;

[0212] 1.10, Maximum possible overlap area: refers to the maximum possible overlap area between two images acquired in sequence in the horizontal or vertical direction during the acquisition of the image sequence. The area is a rectangular area, and its position and size are determined by the moving direction, image size (width and height), overlap default value, and overlap fluctuation value:

[0213] Let the horizontal direction move, the two adjacent images collected, the right image relative to the left image in the horizontal direction overlap rate default value for hodx, its fluctuation value for hofx, the vertical direction overlap rate default value for hody, fluctuation value for hofy, then the maximum overlap rate between the two in the horizontal and vertical direction is maxOvlpX=hodx+hofx and maxOvlpY=hody+hofy respectively. Let OvlpX=maxOvlpX, OvlpY=maxOvlpY, in turn according to formula (1) (2) (3) can be obtained on the maximum possible overlap area in the horizontal direction movement;

[0214] Let the vertical direction move, the two adjacent images collected, the upper image relative to the lower image in the vertical direction overlap rate default value for vody, fluctuation value for vofy, the horizontal direction overlap rate default value for vodx, fluctuation value for vofx;

[0215] Then the maximum overlap rate between the two in the horizontal and vertical direction is maxOvlpX=vodx+vofx and maxOvlpY=vody+vofy respectively. Let OvlpX=maxOvlpX, OvlpY=maxOvlpY, in turn according to formula (1) (2) (3) can be obtained on the maximum possible overlap area in the vertical direction movement;

[0216] 1.11, the minimum possible overlap area: refers to the image sequence in the process of collecting, the image acquisition device along the horizontal or vertical direction, the minimum possible overlap area between the two images collected before and after. The area is a rectangular area, its position and size are determined by the moving direction, image size (width and height), overlap rate default value, overlap rate fluctuation value:

[0217] Let the horizontal direction move, the two adjacent images collected, the right image relative to the left image in the horizontal direction overlap rate default value for hodx, its fluctuation value for hofx, the vertical direction overlap rate default value for hody, fluctuation value for hofy, then the minimum overlap rate between the two in the horizontal and vertical direction is minOvlpX=hodx-hofx and minOvlpY=hody-hofy respectively. Let OvlpX=minOvlpX, OvlpY=minOvlpY, in turn according to formula (1) (2) (3) can be obtained on the minimum possible overlap area in the horizontal direction movement;

[0218] When moving vertically, the two adjacent images collected, the upper image relative to the lower image in the vertical direction overlap rate default value for vody, fluctuation value for vofy, the horizontal direction overlap rate default value for vodx, fluctuation value for vofx, then the minimum overlap rate between the two in the horizontal and vertical direction is minOvlpX = vodx-vofx and minOvlpY = vody-vofy. Let OvlpX = minOvlpX, OvlpY = minOvlpY, in turn according to formula (1) (2) (3) can get the minimum possible overlap area when moving vertically;

[0219] 1.12, interest point: a special point of a specified type in an image, which is composed of the following information:

[0220] position, which describes the coordinates of the point in the image;

[0221] intensity, which describes the degree of particularity of the image at the point;

[0222] The commonly used "specified type" includes Harris corner point, SUSAN corner point, FAST corner point, DOG corner point, KLT corner point, edge point, and spot, etc.

[0223] 1.13, feature descriptor: a feature vector of a specified type calculated in a given neighborhood of a point in an image, which is used to describe the brightness / color change of the image in the neighborhood. The commonly used "specified type" includes HOG, LBP, BRIEF, etc.

[0224] 1.14, feature point: a special local region in an image, which is composed of an interest point and a feature descriptor at the point. Therefore, the type of feature point is determined by the type of interest point and the type of descriptor.

[0225] 1.15, feature extraction region: a region in an image for extracting a feature point of a specified type.

[0226] 1.16, template: a sub-region in an image for template matching: B(p, w, h), where p is the lower left corner of the region, w and h are the width and height of the region, respectively.

[0227] 1.17, template extraction region: a region in an image for extracting a template.

[0228] 1.18, reference image: when performing horizontal direction stitching, the reference image refers to the left image, or the rightmost possible overlap region of the left image; when performing vertical direction stitching, the reference image refers to the lower image, or the uppermost possible overlap region of the lower image.

[0229] 1.19, Target image: when performing horizontal stitching, the target image refers to the right image, or the leftmost possible overlapping region of the right image; when performing vertical stitching, the target image refers to the upper image, or the lowermost possible overlapping region of the upper image.

[0230] 1.20, Homonym point pair: refers to the corresponding position of a same point in space in two different images (i.e., the reference image and the target image), which is composed of a three-tuple: (c; p r ,p o ), where p r and p o are the positions in the reference image and the target image, respectively, and c is the similarity of the two images in the overlapping region (calculated as follows) determined by the corresponding position.

[0231] Each homonym point pair can determine an offset: if the corresponding positions of a certain spatial point in two images A and B are (X A ,Y A ) and (X B ,Y B ), respectively, then the offset of image B relative to image A is:

[0232] OffX = X A -X B ;

[0233] OffY = Y A -Y B ;

[0234] Further, the respective overlapping regions can be obtained according to equations (2) and (3).

[0235] 1.21, Template search region: the range that needs to be searched in the corresponding image to find the optimal matching template of a given template. Let the default values of the offsets of the target image relative to the reference image in the horizontal and vertical directions be OffDefX and OffDefY, respectively, and the offset fluctuation values be OffFluX and OffFluY, respectively, then the search region of the template B(p, w b ,h b ) in the target image in the reference image is:

[0236] S(p) = S0 + p (5);

[0237] S0 = {(x, y) ||x - OffDefX| ≤ OffFluX, |y - OffDefY| ≤ OffFluY};

[0238] 1.22, Connection: used to represent the connection position and connection strength between two images in the image sequence, represented by a five-tuple: [iRef, iObj, Succ, Offset, Conf], where iRef represents the serial number of the reference image in the sequence, iObj represents the serial number of the target image in the sequence, Succ is a Boolean variable used to describe whether the offset between the target image and the reference image is found, Offset is the offset of the target image relative to the reference image, and Conf represents the confidence of the offset.

[0239] II. Operation module

[0240] 2.1, Matching conversion operation module

[0241] For an image pair composed of a target image I o and a reference image I r , if the maximum possible overlapping regions between them are Ω o max and Ω r max , respectively, the matching problem between the target image I o and the reference image I r can be converted into a matching problem between sub-images located in Ω o max and Ω r max :

[0242] (OffX, OffY) = (Xs, Ys) r -(Xs, Ys) o +(OffX, OffY) max (6);

[0243] Where (OffX, OffY) is the offset of the target image I o relative to the reference image I r , (Xs, Ys) r is the left lower corner coordinate of Ω r max in I r , (Xs, Ys) o is the left lower corner coordinate of Ω o max in I o , and (OffX, OffY) max is the offset of Ω o max relative to Ω r max .

[0244] That is, for two images to be matched, their maximum possible overlapping sub-images are first extracted, then matching is performed between the two sub-images, and after the offset between the sub-images is obtained, the offset between the original images is calculated using equation (6);

[0245] 2.2,

Blank detection operation module

[0246] For a given gray image I, the following procedures are performed to obtain its foreground mask M and determine whether it is blank:

[0247] Threshold calculation: the gray histogram of the image I is counted, and a gray threshold Tg is calculated accordingly;

[0248] Extract the foreground mask: if the image is a bright field image (meaning the background is bright), its foreground mask is M = {p > Tg | p ∈ I}; if the image is a fluorescent image (meaning the background is dark), its foreground mask is M = {p < Tg | p ∈ I}, where p is the gray value of each pixel in the image. M is a binary matrix of the same size as I, where the element value of 1 represents a foreground pixel and 0 represents a background pixel;

[0249] Blank judgment: if the proportion of the total number of foreground pixels (i.e. the number of elements with value 1 in the mask M) to the total number of pixels exceeds a given threshold Tb, the image is determined to be blank, otherwise it is non-blank. Tb is a pre-set threshold, with a value of 0-0.05;

[0250] 2.3,

Blur detection operation module

[0251] For an image pair consisting of a target image I o and a reference image I r , and their respective foreground masks Mo and Mr, the following procedures are performed to obtain their respective gradient images Go and Gr, and their respective sharpness So and Sr, and to determine whether one of them is blurred:

[0252] i. For each pixel in the target image, use a gradient operator such as the Robert operator, Sobel operator or Laplacian operator, etc. to calculate its gradient, thereby generating the corresponding gradient image Go;

[0253] ii. For each pixel in the reference image, use the same gradient operator to calculate its gradient, thereby generating the corresponding gradient image Gr;

[0254] iii. According to the foreground mask Mo of the target image and the gradient image Go, the gradient mean value of each foreground pixel of the target image is calculated as the sharpness So of the target image;

[0255] iv. According to the foreground mask Mr of the reference image and the gradient image Gr, the gradient mean value of each foreground pixel of the reference image is calculated as the sharpness Sr of the reference image;

[0256] v. If So / Sr > Ts, the reference image is determined as a blurred image; if Sr / So > Ts, the target image is determined as a blurred image; wherein Ts is a pre-set threshold, and is a floating point number greater than or equal to 3.

[0257] 2.4,

Template extraction operation module

[0258] For a given grayscale image I, a corresponding gradient image G and a template extraction region E, the following process is performed to obtain a template sequence {B k}, k = 0, 1, … K-1 of a preset size.

[0259] According to the gradient image G and the template extraction region E, an integral image S corresponding to the template extraction region E is generated.

[0260] According to the integral image S, the K templates with the largest gradient and the largest size are obtained, denoted as {B k}, k = 0, 1, … K-1. The preset size can be 7x7, 9x9, 11x11, 13x13, etc., so that the template is a square with an odd side length.

[0261] 2.5,

Template matching operation module

[0262] For an image pair composed of a target image I o and a reference image I r , the following process is performed to obtain a set of same-name point pairs C that meet the template matching requirements:

[0263] If the minimum possible overlapping region Eo in the target image is a blank region, then:

[0264] If the minimum possible overlapping region Er in the reference image is a blank region, then:

[0265] If the clarity So of the target image is greater than the clarity Sr of the reference image, then:

[0266] i. Taking the full image of the target image as the template extraction region, the

template extraction operation module

[0267] ii. For each template B o k (p o k ,w,h), its optimal matching template B r k (pr k , c k :

[0268] , c r k , c r k , c k ] = argmax {c = s(B o k , c o k , c r r , c r ∈ S(p o k ) | p o k} ;

[0269] where s(X, Y) is the similarity measure between X and Y (Pearson correlation coefficient, hereinafter), S(p k m ) is the corresponding search region of the template in the reference image. If c k ≥ T r k , C o = (c; p k m ) is added to the set of corresponding point pairs C. Where T r is a pre-set template matching threshold (0.5-1.0, hereinafter).

[0270] Otherwise,

[0271] i. Extract the region in the reference image using the whole reference image as the template, and perform the template extraction operation in the reference image to obtain the template sequence {B k r k , k = 0, 1,... K-1.

[0272] ii. For each template B r k r k , find its optimal matching template B o k o k and the corresponding similarity c k :

[0273] , c o k , c​​​​o k , c k ] = argmax{c = s(B r k (p r k , B o (p o , w, h)) | p o ∈ S(p r k );

[0274] where S(p r k ) is the corresponding search region of the template in the target image. If c k ≥ T m , then C k = (c; p r k , p o k ) is added to the set of corresponding point pairs C.

[0275] Otherwise,

[0276] i. The template extraction region is set to the minimum possible overlap with the reference image, and the template extraction operation module is executed to obtain the template sequence {B r k (p r k , w, h), k = 0, 1,..., K - 1.

[0277] ii. For each template B r k (p r k , w, h), its optimal matching template B o k (p o k , w, h) and the corresponding similarity c k are found in the target image:

[0278] [B o k (p o k , w, h), c k ] = argmax{c = s(B r k (p r k , w, h), B o (p o , w, h)) | p o ∈ S(p rk )};

[0279] where S(p r k ) is the corresponding search area of the template in the target image. If c k ≥ T m , then C k = (c; p r k , p o k ) is added to the set of homonymous point pairs C;

[0280] Otherwise,

[0281] If the smallest possible overlapping area Er in the reference image is blank, then:

[0282] i. Extract the region in the target image with the smallest possible overlapping area as a template, and perform the template extraction module in the target image to obtain the template sequence {B o k (p o k , w, h)}, k = 0, 1,... K - 1.

[0283] ii. For each template B o k (p o k , w, h), find its optimal matching template B r k (p r k , w, h) and the corresponding similarity c k in the reference image:

[0284] [B r k (p r k , w, h), c k ] = argmax {c = s(B o k (p o k , w, h), B r (p r , w, h)) | p r ∈ S(p o k}

[0285] where S(p o k ) is the corresponding search area of the template in the reference image. If c k ≥ T m , then C k= (c; p r k , p o k ) is added to the set of homonym point pairs C.

[0286] Otherwise,

[0287] If the minimum possible overlap region saliency So of the target image is greater than the minimum possible overlap region saliency Sr of the reference image, then:

[0288] i. A template extraction region is extracted with the minimum possible overlap region of the target image as the template, and a template extraction operation is performed on the target image to obtain a template sequence {B o k (p o k , w, h), k = 0, 1,... K - 1.

[0289] ii. For each template B o k (p o k , w, h), the optimal matching template B r k (p r k , w, h) and the corresponding similarity c k are found in the reference image.

[0290] [B r k (p r k , w, h), c k ] = argmax {c = s(B o k (p o k , w, h), B r (p r , w, h)) | p r ∈ S(p o k ).

[0291] If c k ≥ T m , then C k = (c; p r k , p o k ) is added to the set of homonym point pairs C.

[0292] Otherwise,

[0293] i. Extracting regions with the smallest possible overlapping area of the reference image as template, performing the

Template extraction module

[0294] ii. For each template B r k (p r k ,w,h), find its optimal matching template B o k (p o k ,w,h) and the corresponding similarity c k :

[0295] [B o k (p o k ,w,h),c k ]=argmax{c=s(B r k (p r k ,w,h),B o (p o ,w,h))|p o ∈S(p r k )};

[0296] If c k ≥T m , then C k =(c;p r k ,p o k ) is added to the set of corresponding point pairs C.

[0297] 2.6,

Feature matching module

[0298] For an image pair consisting of the target image I o and the reference image I r , the following process is performed to obtain the respective feature point sets and and the set of corresponding point pairs C that meet the feature matching requirements:

[0299] i. Extracting feature points in the target image, for example, taking FAST corner points as interest points and BRIEF descriptors as feature vectors, to obtain the feature point set:

[0300] where p o i is the position of the feature point, i.e. the position of the interest point contained in the feature point, and v o i is the descriptor at the feature point, i.e. the feature vector.

[0301] ii. Extract feature points in the reference image, the type of feature points adopted should be at least the same as that in the target image, and the type of descriptor of the feature points should be the same as that in the target image, to obtain a set of feature points:

[0302] where p r j is the position of the feature point, i.e. the position of the interest point contained in the feature point, and v r j is the descriptor at the feature point, i.e. the feature vector.

[0303] iii. Obtain a set of homonym point pairs satisfying the feature matching requirement:

[0304] C = {C mn = (c r m , p o n ) | c = s (v r m , v o n ) T f , 0≤m≤M-1,0≤n≤N-1}

[0305] where s(X,Y) is a similarity measure of variables X and Y, which can be taken as the Pearson correlation coefficient, T f is a preset feature matching threshold, and the value range is 0.5-1.0.

[0306] 2.7,

Horizontal Direction Mean Matching Operation Module

[0307] For two horizontally adjacent field of view images, the maximum possible overlapping region of the left image is denoted as I L , and the maximum possible overlapping region of the right image is denoted as I R . The following process is performed to obtain the optimal offset o R =(OffX,OffY) of I L relative to I b :

[0308] i. Calculate the column mean curve C L = {c R 0, c L 1, … c L L L w-1 ​​} and C R = {c R 0, c R 1,... c R w-1}, where w is the width of I L ; then, a moving correlation calculation is made on C R with C L as a template, and the moving range is 2 times the fluctuation value of the horizontal direction offset. Let the maximum correlation position found in the moving process be k x , and the corresponding correlation be c x . If c x > T c , then the optimal offset of I R relative to I L in the horizontal direction is OffX = k x , and the next step is entered; otherwise, the matching fails, and the module is exited. Where T c is a preset correlation threshold, and the value range is 0.5-1.0, and the same below.

[0309] ii. The row mean curves C L = {r x 0, r R 1,... r x L} and C L = {r L 0, r L 1,... r h-1 R} of the subgraphs I R [k R , w; 0, h] and I R [0, w-k h-1 ; 0, h] are respectively calculated, where h is the height of I L ; then, a moving correlation calculation is made on C R with C L as a template, and the moving range is 2 times the fluctuation value of the vertical direction offset. Let the maximum correlation position found in the moving process be k y , and the corresponding correlation be c y . If c y > T c , then the optimal offset of I R relative to I L in the vertical direction is OffY = k y , and the matching succeeds; otherwise, the matching fails.

[0310] 2.8,

Vertical Direction Mean Matching Operation Module

[0311] For two adjacent visual field images above and below, let the maximum possible overlapping area of the lower image be I BThe most likely overlapping region in the above figure is I. T Perform the following procedure to obtain I T Compared to I B Optimal offset o b =(OffX,OffY):

[0312] i. Calculate I respectively B and I T The row mean curve C B ={r B 0,r B 1,…r B h-1} and C T ={r T 0,r T 1,…r T h-1}, where h is I B The height; then, with C B For templates in C T The above calculations involve movement within a range equal to twice the vertical offset fluctuation value. Let k be the maximum correlation position found during the movement. y The corresponding correlation is c. y If c y >T c , then I T Compared to I B The optimal offset in the vertical direction is OffY = k y Proceed to the next step; otherwise, if the matching fails, exit this module.

[0313] ii. Calculate subgraph I separately B [0,w;k y [h] and I T [0,w;0,hk y The column mean curve C B ={c B 0,c B 1,…c B w-1} and C T ={c T 0,c T 1,…c T w-1}, where w is I B The width; then, with C T For templates in C B The above calculations involve movement within a range equal to twice the horizontal offset fluctuation value. Let k be the maximum correlation position found during the movement. x The corresponding correlation is c. x If c x>T c , then I T Compared to I B The horizontal offset is OffX = k x If the match is successful, then the match is successful; otherwise, the match fails.

[0314] 2.9. [Overall Optimization Operation Module]:

[0315] For target image I o And reference Figure I r Given a set C of pairs of points with the same name, solve the following optimization problem to find the optimal offset o between them. b and similarity c b :

[0316] [o b ,c b ] = argmax{c = f(C mn ,I o ,I r )|C mn ∈C};

[0317] f(C mn ,I o ,I r )=s(I o (Ω o ),I r (Ω r ));

[0318] Ω r =Ω∩(Ω+o),;

[0319] Ω o =(Ω∩(Ω+o))-o;

[0320] Ω = [0, W; 0, H];

[0321] o=p r m -p o n ;

[0322] Where, p r m and p o n The feature points F are respectively r m and F o n The position of Ω is given by Ω, where o is the offset of the target image relative to the reference image, W and H are the width and height of the image, respectively, and Ω is the rectangular region containing the image. r For the overlapping area in the reference figure, Ω oLet s(X,Y) be the overlapping region in the target image, and let s(X,Y) be the similarity measure between variables X and Y, which can be the Pearson correlation coefficient.

[0323] 2.10, [Offset Optimization Operation Module]:

[0324] For target image I o And reference Figure I r The current optimal offset between b The best offset o is obtained within its neighborhood of radius 1 through the following calculation. max and corresponding similarity c max :

[0325] [o max ,c max ] = argmax{c = g(o, I o ,I r )|||oo b ||1≤1};

[0326] g(o,I o ,I r )=s(I o (Ω o ),I r (Ω r ));

[0327] Ω r =Ω∩(Ω+o);

[0328] Ω o =(Ω∩(Ω+o))-o;

[0329] Ω = [0, W; 0, H];

[0330] Where ||●||1 is the 1-norm, o b Ω represents the offset of the target image relative to the reference image, W and H are the width and height of the image, respectively, and Ω is the rectangular region containing the image. r For the overlapping area in the reference figure, Ω o Let s(X,Y) be the overlapping region in the target image, and let s(X,Y) be the similarity measure between variables X and Y, which can be the Pearson correlation coefficient.

[0331] 2.11, [Original Resolution Optimization Operation Module]:

[0332] Let I o and I r These are target images I oc And reference Figure I rc A version with half the resolution, known I o Compared to I r The optimal offset is o b , then Ioc optimal offset o rc of I max and similarity c max can be obtained by the following process:

[0333] [o max ,c max ]=argmax{c=g(o,I oc ,I rc )|||o-2o b ||1≤1};

[0334] g(o,I oc ,I rc )=s(I oc (Ω o ),I rc (Ω r ));

[0335] Ω r =Ω∩(Ω+o),;

[0336] Ω o =(Ω∩(Ω+o))-o;

[0337] Ω=[0,W;0,H];

[0338] where ||●||1 is 1-norm, W and H are the width and height of image I oc , Ω is the rectangular region where the image lies, Ω r is the overlapping region in the reference image I rc , Ω o is the overlapping region in the target image I oc , s(X,Y) is the similarity measure of variables X and Y, which can be taken as Pearson correlation coefficient.

[0339] 2.12,

Overlapping rate updating operation module

[0340] Suppose the optimal offset o o between the target image I r and the reference image I b has been found, then the overlapping rate in the stitching process is updated as follows:

[0341] If I o and I r are horizontally adjacent, then the overlapping rates OffX and OffY are calculated according to (1), and then hox=OffX, hofx=0.05, hody=1.0, hofy=0.05.

[0342] If I o and I rIf the two images are vertically adjacent, then the overlap ratio OffX and OffY are calculated according to (1), and then vofx = 1.0, vofy = 0.05, vody = OffY, vofy = 0.05;

[0343] 2.13,

Image Matching Operation Module One

[0344] For an image pair consisting of a target image I o and a reference image I r , the following procedure is performed,

[0345] i. Perform the

Feature Matching Operation Module

[0346] ii. Perform the

Overall Optimization Operation Module

[0347] iii. Perform the

Offset Optimization Operation Module

[0348] 2.14,

Image Matching Operation Module Two

[0349] For an image pair consisting of a target image I o and a reference image I r , the following procedure is performed,

[0350] i. Perform the

Template Matching Operation Module

[0351] ii. Perform the

Overall Optimization Operation Module

[0352] 2.15,

Image Matching Operation Module Three

[0353] For the left and right adjacent image pair, the following process is performed,

[0354] i. Perform the

Horizontal Direction Mean Value Matching Operation Module

[0355] ii. Perform the

Offset Optimization Operation Module

[0356] 2.16,

Image Matching Operation Module Four

[0357] For the upper and lower adjacent image pair, the following process is performed,

[0358] i. Perform the

Vertical Direction Mean Value Matching Operation Module

[0359] ii. Perform the

Offset Optimization Operation Module

[0360] Three, Algorithm Flow:

[0361] The stitching algorithm of the present application is composed of the following stages:

[0362] 3.1,

Initialization

[0363] A Boolean variable ho_known is set up to represent whether the horizontal direction overlap rate is known, and a Boolean variable vo_known is set up to represent whether the vertical direction overlap rate is known. A Boolean variable downsample is set up to represent whether the maximum possible overlap region is downsampled.

[0364] If the horizontal overlap ratio information of the image sequence is known, the known hodx, hofx, hofy are inputted, and ho_known = true is set; otherwise, hodx = 1.0, hofx = 0.5, hofy = 0.05 are set, and the Boolean variable ho_known = false is set; at this time, the maximum possible overlap range in the horizontal direction is the width of the entire image.

[0365] If the vertical overlap ratio information of the image sequence is known, the known vody, vofy, vofx are inputted, and vo_known = true is set; otherwise, vody = 1.0, vofy = 0.5, vofx = 0.05 are set, and the Boolean variable vo_known = false is set; at this time, the maximum possible overlap range in the vertical direction is the height of the entire image.

[0366] 3.2,

Local stitching

[0367] The purpose of this stage is to calculate the connection relationship between each two adjacent images in the image sequence. For the image I r,c , r = 0, 1, …, R-1, c = 0, 1, …, C-1, the following processing is performed in turn:

[0368] Step 1 image loading. Load the image I r,c into the memory;

[0369] Step 2 graying. Convert the image to a gray image for subsequent calculation;

[0370] Step 3 intercept the maximum possible overlap area subgraph:

[0371] If r > 0, the uppermost possible overlap area of the current image is intercepted, denoted as I r,c [U];

[0372] If r < R-1, the lowermost possible overlap area of the current image is intercepted, denoted as I r,c [D];

[0373] If c > 0, the leftmost possible overlap area of the current image is intercepted, denoted as I r,c [L];

[0374] If c < C-1, the rightmost possible overlap area of the current image is intercepted, denoted as I r,c [R];

[0375] After the subgraph in each direction is intercepted, the current image is removed from the memory to save memory.

[0376] Step 4 bidirectional stitching, that is, the current image is stitched with the horizontal and vertical adjacent images that have been processed:

[0377] Step4.1 If r>0, then perform vertical direction splicing:

[0378] Step4.1.0 Data preparation:

[0379] Take the upper sub-image I of the current image r,c [U] as the "reference image", denoted as I r ; take the lower sub-image I of the upper neighboring image of the current image r-1,c [D] as the "target image", denoted as I o .

[0380] Step4.1.1 Down-sampling:

[0381] If the width and height of the target image are both greater than a given threshold Ts, then backup the target image and the reference image, denoted as I oc and I rc , respectively, and then down-sample the reference image and the target image to reduce their resolution by half, while assigning the down-sampling flag as downsample=true; otherwise, assign the down-sampling flag as downsample=false.

[0382] Step4.1.2 Blank detection:

[0383] Perform blank detection on the target image and the reference image by executing the

Blank Detection Module

[0384] Otherwise, proceed to the following fuzzy detection step.

[0385] Step4.1.3 Fuzzy detection:

[0386] Execute the

Fuzzy Detection Module

[0387] Step4.1.4 Image matching:

[0388] If vo_known==false, then:

[0389] i. Execute the

Image Matching Module One

[0390] ii. Execute the

Image Matching Module Two

[0391] iii. If downsample == true, execute the [Original Resolution Optimization Module], then execute the [Overlap Ratio Update Module];

[0392] If vo_known == true, then:

[0393] a. Execute the [Image Matching Module Four], if successful, go to b,

[0394] b. If downsample == true, execute the [Original Resolution Optimization Module]

[0395] Step 4.1.5 Record the connection relationship:

[0396] If the above matching is successful, then:

[0397] [iRef, iObj, Succ, Offset, Conf] = [r*C + c, (r-1)*C + c, true, o max ,c max ] ;

[0398] [iRef, iObj, Succ, Offset, Conf] = [(r-1)*C + c, r*C + c, true, -o max ,c max ] ;

[0399] Otherwise:

[0400] [iRef, iObj, Succ, Offset, Conf] = [r*C + c, (r-1)*C + c, false, 0, -10] ;

[0401] [iRef, iObj, Succ, Offset, Conf] = [(r-1)*C + c, r*C + c, false, 0, -10] ;

[0402] Where o max is the offset of the target image relative to the reference image found in the above matching step, and c max is the similarity corresponding to the offset.

[0403] Step 4.2 If c > 0, proceed with horizontal direction stitching:

[0404] Step 4.2.0 Data preparation:

[0405] Take the right sub-image I r,c-1 [R] of the left neighbor image of the current image as the "reference image", and record

[0406] I r , and the left sub-image Ir,c [L], called "target image", denoted as I o .

[0407] Step 4.2.1 Down-sampling:

[0408] If the width and height of the target image are both greater than a given threshold Ts, then backup the target image and the reference image, denoted as I oc and I rc , respectively. Then down-sample the reference image and the target image to reduce their resolution by half, and assign the down-sampling flag as downsample = true; otherwise, assign the down-sampling flag as downsample = false.

[0409] Step 4.2.2 Blank detection:

[0410] Perform blank detection on the target image and the reference image by executing the

Blank Detection Operation Module

[0411] Step 4.2.3 Fuzzy detection:

[0412] Perform fuzzy detection on the target image and the reference image by executing the

Fuzzy Detection Operation Module

[0413] otherwise, proceed to the following image matching step.

[0414] Step 4.2.4 Image matching:

[0415] If ho_known == false, then:

[0416] i. Execute the

Image Matching Operation Module One

[0417] ii. Execute the

Image Matching Operation Module Two

[0418] iii. If downsample == true, then execute the

Original Resolution Optimization Module

Overlap Ratio Update Operation Module

[0419] If vo_known == true, then:

[0420] a. Execute the

Image Matching Operation Module Three

[0421] b. If downsample == true, execute the [original resolution optimization operation module];

[0422] Step 4.2.5 record the connection relationship:

[0423] If the above matching is successful, then:

[0424] [iRef, iObj, Succ, Offset, Conf] = [r*C+c-1, r*C+c, true, o max ,c max ];

[0425] [iRef, iObj, Succ, Offset, Conf] = [r*C+c, r*C+c-1, true, -o max ,c max ];

[0426] Otherwise:

[0427] [iRef, iObj, Succ, Offset, Conf] = [r*C+c-1, r*C+c, false, 0, -10];

[0428] [iRef, iObj, Succ, Offset, Conf] = [r*C+c, r*C+c-1, false, 0, -10];

[0429] Where o max is the offset of the target graph relative to the reference graph found in the above matching step, and c max is the similarity corresponding to the offset.

[0430] 3.3, [coordinate assignment]:

[0431] Through the above [local stitching] stage, the connection relationship between any two adjacent images of the entire image sequence (if the matching is successful, expressed by the offset value) is obtained. For an image sequence of R rows and C columns, there are K = R*(C-1) + C*(R-1) connection relationships: {G k}, k = 0, 1, …, K-1. The present application calculates the position of each field of view image in the entire panoramic image {P[n]}, n = 0, 1, …, R*C-1 through the following algorithm:

[0432] Step 0 initialize the stitched field of view list to be empty;

[0433] Step 1 find the one with the largest confidence from all the connection relationships with successful matching G max as the starting point, and add it to the stitched field of view list (at this time, the list only has the serial numbers of two fields of view), and P[Gmax (iRef)] = (0, 0), P[G max (iObj)] = G max (Offset), where: G max (iRef) denotes iRef in the connection relation, i.e. the reference picture sequence number, G max (iObj) denotes iObj in the connection relation, i.e. the target picture sequence number, G max (Offset) denotes Offset in the connection relation, same below;

[0434] Step 2 searches for an optimal connection relation G max from the unused connection relations in the spliced view list, which satisfies: 1) a successful match, i.e. Succ in it is true, 2) connected to the spliced view list, i.e. iRef in it is in the spliced view list, and 3) the maximum confidence, i.e. Conf in it is the maximum.

[0435] Step 3 if G max exists, then adds G max (iObj) to the spliced view list, and the coordinate position is calculated as follows:

[0436] P[G max (iObj)] = P[G max (iRef)] + G max (Offset),

[0437] and then goes to Step 2; otherwise, goes to the next step.

[0438] Step 4 if the length of the spliced view list is not equal to R*C, then:

[0439] Step 4.1 calculates the average values of the horizontal and vertical offsets HOffset and VOffset from the used connection relations.

[0440] Step 4.2 searches for a view iDst adjacent to a spliced view iSrc from the views not yet added to the spliced view list, and then adds it to the spliced view list and calculates its coordinate position.

[0441] If iSrc and iDst are horizontally adjacent, then the coordinate position of iDst is P[iDst] = P[iSrc] + HOffset.

[0442] If iSrc and iDst are vertically adjacent, then the coordinate position of iDst is P[iDst] = P[iSrc] + VOffset.

[0443] Step4.3 If the spliced view list is still not equal to R*C, go to Step4.2, otherwise exit.

[0444] 3.4,

Image Fusion

[0445] As the last stage of image stitching, it aims to eliminate the inconsistency of illumination and color difference between adjacent images in the overlapping area, so as to realize the smooth transition between adjacent images in the panoramic image. Considering the single-side blur problem of the overlapping area (see FIG. 4), the present application proposes the following single-side fusion algorithm based on sharpness:

[0446] Step 1 Let the offset of image B relative to image A be o, then calculate the overlapping area Ω of the two according to formula (2) A and Ω B ;

[0447] Step 2 Calculate the sharpness S A and S B of sub-images A[Ω A ] and B[Ω B ] using the same difference or gradient calculation method;

[0448] Step 3 If S A > S B , then:

[0449] M = A[Ω A1 ] ∪ F(A[Ω A2 ], B[Ω B2 ]);

[0450] Ω A = Ω A1 ∪ Ω A2 ;

[0451] Ω B = Ω B1 ∪ Ω B2 ;

[0452] Otherwise,

[0453] M = B[Ω B1 ] ∪ F(A[Ω A2 ], B[Ω B2 ]);

[0454] Where M is the fusion result of the overlapping area sub-image, F(I1, I2) is a fusion operator for any image I1 and I2, such as the most commonly used linear fusion, ∪ is the union operator. The parameter w is the fusion width, which divides the overlapping area into two sub-regions, and can be taken as 10-30.

[0455] Please refer to FIG. 9 and FIG. 10, FIG. 9 is a schematic diagram of a fusion algorithm based on definition provided by an embodiment of the present application, and FIG. 10 is a flowchart of a stitching algorithm provided by an embodiment of the present application; in combination with the method embodiments shown in FIG. 1 to FIG. 10, the processing method of the sequence image stitching provided by an embodiment of the present application at least includes the following beneficial effects: it can be used in the sequence image stitching occasions where only translation transformation exists between any adjacent images, such as various automatic scanning devices (pathological section scanner, AOI detection device, etc.). By introducing fast blank detection and blur detection, false matching is avoided and the speed is improved; for the image sequence whose overlap rate range is unknown, the overlap rate range is found out through the fast matching strategy combining local and global, thereby improving the universality of the algorithm; for the case where the overlap rate range is known, the present application proposes a fast mean value matching method, which improves the stitching speed; by introducing the fusion algorithm based on definition, the proportion of the fuzzy area in the panoramic image is reduced, thereby improving the image quality.

[0456] Please refer to FIG. 11, which is a structural block diagram of a processing system of sequence image stitching provided by an embodiment of the present application, the processing system of sequence image stitching includes:

[0457] A sequence image unit 100 is configured to acquire image sequence data to be stitched, the image sequence data including image data in multiple rows and multiple columns;

[0458] An initialization unit 200 is configured to perform initialization processing according to the image sequence data to be stitched, and obtain Boolean variable information;

[0459] A local stitching unit 300 is configured to traverse the image sequence data to be stitched according to a preset order, perform local stitching processing on adjacent two image data traversed according to the Boolean variable information, and obtain connection relationship information;

[0460] A coordinate allocation unit 400 is configured to perform coordinate allocation processing on the image sequence data to be stitched according to the connection relationship information, and obtain coordinate allocation data;

[0461] A stitched image unit 500 is configured to perform stitching on the image data in multiple rows and multiple columns in the image sequence data according to the coordinate allocation data, and obtain stitched image data,

[0462] Exemplarily, the initialization unit 200 is specifically configured to:

[0463] perform initialization processing according to the image sequence data to be stitched, and acquire horizontal direction overlap rate, vertical direction overlap rate and overlap region-down sampling information of adjacent image data;

[0464] generate Boolean variable information according to the horizontal direction overlap rate, the vertical direction overlap rate and the overlap region-down sampling information.

[0465] Exemplarily, the local stitching unit 300 is further configured to:

[0466] traverse the image sequence data to be stitched, and perform the following processing on the traversed image data:

[0467] load the image data into a preset memory;

[0468] convert the image data into grayscale image data;

[0469] perform clipping processing on the grayscale image data according to the Boolean variable information to obtain a maximum possible overlap region subgraph corresponding to the image data, and remove the image data from the preset memory, wherein the maximum possible overlap region subgraph includes one or more of an uppermost possible overlap region, a lowermost possible overlap region, a rightmost possible overlap region, and a leftmost possible overlap region.

[0470] Exemplarily, the local stitching unit 300 is specifically configured to:

[0471] traverse the image sequence data to be stitched according to a preset order, and perform the following processing on the traversed image data:

[0472] determine an image adjacent to the image data as adjacent image data;

[0473] take the maximum possible overlap region subgraph corresponding to the image data as a reference graph, and take the maximum possible overlap region subgraph of the adjacent image data as a target graph;

[0474] if the size of the target graph is greater than a preset threshold, down-sample the reference graph and the target graph to obtain a down-sampled reference graph and a down-sampled target graph;

[0475] perform blank detection on the target graph and the reference graph, and if one of the target graph and the reference graph is detected as blank, stop the local stitching processing;

[0476] perform blur detection on the target graph and the reference graph, and if one of the target graph and the reference graph is detected as blur, stop the local stitching processing;

[0477] perform local stitching processing on the target graph and the reference graph to obtain sub-connection relationship information;

[0478] after the traversal is completed, obtain connection relationship information according to all the sub-connection relationship information.

[0479] Exemplarily, the local stitching unit 300 is further configured to:

[0480] determine whether the Boolean variable corresponding to the stitching direction overlap rate is true according to the Boolean variable information;

[0481] If the Boolean variable is true, one or more local stitching processes of feature matching operation, overall optimization operation, offset optimization operation are performed on the target image and the reference image;

[0482] If the stitching is successful, the Boolean variable corresponding to the stitching direction overlap rate is assigned true, and the sub-connection relationship information is obtained.

[0483] If the stitching fails, one or more local stitching processes of template matching operation, overall optimization operation are performed on the target image and the reference image, if the stitching is successful, the Boolean variable corresponding to the stitching direction overlap rate is assigned true, and the sub-connection relationship information is obtained.

[0484] If the Boolean variable is true, one or more local stitching processes of stitching direction mean matching operation, offset optimization operation are performed on the target image and the reference image, if the stitching is successful, the sub-connection relationship information is obtained.

[0485] If the reference image and the target image have been down-sampled, the original resolution optimization operation is performed on the reference image and the target image, and the overlap rate update operation is performed on the Boolean variable information.

[0486] Exemplarily, the coordinate assignment unit 400 is specifically configured to:

[0487] Iterate the connection relationship information to obtain the sub-connection relationship information corresponding to the maximum confidence;

[0488] Add the image data corresponding to the sub-connection relationship information to the stitched field of view list;

[0489] Centering on the stitched field of view list, sequentially match the connection relationship information and the image sequence data to be stitched to obtain the coordinate assignment data.

[0490] Exemplarily, the stitching image unit 500 is further configured to:

[0491] According to the preset difference algorithm or the preset gradient algorithm, the definition of the overlapping area of the image data is calculated to obtain definition data;

[0492] According to the definition data, the image fusion processing is performed on the stitched image data to obtain the image fused stitched image data.

[0493] It should be noted that the processing system for stitching sequence images provided by the embodiments of the present application corresponds to the method embodiments shown in FIGS. 1-10, and to avoid repetition, this will not be repeated here.

[0494] The application further provides an electronic device, as shown in FIG. 12, which is a structural block diagram of an electronic device according to an embodiment of the application. The electronic device can include a processor 510, a communication interface 520, a memory 530, and at least one communication bus 540. The communication bus 540 is used to realize direct connection and communication among the components. The communication interface 520 of the electronic device in the embodiment of the application is used to communicate signaling or data with other node devices. The processor 510 can be an integrated circuit chip with signal processing capability.

[0495] The processor 510 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. It can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the application. The general-purpose processor can be a microprocessor or the processor 510 can also be any conventional processor.

[0496] The memory 530 can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory 530 stores computer readable instructions, which, when executed by the processor 510, enable the electronic device to perform the steps described above in the method embodiments of FIGS. 1 to 10.

[0497] Optionally, the electronic device can further include a storage controller and an input / output unit.

[0498] The memory 530, the storage controller, the processor 510, the peripheral interface, and the input / output unit are directly or indirectly electrically connected to each other to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 540. The processor 510 is configured to execute executable modules stored in the memory 530, such as software function modules or computer programs included in the electronic device.

[0499] The input / output unit is used to provide a user with creating a task and creating a start optional period or a preset execution time for the task to realize the interaction between the user and the server. The input / output unit can be, but is not limited to, a mouse, a keyboard and the like.

[0500] It can be understood that the structure shown in FIG. 12 is only schematic, and the electronic device can further include more or less components than those shown in FIG. 12, or have a different configuration from that shown in FIG. 12. Each component shown in FIG. 12 can be implemented in hardware, software, or a combination thereof.

[0501] The embodiments of the present application further provide a storage medium, which has instructions stored thereon, when the instructions are run on a computer, the computer program is executed by a processor to implement the method of the method embodiments. To avoid repetition, it will not be described here.

[0502] The present application also provides a computer program product, which, when run on a computer, causes the computer to execute the method of the method embodiments.

[0503] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other means. The apparatus embodiments described above are only schematic, for example, the flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operations of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from those noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0504] In addition, each functional module in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0505] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0506] The above merely provides an example of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, and thus, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0507] The above merely provides an example of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, and thus, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0508] It should be noted that, in the present document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

Claims

A processing method of sequence image splicing, characterized in that, The method comprises the following steps: obtaining image sequence data to be spliced, the image sequence data comprising a plurality of rows and a plurality of columns of image data; performing initialization processing on the image sequence data to be spliced to obtain Boolean variable information; traversing the image sequence data to be spliced according to a preset order, and performing local splicing processing on adjacent two pieces of image data traversed according to the Boolean variable information to obtain connection relationship information; performing coordinate allocation processing on the image sequence data to be spliced according to the connection relationship information to obtain coordinate allocation data; splicing a plurality of rows and a plurality of columns of image data in the image sequence data according to the coordinate allocation data to obtain spliced image data. The processing method of sequence image stitching according to claim 1, characterized in that, The step of performing initialization processing on the image sequence data to be spliced to obtain Boolean variable information comprises the following steps: performing initialization processing on the image sequence data to be spliced to obtain horizontal direction overlap rate, vertical direction overlap rate, and overlap region-desampling information of adjacent image data; generating Boolean variable information according to the horizontal direction overlap rate, the vertical direction overlap rate, and the overlap region-desampling information. The processing method of sequence image stitching according to claim 2, characterized in that, Before the step of traversing the image sequence data to be spliced according to a preset order, and performing local splicing processing on adjacent two pieces of image data traversed according to the Boolean variable information to obtain connection relationship information, the method further comprises the following steps: traversing the image sequence data to be spliced, and performing the following processing on the image data traversed: loading the image data into a preset memory; converting the image data into grayscale image data; performing intercepting processing on the grayscale image data according to the Boolean variable information to obtain a maximum possible overlap region subgraph corresponding to the image data, and removing the image data from the preset memory, wherein the maximum possible overlap region subgraph comprises one or more of an uppermost maximum possible overlap region, a lowermost maximum possible overlap region, a rightmost maximum possible overlap region, and a leftmost maximum possible overlap region. The processing method of sequence image stitching according to claim 3, characterized in that, The step of traversing the image sequence data to be spliced according to a preset order, and performing local splicing processing on adjacent two pieces of image data traversed according to the Boolean variable information to obtain connection relationship information comprises the following steps: traversing the image sequence data to be spliced according to a preset order, and performing the following processing on the image data traversed: determining image data adjacent to the image data as adjacent image data; taking the maximum possible overlap region subgraph corresponding to the image data as a reference graph, and taking the maximum possible overlap region subgraph of the adjacent image data as a target graph; if the size of the target graph is greater than a preset threshold, performing desampling on the reference graph and the target graph to obtain desampled reference graph and target graph; performing blank detection on the target graph and the reference graph, and if one of the target graph and the reference graph is detected as blank, stopping the local splicing processing; performing blur detection on the target graph and the reference graph, and if one of the target graph and the reference graph is detected as blur, stopping the local splicing processing; The target graph and the reference graph are subjected to local splicing processing to obtain sub-connection relationship information; After the traversal is completed, connection relationship information is obtained according to all the sub-connection relationship information. The processing method of sequence image stitching according to claim 4, characterized in that, The step of subjecting the target graph and the reference graph to local splicing processing to obtain sub-connection relationship information includes: determining whether the Boolean variable corresponding to the splicing direction overlap rate is true according to the Boolean variable information; if the Boolean variable is not true, performing one or more local splicing processes of feature matching operation, overall optimization operation, and offset optimization operation on the target graph and the reference graph; if splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and sub-connection relationship information is obtained; if splicing fails, performing one or more local splicing processes of template matching operation and overall optimization operation on the target graph and the reference graph, if splicing is successful, the Boolean variable corresponding to the splicing direction overlap rate is assigned a value of true, and sub-connection relationship information is obtained; if the Boolean variable is true, performing one or more local splicing processes of splicing direction mean matching operation and offset optimization operation on the target graph and the reference graph, if splicing is successful, sub-connection relationship information is obtained; if the reference graph and the target graph have been subjected to down-sampling, performing original resolution optimization operation on the reference graph and the target graph, and performing overlap rate updating operation on the Boolean variable information. The processing method of sequence image stitching according to claim 1, characterized in that, The connection relationship information includes a plurality of sub-connection relationship information, and the sub-connection relationship information includes five-tuple data [iRef, iObj, Succ, Offset, Conf], wherein iRef represents the serial number of the reference image in the sequence in the connection relationship, iObj represents the serial number of the target image in the sequence in the connection relationship, Succ is a Boolean variable, Offset is the offset of the target graph relative to the reference graph, and Conf represents the confidence of the offset; the step of performing coordinate allocation processing on the image sequence data to be spliced according to the connection relationship information to obtain coordinate allocation data includes: traversing the connection relationship information to obtain sub-connection relationship information corresponding to the maximum confidence; adding the image data corresponding to the sub-connection relationship information to a spliced field of view list; sequentially matching the connection relationship information and the image sequence data to be spliced based on the spliced field of view list as the center to obtain coordinate allocation data. The processing method of sequence image stitching according to claim 6, characterized in that, After the step of splicing the image data of multiple rows and multiple columns in the image sequence data according to the coordinate allocation data to obtain spliced image data, the method further includes: performing clarity calculation on the overlapping area of the image data according to a preset difference algorithm or a preset gradient algorithm to obtain clarity data; performing image fusion processing on the spliced image data according to the clarity data to obtain image-fused spliced image data. A processing system for sequence image stitching, characterized in that, It includes: a sequence image unit configured to obtain image sequence data to be spliced, the image sequence data including image data of multiple rows and multiple columns; an initialization unit configured to perform initialization processing on the image sequence data to be spliced to obtain Boolean variable information; The local splicing unit is configured to traverse the image sequence data to be spliced according to a preset order, perform local splicing processing on two adjacent image data traversed according to the Boolean variable information, and obtain connection relationship information. The coordinate allocation unit is configured to perform coordinate allocation processing on the image sequence data to be spliced according to the connection relationship information, and obtain coordinate allocation data. The spliced image unit is configured to splice multiple rows and multiple columns of image data in the image sequence data according to the coordinate allocation data, and obtain spliced image data. An electronic device, characterized by comprising: The memory, the processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the processing method for splicing sequence images according to any one of claims 1 to 7 when executing the computer program. The computer readable storage medium stores instructions, and when the instructions are executed on the computer, the computer executes the processing method for splicing sequence images according to any one of claims 1 to 7. A computer-readable storage medium, characterized by, ​

Images