A heterogeneous image registration method and system

Abstract

The present invention discloses a heterogeneous image registration method and system, including: using the Canny edge detection operator to perform edge detection on the collected image, extracting the contour curve segment in the edge image in combination with the curvature scale space strategy; detecting the feature points in the contour curve segment based on the feature point detection strategy of the global and local curvature, and obtaining the nearest local curvature minimum values ​​of the feature points pointing to the contour starting point and end point respectively; calculating the number of neighborhood sampling points and the neighborhood auxiliary feature points of the neighborhood on both sides of the feature point according to the local curvature minimum value, using the neighborhood auxiliary feature points and the feature points to form a feature triangle and calculating the angle bisector vector and main direction corresponding to the feature point in the feature triangle; the main direction points to the concave side of the feature point contour, and completes the main direction allocation. The method of the present invention has significance and accuracy, and has high adaptability to the image registration scene of power equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of image allocation, and in particular to a heterogeneous image registration method and system. Background Art

[0002] Since automated inspection reduces the demand for human resources in the inspection of high-voltage equipment in power grids and can improve inspection efficiency, automated inspection technology based on inspection robots and drones has become an important part of the intelligent diagnosis system of power grid equipment. Both inspection robots and drones are equipped with infrared and visible light binocular cameras, which are responsible for capturing the operating status of power equipment under working conditions and uploading image data to the database of the maintenance unit for status monitoring and fault diagnosis. These automated inspection equipment mainly work using multi-mode imaging sensors and image processing technology. Therefore, the development of efficient and reliable image processing technology for power equipment is very important for the construction of diagnostic systems, such as image recognition, fusion and allocation technology. Among them, image allocation technology is the pre-processing work of various image technology applications, especially image stitching, fusion and target detection. The purpose of image allocation is to align multiple images of the same scene taken by different sensors, at different times and from different perspectives. Visible light images can capture the appearance information of power equipment, while infrared images can capture the thermal radiation information of power equipment. Therefore, the image allocation and recognition technology of infrared and visible light for power equipment can present various information of power equipment in one image, which greatly facilitates the fault detection of power equipment.

[0003] However, achieving accurate alignment of infrared and visible light images of power equipment is currently a very challenging task, and existing algorithms are difficult to achieve accurate alignment because the resolution, spectrum, and viewing angle of infrared and visible light images are significantly different. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a heterogeneous image registration method and system, which can solve the problem of accurate matching of infrared and visible light images of power equipment.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: including, using the Canny edge detection operator to perform edge detection on the collected image, and extracting the contour curve segment in the edge image in combination with the curvature scale space strategy; detecting the feature points in the contour curve segment based on the feature point detection strategy of global and local curvature, and obtaining the nearest local curvature minimum value of the feature point pointing to the contour starting point and end point respectively; calculating the number of neighborhood sampling points and the neighborhood auxiliary feature points on both sides of the feature point according to the local curvature minimum value; using the neighborhood auxiliary feature points and the feature points to form a feature triangle and calculating the angle bisector vector and the main direction corresponding to the feature point in the feature triangle; the main direction points to the concave side of the feature point contour to complete the main direction allocation.

[0008] As a preferred solution of the heterogeneous image registration method described in the present invention, the feature point detection includes: using the Canny edge detection operator to detect the edge binary image of the image, extracting the contour curve segment in the edge image; calculating the curvature of each point in the contour curve segment, taking the curvature maximum point on the contour as a candidate feature point and storing the nearest local minimum points at both ends of each maximum point; calculating the average curvature value of each candidate feature point in the neighborhood of the contour curve segment; setting a curvature multiple threshold, when the curvature of the candidate feature point is less than the average curvature value and the value of the curvature multiple threshold, the feature point is detected. When the angle between a feature point and two adjacent feature points on the same contour is less than the threshold, the candidate feature point is defined as the feature point, otherwise, the candidate feature point is eliminated; the first point and the last point of each non-closed contour are taken as the feature points, and the feature point extraction is completed.

[0009] As a preferred solution of the heterogeneous image registration method described in the present invention, the local curvature minimum points pointing to the starting point and the end point of the contour closest to the feature point in the contour curve segment are detected respectively; from the first point of each contour to the last point, if the curvature of a certain point is less than the curvature of the left and right points at the same time, it is defined as the local curvature minimum point; the feature point and its corresponding local curvature minimum points pointing to the starting point and the end point of the contour are respectively recorded as and

[0010] As a preferred solution of the heterogeneous image registration method described in the present invention, the calculation of the number of neighborhood sampling points includes:

[0011] λ L =f-kL

[0012] λ R =kR-f

[0013] Where, f∈[2,n], n: the number of feature points included in each contour, λ L and λ R are respectively the number of neighborhood sampling points of the feature point.

[0014] As a preferred solution of the heterogeneous image registration method described in the present invention, the neighborhood auxiliary feature points include Gaussian weighting of the horizontal coordinates and vertical coordinates of all points between the kLth point and the fth point in the jth contour, as follows:

[0015]

[0016]

[0017]

[0018] Among them, G σ : one-dimensional Gaussian kernel, and The neighborhood auxiliary feature points, when f=1 or n, are calculated as follows:

[0019]

[0020] As a preferred solution of the heterogeneous image registration method described in the present invention, the calculation of the angle bisector vector includes:

[0021]

[0022]

[0023]

[0024] in, The feature points are The coordinates of The angle bisector vector.

[0025] As a preferred solution of the heterogeneous image registration method described in the present invention, the calculation of the main direction includes:

[0026]

[0027] in, Feature Points The contour angle direction of the feature point is the main direction of the feature point in the allocation.

[0028] As a preferred solution of the heterogeneous image registration method described in the present invention, wherein: edge detection of the image includes: performing Gaussian filtering on the grayscale image; calculating the gradient matrix and direction matrix in the filtered image; performing non-maximum suppression on the gradient matrix to obtain a non-maximum suppressed image; and using a dual threshold strategy to detect and connect image edge contours.

[0029] As a preferred solution of the heterogeneous image registration method described in the present invention, wherein: extracting the contour curve segment includes, starting from the first row and the first column of the edge image, traversing the search in units of rows; if a certain point is found to be an edge point, it is marked as the first contour point and is set to 0; finding whether there is the edge point in the neighborhood of the contour point with a side length of 1, if so, adding it to the contour neighborhood as the second contour point and setting it to 0; repeating the search until there is no edge point in the neighborhood of a certain point, and defining the stopping point as the last point of the contour; each contour is a collection of the edge points, and all the acquired contours are integrated to form a contour collection, as follows,

[0030]

[0031] Among them, Γ j : the jth contour curve segment in the set S, n: contour Γ j The number of pixels contained, N s : The total number of contour curve segments in the set S.

[0032] As a preferred solution of the heterogeneous image registration system described in the present invention, it includes: an information acquisition module for collecting the image data information and obtaining feature information; a data processing center module for receiving, calculating, storing and outputting the data information to be processed, which includes a calculation unit, a database and an input-output management unit. The calculation unit is connected to the acquisition module and is used to receive the data information obtained by the information acquisition module for calculation processing, calculate the local curvature minimum, the number of neighborhood sampling points, the neighborhood auxiliary feature points and the angle bisector vector and the main direction. The database is connected to each module and is used to store all received data information and provide allocation and supply services for the data processing center module. The input-output management unit is used to receive information from each module and output the calculation result of the calculation unit; the allocation module is connected to the data processing center module and is used to read the calculation result of the calculation unit, control the main direction to point to the concave side of the feature point contour, complete the main direction allocation, and achieve accurate matching.

[0033] Beneficial effects of the present invention: The method of the present invention calculates the main direction through the contour features between images. On the one hand, since the resolution, spectrum and viewing angle of infrared and visible light images are quite different and the registration problem when these three scenes coexist cannot be achieved, each feature point is assigned a unique direction parameter to perform accurate description operations, so that the extracted feature points are significant and accurate when matched; on the other hand, since the infrared and visible light images of power equipment have extremely significant contour characteristics, the method of the present invention has high adaptability to the image registration scenario of power equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0035] Figure 1 A schematic diagram of the process of a heterogeneous image registration method according to the first embodiment of the present invention;

[0036] Figure 2 A schematic diagram of an image contour set extraction process of a heterogeneous image registration method according to the first embodiment of the present invention;

[0037] Figure 3 It is a schematic diagram of calculating the contour angle main direction CAO of a heterogeneous image registration method described in the first embodiment of the present invention;

[0038] Figure 4 An infrared schematic diagram of an electric power device according to a heterogeneous image registration method described in the first embodiment of the present invention;

[0039] Figure 5 A visible light schematic diagram of an electric power device according to a heterogeneous image registration method described in the first embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of quantitative comparison results of main direction experiments of a heterogeneous image registration method according to the first embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of image comparison results when the rotation angle of a heterogeneous image registration method according to the first embodiment of the present invention is 0;

[0042] Figure 8 This is a schematic diagram of the module structure distribution of a heterogeneous image registration system described in the second embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0045] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0046] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0047] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0048] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] Example 1

[0050] The point feature-based allocation method is highly adaptable to image perspective transformation and image registration scenarios taken at different times, and is therefore widely used and studied. However, the point feature-based method still has obvious problems in registering infrared and visible light images. For example, the resolution, spectrum, and perspective of infrared and visible light images are quite different. The existing point feature-based method cannot achieve registration when these three scenes coexist. The main problem is that the description of feature points is not accurate enough.

[0051] Reference Figure 1 to Figure 7 , which is the first embodiment of the present invention, provides a heterogeneous image registration method, comprising:

[0052] S1: Use the Canny edge detection operator to perform edge detection on the collected image, and combine the curvature scale space strategy to extract the contour curve segments in the edge image. It should be noted that edge detection on the image includes:

[0053] Perform Gaussian filtering on the grayscale image;

[0054] Calculate the gradient matrix and direction matrix in the filtered image, and use Sx and Sy to convolve with the image respectively to obtain two identical image matrices G x and G y , and calculate the square sum of the elements at the same position of the matrix (i.e., gradient) and the positive and negative tangent function values ​​(direction), and obtain the gradient matrix and direction matrix respectively, as follows:

[0055]

[0056]

[0057] θ=arctan(G y / G x )

[0058] Perform non-maximum suppression on the gradient matrix, find the maximum value in the gradient matrix (if the gradient of a point is greater than the gradient of all points in the square neighborhood with a side length of 3, it is a maximum value), and obtain a non-maximum suppression image;

[0059] A double threshold strategy is used to detect and connect image edge contours. Two thresholds th1 and th2 are set for the non-maximum suppressed image, and the relationship between the two is th1=0.4th2. The grayscale values ​​of pixels with gradient values ​​less than th1 are set to 0 to obtain Image 1, and the grayscale values ​​of pixels with gradient values ​​less than th2 are set to 0 to obtain Image 2 (due to the higher threshold of Image 2, most of the noise is removed, but at the same time, useful edge information is also lost, while the threshold of Image 1 is lower, retaining more information, and Image 2 is used as the basis and Image 1 is used as a supplement to connect the edges of the images).

[0060] Specifically, they include:

[0061] Scan image 2. When encountering a non-zero grayscale pixel p(x,y), trace the contour line starting from p(x,y) until the end point of the contour line is q(x,y);

[0062] Observe the neighborhood of point s(x,y) in image 1 that corresponds to the position of point q(x,y) in image 2;

[0063] If there is a non-zero pixel s(x,y) in the vicinity of point s(x,y), it will be merged into image 2 as point r(x,y);

[0064] Repeat the above steps until it is no longer possible to scan in Image 1 and Image 2;

[0065] After completing the connection of the contour line containing p(x,y), mark this contour line as visited;

[0066] The search for the next contour line is repeated in sequence until no new contour line is found, and the search is stopped, completing the edge detection of the Canny operator.

[0067] Further, see Figure 2 , extracting contour curve segments includes:

[0068] Start from the first row and first column of the edge image and loop through the search in units of rows;

[0069] If a point is found to be an edge point, it is marked as the first contour point and set to 0;

[0070] Find out whether there is an edge point in the neighborhood of the contour point with a side length of 1. If so, add it to the contour neighborhood as the second contour point and set it to 0;

[0071] Repeat the search until there is no edge point in the neighborhood of a certain point, and define the stopping point as the last point of the contour;

[0072] Each contour is a set of edge points. All the obtained contours are integrated to form a contour set, as follows:

[0073]

[0074] Among them, Γ j : the jth contour curve segment in the set S, n: contour Γ j The number of pixels contained, N s : The total number of contour curve segments in the set S.

[0075] S2: Detect feature points in the contour curve segment based on the feature point detection strategy of global and local curvature, and obtain the nearest local curvature minimum points pointing to the contour starting point and end point. It should be noted that the detection of feature points in this step includes:

[0076] Use the Canny edge detection operator to detect the edge binary image of the image and extract the contour curve segments in the edge image;

[0077] Calculate the curvature of each point in the contour curve segment, take the curvature maximum point on the contour as the candidate feature point and store the local minimum points at both ends of each maximum point;

[0078] Calculate the average curvature value Kmean of each candidate feature point in the neighborhood of the contour curve segment;

[0079] Set the curvature multiple threshold C. When the curvature of a candidate feature point is less than the product of the mean curvature value Kmean and the threshold C, the candidate feature point is eliminated.

[0080] When the curvature of a candidate feature point is greater than the product of the mean curvature value Kmean and the threshold C, calculate whether the candidate feature point is on the circular contour. If so, remove it; if not, retain it.

[0081] The positional relationship of the retained candidate feature points is calculated. If the angle between a feature point and its two adjacent feature points on the same contour is less than the threshold θ, the candidate feature point is defined as a feature point. Otherwise, the candidate feature point is eliminated.

[0082] The first and last points of each non-closed contour are taken as feature points, and the feature point extraction is completed.

[0083] Further:

[0084] Detect the feature points in the contour curve segment respectively The nearest local curvature minima pointing to the start and end points of the contour and

[0085] Traversing from the first point of each contour to the last point, if the curvature of a point is less than the curvatures of the two left and right points at the same time, it is defined as the local curvature minimum point.

[0086] S3: Calculate the number of neighborhood sampling points and the neighborhood auxiliary feature points on both sides of the feature point according to the local curvature minimum. It should also be noted that calculating the number of neighborhood sampling points includes:

[0087] λ L =f-kL

[0088] λ R =kR-f

[0089] Where, f∈[2,n], n: the number of feature points included in each contour, λ L and λ R are the number of neighborhood sampling points of the feature points.

[0090] Specifically, the neighborhood auxiliary feature points include:

[0091] Gaussian weighting is performed on the horizontal and vertical coordinates of all points between the kLth point and the fth point in the jth contour, as follows:

[0092]

[0093]

[0094]

[0095] Among them, G σ : one-dimensional Gaussian kernel, and Neighborhood auxiliary feature points, when f = 1 or n, are calculated as follows:

[0096]

[0097]

[0098] S4: Use the neighborhood auxiliary feature points and feature points to form a feature triangle and calculate the angle bisector vector and main direction corresponding to the feature points in the feature triangle. Figure 3 , calculating the angle bisector vector includes:

[0099]

[0100]

[0101]

[0102] in, The feature points The coordinates of Angle bisector vector.

[0103] Furthermore, the calculation of the main direction includes:

[0104]

[0105] in, Feature Points The contour angle direction of the feature point is the main direction of the feature point in the allocation.

[0106] Preferably, it should be noted that:

[0107] Each feature point and its neighboring auxiliary feature points form a feature triangle;

[0108] The feature triangles corresponding to the same feature point in the infrared and visible light images are similar triangles. At this time, the angle corresponding to the angle bisector vector is selected as the main direction to ensure that the main directions of the same feature point in different images are the same;

[0109] When the main directions are the same, the corresponding image registration strategy can successfully register images with different perspectives.

[0110] S5: The main direction points to the concave side of the feature point contour, completing the main direction allocation.

[0111] It should also be noted that this embodiment uses three existing registration methods as examples to better understand the method of the present invention. First, a method based on infrared and visible light camera calibration parameters requires known camera prior parameters and can only align a group of images taken by the camera at the same time, and cannot align images taken at different times and different positions. Once calibrated, the registration error at different distances is constant. Therefore, this method is only suitable for limited scenes and cannot achieve image registration at different times and different positions. Second, a region-based infrared and visible light image registration method mainly uses some similarity measurement algorithms to determine the similarity of certain regions in infrared and visible light images to find the global optimal transformation parameters. This method is only suitable for situations where there is no obvious difference in viewing angle between infrared and visible light images and the overlapping area between images is relatively large, and its application scenarios are still limited. Third, a feature-based registration method mainly processes some point, line and surface features in the image to align the image. However, the resolution, spectrum and viewing angle of infrared and visible light images are quite different. The existing point feature-based method cannot achieve registration when these three scenes coexist, and the description of feature points is not accurate enough.

[0112] Generally speaking, the main direction assignment of feature points is one of the most important steps in describing feature points. Its purpose is to assign unique direction parameters to each feature point and highlight the image features of the point so that the extracted feature descriptors are significant. The existing main direction assignment SIFT, PIIFD, and SURF algorithms all perform main direction assignment based on the gradient features of the image. However, the spectra of infrared and visible light images are very different, and the extracted gradient features are not as highly correlated as those between homologous images. The method of the present invention mainly relies on the contour features between images to calculate the main direction. Since both infrared and visible light images of power equipment have extremely significant contour features, the method of the present invention has high adaptability to image registration scenarios of power equipment.

[0113] Preferably, refer to Figure 4 and Figure 5 In order to verify and illustrate the technical effect of the method of the present invention, this embodiment selects the traditional SIFT method, PIIFD method, SURF method and the method of the present invention for comparative testing, and compares the test results by means of scientific demonstration to verify the real effect of the method of the present invention; the traditional SIFT method, PIIFD method and SURF method are not accurate in main direction allocation and have weak adaptability. In order to verify the higher accuracy of the method of the present invention compared with the traditional method, this embodiment will use the traditional method and the method of the present invention to respectively perform allocation comparison tests on one infrared image and six visible light images of the same power equipment;

[0114] Test conditions: (1) One infrared image (resolution 120×280) and six visible light images of the same power equipment were used as comparison images;

[0115] (2) The rotation angles of the six visible light images relative to the infrared image are 0° (resolution 120×280), 60° (resolution 252×203), 120° (resolution 252×203), 180° (resolution 120×280), 240° (resolution 252×203), and 300° (resolution 252×203);

[0116] (3) Turn on the automated test equipment and use MATLB simulation to run the output result curve.

[0117] Reference Figure 6 , which is a comparison curve diagram of the main direction allocation test results of the traditional method and the method of the present invention (i.e., CAO). In the rotation angle from 0° to 300°, the curve output by the method of the present invention is much higher than the number of feature points allocated by the three traditional methods; Figure 7 , which is a schematic diagram of image comparison results when the rotation angle is 0°. It can be intuitively seen that the method of the present invention shows the most correct points at different rotation angles, and the number of correct points obtained by the method of the present invention is three times that of the traditional method. According to the image results, the main directions of the feature points of the method of the present invention all point to the concave side of the contour, verifying that the method of the present invention has high accuracy and adaptability.

[0118] Example 2

[0119] Reference Figure 8 , which is a second embodiment of the present invention. This embodiment is different from the first embodiment in that it provides a heterogeneous image registration system, including:

[0120] The information acquisition module 100 is used to collect image data information and obtain feature information; data acquisition is a bridge connecting the computer and the external physical world, which is composed of sensors and controllers.

[0121] The data processing center module 200 is used to receive, calculate, store and output data information to be processed, which includes a calculation unit 201, a database 202 and an input-output management unit 203. The calculation unit 201 is connected to the acquisition module 100, and is used to receive the data information obtained by the information acquisition module 100 for calculation and processing, and calculate the local curvature minimum, the number of neighborhood sampling points, the neighborhood auxiliary feature points and the angle bisector vector and the main direction. The database 202 is connected to each module, and is used to store all received data information, and provide allocation and supply services for the data processing center module 200. The input-output management unit 203 is used to receive information from each module and output the calculation results of the calculation unit 201.

[0122] The allocation module 300 is connected to the data processing center module 200, and is used to read the calculation result of the calculation unit 201, control the main direction to point to the concave side of the feature point contour, complete the main direction allocation, and achieve accurate matching.

[0123] Preferably, the information acquisition module 100 is a communication module based on a remote data acquisition platform, which integrates a communication chip and a storage chip on a circuit board, so that it has the functions of sending and receiving information, communication, and data transmission through the remote data acquisition platform, and the computer, single-chip microcomputer, and ARM are connected to the remote data acquisition platform through the RS232 serial port, and the information acquisition module 100 is controlled by AI instructions to realize the data communication function.

[0124] Preferably, it should also be noted that the data processing center module 200 is mainly divided into three levels, including a control layer, an operation layer and a storage layer. The control layer is the command and control center of the data processing center module 200, and is composed of an instruction register IR, an instruction decoder ID and an operation controller OC. The control layer can take out each instruction from the memory in sequence according to the program pre-compiled by the user, put it in the instruction register IR, analyze and determine it through the instruction decoder, notify the operation controller OC to operate, and send micro-operation control signals to the corresponding components according to the determined timing; the operation layer is the core of the data processing center module 200, and can perform arithmetic operations (such as addition, subtraction, multiplication, division and additional operations) and logical operations (such as shifting, logical testing or comparison of two values). It is connected to the control layer and performs operations by receiving control signals from the control layer; the storage layer is the database of the data processing center module 200, and can store data (data to be processed and data that has been processed).

[0125] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0126] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0127] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques described in the present invention, the present invention also includes the computer itself. The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on a display.

[0128] As used in this application, the terms "component", "module", "system", etc. are intended to refer to a computer-related entity, which can be hardware, firmware, a combination of hardware and software, software, or software in operation. For example, a component can be, but is not limited to: a process running on a processor, a processor, an object, an executable file, a thread in execution, a program, and / or a computer. As an example, an application running on a computing device and the computing device can both be components. One or more components can exist in a process and / or thread in execution, and a component can be located in a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures thereon. These components can communicate in a local and / or remote process manner, such as according to a signal having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or interacts with other systems in a signal manner through a network such as the Internet).

[0129] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A heterogeneous image registration method, characterized in that: include, The Canny edge detection operator is used to detect the edge of the collected image, and the contour curve segments in the edge image are extracted by combining the curvature scale space strategy; The feature point detection strategy based on global and local curvature detects the feature points in the contour curve segment, and obtains the nearest local curvature minimum values ​​of the feature points pointing to the starting point and the end point of the contour respectively; Calculate the number of neighborhood sampling points and the neighborhood auxiliary feature points on both sides of the feature point according to the local curvature minimum value; Using the neighborhood auxiliary feature points and the feature points to form a feature triangle and calculating the angle bisector vector and the main direction corresponding to the feature points in the feature triangle; The main direction points to the concave side of the feature point contour, completing the main direction allocation; Detecting the feature points includes: Detecting an edge binary image of an image using the Canny edge detection operator, and extracting the contour curve segment in the edge image; Calculate the curvature of each point in the contour curve segment, take the curvature maximum point on the contour as a candidate feature point and store the nearest local minimum points at both ends of each maximum point; Calculate the average curvature value of each candidate feature point in the neighborhood of the contour curve segment; Setting a curvature multiple threshold, when the curvature of the candidate feature point is less than the product of the average curvature value and the threshold, the candidate feature point is eliminated; When the curvature of the candidate feature point is greater than the product of the average curvature value and the threshold, calculate whether the candidate feature point is on the circular contour, if so, remove it, if not, retain it; Calculating the positional relationship of the retained candidate feature points, if the angle between a feature point and two adjacent feature points on the same contour is less than the threshold, the candidate feature point is defined as the feature point, otherwise, the candidate feature point is eliminated; The first point and the last point of each non-closed contour are taken as the feature points, and the feature point extraction is completed; Respectively detecting the local curvature minimum points in the contour curve segment that are closest to the feature point and point to the contour starting point and end point; Traversing from the first point to the last point of each contour, if the curvature of a certain point is smaller than the curvatures of the left and right points at the same time, it is defined as the local curvature minimum point; The feature point and its corresponding local curvature minimum point pointing to the starting point and the end point of the contour are respectively recorded as and Calculating the number of neighborhood sampling points includes, λ L =f-kL λ R =kR-f Where, f∈[2,n], n: the number of feature points included in each contour, λ L and λ R are respectively the number of neighborhood sampling points of the feature point; The neighborhood auxiliary feature points include: Gaussian weighting is performed on the horizontal and vertical coordinates of all points between the kLth point and the fth point in the jth contour, as follows: Among them, G σ : one-dimensional Gaussian kernel, and The neighborhood auxiliary feature points, when f=1 or n, are calculated as follows: Calculating the angle bisector vector includes, in, The feature points are The coordinates of the angle bisector vector; Calculating the principal directions comprises, in, Feature Points The contour angle direction of the feature point is the main direction of the feature point in the allocation.

2. The heterogeneous image registration method according to claim 1, characterized in that: Performing edge detection on the image includes: Perform Gaussian filtering on the grayscale image; Calculating a gradient matrix and a direction matrix in the filtered image; Performing non-maximum suppression on the gradient matrix to obtain a non-maximum suppressed image; Use a double threshold strategy to detect and connect image edge contours.

3. The heterogeneous image registration method according to claim 2, characterized in that: Extracting the contour curve segment comprises: Loop through and search in units of rows starting from the first row and first column of the edge image; If a point is found to be an edge point, it is marked as the first contour point and set to 0; Find whether there is the edge point in the neighborhood of the contour point with a side length of 1. If yes, add it to the contour neighborhood as the second contour point and set it to 0; Repeat the search until there is no edge point in the neighborhood of a certain point, and define the stopping point as the last point of the contour; Each contour is a set of edge points. All the acquired contours are integrated to form a contour set, as follows: Among them, Γ j : the jth contour curve segment in the set S, n: contour Γ j The number of pixels contained, N s : The total number of contour curve segments in the set S.

4. A system for implementing the heterogeneous image registration method according to any one of claims 1 to 3, characterized in that: include, An information collection module (100) is used to collect the image data information and obtain feature information; The data processing center module (200) is used to receive, calculate, store and output data information to be processed, and comprises a calculation unit (201), a database (202) and an input-output management unit (203). The calculation unit (201) is connected to the acquisition module (100) and is used to receive the data information acquired by the information acquisition module (100) for calculation processing, and calculate the local curvature minimum value, the number of neighborhood sampling points, the neighborhood auxiliary feature points and the angle bisector vector and main direction. The database (202) is connected to each module and is used to store all received data information and provide a deployment and supply service for the data processing center module (200). The input-output management unit (203) is used to receive information from each module and output the calculation result of the calculation unit (201); The allocation module (300) is connected to the data processing center module (200) and is used to read the operation result of the operation unit (201), control the main direction to point to the concave side of the feature point contour, complete the main direction allocation, and achieve accurate matching.

Images