Image change detection method and device based on feature matching

Abstract

The invention discloses an image change detection method and device based on feature matching, and relates to the technical field of pipeline detection, and the method comprises the steps: carrying out the image matching of a historical inspection image set of a previous time period through a plurality of regions of each current image in a current inspection image set of a current time period, and obtaining a matched image set; extracting an overlapping region in the current inspection image set and the corresponding matching image set to determine an aligned image; wherein the alignment image comprises a current alignment image and a preorder alignment image; and performing time sequence change detection on the aligned image by using the trained change detection model to obtain a change detection result of the current image in the current inspection image set. The problems that an existing change detection method is complex and time-consuming, and changes above a pipeline are difficult to detect in time are solved. Events threatening pipeline transportation can be quickly responded and prevented, so that the accident rate can be reduced, and the pipeline management efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of pipeline detection, and in particular, to an image change detection method and device based on feature matching. Background Art

[0002] With the rapid development of the economy, the demand for oil and gas resources is increasing day by day, and the mileage of underground pipelines laid has also increased accordingly. If there are accumulations, heavy vehicles, building construction, or natural disasters such as landslides, collapses, and floods above or around the pipeline, pressure will be exerted on the pipeline, resulting in pipeline deformation, settlement, rupture, and leakage. Therefore, in the process of pipeline transportation, the safety of the area above and around is of crucial importance.

[0003] Performing change detection on the pipeline area can timely detect changes in the pipeline surrounding environment, prevent potential safety hazards and accidents, and ensure the efficiency and safety of pipeline transportation. Change detection is a method for identifying, analyzing, and quantifying changes in the earth's surface or target objects over time. Existing change detection methods mainly generate map tiles from orthophotos to detect and analyze change features of data at different time points in the same area. However, during pipeline inspection with a small time interval, it is difficult and time-consuming to produce map tiles, and it is difficult to timely detect changes above the pipeline. Summary of the Invention

[0004] The embodiments of the present application provide an image change detection method and device based on feature matching, which solve the problems that the existing change detection methods are complex and time-consuming, and it is difficult to timely detect changes above the pipeline.

[0005] In a first aspect, the embodiments of the present application provide an image change detection method based on feature matching, including: performing image matching on a historical inspection image set of a previous time period by using multiple regions of each current image in a current inspection image set of the current time period to obtain a matching image set; extracting overlapping regions in the current inspection image set and its corresponding matching image set to determine aligned images; wherein, the aligned images include a current aligned image and a previous aligned image; performing time series change detection on the aligned images by using a trained change detection model to obtain a change detection result of the current image in the current inspection image set.

[0006] In combination with the first aspect, in a possible implementation, the step of using multiple regions of each current image in the current inspection image set of the current time period to perform image matching on the historical inspection image set of its previous time period to obtain a matching image set includes: performing central region matching on the current images in the current inspection image set with the previous images in the historical inspection image set respectively to obtain the central matching images of the current images; performing direction matching on the current images in the current inspection image set with the previous images in the historical inspection image set respectively to obtain the direction matching images of the current images; and obtaining the matching image set of each current image in the current inspection image set according to the central matching images and the direction matching images.

[0007] In combination with the first aspect, in a possible implementation, the step of performing central region matching on the current images in the current inspection image set with the previous images in the historical inspection image set respectively to obtain the central matching images of the current images includes: determining the central longitude and latitude of the current image in the current inspection image set and the previous image in the historical inspection image set; respectively determining the central distances between the central longitude and latitude of the current image and the central longitude and latitude of at least some of the previous images in the historical inspection image set; determining the minimum value of the central distances of the current image; and if the minimum value of the central distances is less than the central threshold, using the previous image corresponding to the minimum value of the central distances as the central matching image of the current image.

[0008] In combination with the first aspect, in a possible implementation, the step of performing direction matching on the current images in the current inspection image set with the previous images in the historical inspection image set respectively to obtain the direction matching images of the current images includes: determining the longitude and latitude coordinates of the four vertices of the current image / previous image according to the image field of view range and the central longitude and latitude of the current image / previous image; determining the edge line segments of the four sides of the current image / previous image according to the longitude and latitude coordinates of the four vertices of the current image / previous image; respectively determining the edge distances between each edge line segment of the current image and the corresponding edge line segments of at least some of the previous images in the historical inspection image set; respectively determining the minimum values of the multiple edge distances of the current image; and if the minimum value of the edge distances is less than the edge preset value, using the previous image corresponding to the minimum value of the edge distances as the direction matching image of the corresponding edge of the current image.

[0009] Combined with the first aspect, in a possible implementation manner, the extracting the overlapping regions in the current inspection image set and its corresponding matching image set to determine the aligned image includes: segmenting the current image in the current inspection image set and the previous image in its corresponding matching image set to obtain the current segmented image and the previous segmented image; shrinking the current segmented image / previous segmented image, and extracting feature points therein for feature point matching to obtain matching point pairs; wherein, the matching point pairs include current matching points and previous matching points; restoring the current matching points / previous matching points to the current segmented image / previous segmented image, and merging the current matching points / previous matching points in each of the current segmented images / previous segmented images; determining the overlapping region according to the merged current matching points and previous matching points to obtain the aligned image.

[0010] Combined with the first aspect, in a possible implementation manner, before using the trained change detection model to perform time series change detection on the aligned image, it further includes: judging whether the original size of the aligned image is greater than a set template size; if the original size of the aligned image is greater than the template size, then segmenting the aligned image into several segmented images of the template size, and when the width and / or height of the remaining image after segmentation is less than the template size, making up within the range of the aligned image and then segmenting it into the segmented images; if the original size of the aligned image is less than the template size, then padding the aligned image to the template size to obtain the padded aligned image.

[0011] Combined with the first aspect, in a possible implementation manner, the using the trained change detection model to perform time series change detection on the aligned image to obtain the change detection result of the current image in the current inspection image set includes: performing time series change detection on the aligned image of the template size to obtain a binary image; restoring the binary image of the template size to the original size of the aligned image; superimposing the binary image of the original size with the corresponding aligned image to obtain a global grayscale image; performing binarization on the global grayscale image to obtain the change detection result of the current image.

[0012] Combined with the first aspect, in a possible implementation manner, after obtaining the change detection result of the current image in the current inspection image set, it further includes: adding at least part of the change detection results to the sample library of the change detection model for updating the change detection model; inputting the to-be-inspected inspection image into the updated change detection model to obtain the change detection result of the to-be-inspected inspection image.

[0013] In combination with the first aspect, in a possible implementation manner, before adding at least part of the change detection results to the sample library of the change detection model, the following steps are further included: converting the change detection results into labeled samples; screening the labeled samples, removing the misdetected contours therein, supplementing the undetected contours, and adjusting the labeled bounding boxes until they fit the contours in the labeled samples; wherein, converting the change detection results into labeled samples includes: storing the change detection results in a set image format to obtain a change detection image; extracting the contours in the change detection image and writing the extracted contours into a file and storing them as labeled samples.

[0014] In a second aspect, an embodiment of the present application provides an image change detection device based on feature matching, including: a matching module, configured to perform image matching on a historical inspection image set of a previous time period by using multiple regions of each current image in the current inspection image set of the current time period to obtain a matching image set; an extraction module, configured to extract the overlapping regions in the current inspection image set and its corresponding matching image set to determine aligned images; wherein, the aligned images include a current aligned image and a previous aligned image; a detection module, configured to perform time series change detection on the aligned images by using a trained change detection model to obtain change detection results of the current images in the current inspection image set.

[0015] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: Through image matching, the embodiments of the present application can ensure the integrity of the detection area during the detection process and reduce undetected targets; determining the aligned images can ensure the accuracy of the overlapping regions. It effectively solves the problems in the prior art that the change detection method is complex and time-consuming, and it is difficult to detect changes above the pipeline in a timely manner. It avoids the time-consuming and complex production of map tiles, can meet the requirements of pipeline inspection with a small interval, quickly respond to and prevent events threatening pipeline transportation, and thus can reduce the accident rate, improve the pipeline management efficiency, and ensure the safe operation of the pipeline. Description of the Drawings

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for describing the embodiments of the present application or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 It is a flowchart of an image change detection method based on feature matching provided by an embodiment of the present application; Figure 2 It is an example diagram of image matching provided by an embodiment of the present application; Figure 3 Flow chart of the method for obtaining a matching image set provided by an embodiment of the present application; Figure 4 Flow chart of the method for central region matching provided by an embodiment of the present application; Figure 5 Flow chart of the method for direction matching provided by an embodiment of the present application; Figure 6 Flow chart of the method for determining an aligned image provided by an embodiment of the present application; Figure 7 Example diagram of segmenting an aligned image provided by an embodiment of the present application; Figure 8 Example diagram of matching point pairs between a current image and a previous image provided by an embodiment of the present application; Figure 9 Example diagram of a current aligned image provided by an embodiment of the present application; Figure 10 Example diagram of a previous aligned image provided by an embodiment of the present application; Figure 11 Example diagram of an aligned image before anti-color processing provided by an embodiment of the present application; Figure 12 Example diagram of an aligned image after anti-color processing provided by an embodiment of the present application; Figure 13 Example diagram of a black and white binary image provided by an embodiment of the present application; Figure 14 Structural schematic diagram of an image change detection device based on feature matching provided by an embodiment of the present application. Detailed implementation manners

[0018] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0019] The following explains some technologies related to the embodiments of the present application to facilitate understanding. It should be considered that they are merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted in the following description.

[0020] Pipeline area change detection is mainly carried out through periodic inspections by drones. The inspection images of the surrounding areas of the pipeline are collected at different times, and the inspection images of the same pipeline area at different times are compared and detected to extract the areas in the inspection images of the current period (current images) that are different from the historical inspection image set (previous images) of the previous period. During the inspection process, it is recommended that the angle between the drone and the pod camera be greater than 70 degrees, that is, the pod camera should try to keep vertical shooting, and the angle with the vertical direction does not exceed 20 degrees. During the pipeline inspection process, the changing objects in the change detection mainly include targets such as vehicles, earthmoving, water accumulation, accumulation, and buildings.

[0021] Those skilled in the art should be aware that when the drone conducts inspections at each time period (the time required for the drone to fly through the inspection area and complete one inspection), the times of the collected inspection images are different. That is, the current images / previous images in the current inspection image set / historical inspection image set can be stored in order according to their collection times.

[0022] Figure 1 It is a flowchart of an image change detection method based on feature matching provided by an embodiment of the present application, including steps 101 to 103. Among them, Figure 1 It is only an execution order shown in the embodiment of the present application and does not represent the only execution order of an image change detection method based on feature matching. Under the condition that the final result can be achieved, Figure 1 The shown steps can be executed in parallel or reversed.

[0023] Step 101: Use multiple regions of each current image in the current inspection image set of the current time period to perform image matching on the historical inspection image set of the previous time period to obtain a matching image set. In the embodiment of the present application, the current inspection image set is the inspection images of different regions on the pipeline area collected by the drone during the inspection of the pipeline in the current time period, that is, the current images (usually the inspection images are collected at the set positioning points). The historical inspection image set is the inspection images of different regions on the pipeline area collected by the drone during the inspection in the previous time period of the current time period, that is, the previous images. It can be understood that for multiple inspection images in the current inspection image set and the historical inspection image set, the collection positions of each pair are the same or adjacent, and the collection time periods are different. Moreover, when performing image matching, the current image can perform image matching once for each real-time collected inspection image in the current time period, or after collecting multiple inspection images in the current time period, perform image matching by traversing them in turn.

[0024] Furthermore, traverse each current image in the current inspection image set, and use each current image to perform image matching with each (or at least part of) previous image in the historical inspection image set in turn. Image matching includes central region matching and direction matching. Such as Figure 2As shown, where the solid arrows represent the central region matching and the dashed arrows represent the direction matching.

[0025] In the embodiment of the present application, during the drone inspection process, inspection images are collected and saved at a given series of positioning points to obtain a historical inspection image set and a current inspection image set. During the inspection processes at different time periods, due to deviations in the drone positioning information, even when collecting inspection images at the same positioning points, the longitude and latitude information carried by the inspection images is not equal, and the inspection images in the two inspection image sets cannot be completely matched. Moreover, after the inspection images are matched, due to the momentary changes in the pose of the drone and the angle of the pod camera, the matched inspection images only have an overlapping region in the middle area. To ensure the integrity of the detection area during the detection process and reduce missed detection targets, the inspection images in the two inspection image sets are matched one-to-many through the steps in Figure 3 to perform one-to-many matching on the inspection images in the two inspection image sets.

[0026] The specific implementation method of step 101 is as shown in Figure 3 and includes steps 301 to 303, which are as follows.

[0027] Step 301: Perform central region matching on the current images in the current inspection image set with the previous images in the historical inspection image set respectively to obtain the central matching images of the current images. In the embodiment of the present application, determine the central longitude and latitude of the current images in the current inspection image set and the previous images in the historical inspection image set. Determine the central distance between the central longitude and latitude of the current images and the central longitude and latitude of at least some of the previous images in the historical inspection image set respectively. Determine the minimum value of the central distance of the current images. If the minimum value of the central distance is less than the central threshold, then use the previous image corresponding to the minimum value of the central distance as the central matching image of the current image.

[0028] Specifically as shown in Figure 4 , determine the central longitude and latitude of the current images and the previous images respectively. Here, the central longitude and latitude of all current images and all previous images can be obtained in advance, or the central longitude and latitude can be obtained when using a specific current image / previous image.

[0029] Traverse in sequence and calculate the central distance between the central longitude and latitude of each current image and the central longitude and latitude of at least some of the previous images in the historical inspection image set respectively. Determine the minimum value of the central distance of the current images, and compare whether the minimum value of the central distance is less than the central threshold. If the minimum value of the central distance is less than the central threshold, then the previous image corresponding to the minimum value of the central distance is the central matching image of the current image. Otherwise, there is no central matching image for this current image.

[0030] Setting the center threshold too large will result in too little common area between the current image and the corresponding matching image, affecting the subsequent detection accuracy. Exemplarily, the center threshold here is determined by rounding 20% of the distance represented by the short side pixels of the image field of view (the range that the pod camera can capture).

[0031] It should be noted that "at least part" here refers to the first number (which can be set to 10 - 20) of the previous images adjacent to the center matching image of the previous current image adjacent to the current image. Or it refers to the second number (which can be set to 10 - 20) of the previous images in the matching time sequence direction of the center matching image of the previous current image adjacent to the current image. Among them, the matching time sequence direction is the time sequence change direction (i.e., the matching time sequence direction) in which the previous images in the historical inspection image set are sequentially matched according to the matching situation between the previous multiple current images in the current inspection image set and the previous images in the historical inspection image set.

[0032] Step 302: Perform direction matching between the current images in the current inspection image set and the previous images in the historical inspection image set to obtain the direction matching images of the current images. In the embodiments of the present application, according to the image field of view and the central longitude and latitude of the current image / previous image, the longitude and latitude coordinates of the four vertices of the current image / previous image are determined. According to the longitude and latitude coordinates of the four vertices of the current image / previous image, the edge line segments of the four sides of the current image / previous image are determined. The edge distances between each edge line segment of the current image and the corresponding edge line segments of at least part of the previous images in the historical inspection image set are respectively determined. The minimum value of the multiple edge distances of the current image is respectively determined. If the minimum value of the edge distance is less than the edge preset value, the previous image corresponding to the minimum value of the edge distance is used as the direction matching image of the corresponding edge of the current image.

[0033] Specifically, as Figure 5 shown, the longitude and latitude coordinates of the four vertices of the current image / previous image are determined according to the image field of view and the central longitude and latitude of the current image / previous image. Specifically, according to the camera parameters, the image acquisition height, and the pixel width and height of the image, the actual width and height of the image field of view of the current image / previous image are determined, and then the actual width and height of the image field of view are converted into the longitude and latitude change amounts in the longitude-latitude-height coordinate system.

[0034] Furthermore, the longitude and latitude coordinates of the four vertices of the current image / previous image are as follows: , , , .

[0035] In the formula, Indicates the latitude and longitude coordinates of the upper left corner of the current image / preceding image, Indicates the latitude and longitude coordinates of the upper right corner of the current image / preceding image, Indicates the latitude and longitude coordinates of the lower left corner of the current image / preceding image, Indicates the latitude and longitude coordinates of the lower right corner of the current image / preceding image, Indicates the central latitude and longitude, Indicates the latitude change amount of the current image / preceding image, Indicates the longitude change amount of the current image / preceding image.

[0036] According to the latitude and longitude coordinates of the four vertices, four side line segments are determined by pairwise combining the four vertices in the order of top, bottom, left, and right. That is, the latitude and longitude coordinates of the upper left corner and the upper right corner can determine the upper side line segment, and the determination methods of the side line segments in the other three directions are the same, so they will not be elaborated here one by one.

[0037] Determine the edge distance between the upper side line segment of each current image and the upper side line segments of at least some of the preceding images in the historical inspection image set. Specifically, determine the positional relationship (intersecting, parallel, skew) between the two upper side line segments according to the latitude and longitude coordinates of the vertices corresponding to the upper side line segments of the current image and the preceding image, and calculate the edge distance between the two according to the positional relationship. Given the endpoint coordinates and positional relationship of two line segments, how to calculate the distance between the two line segments is a conventional technical means in the art and will not be elaborated here. Use the above method to calculate the edge distances between the lower, left, and right side line segments of each current image and the corresponding lower, left, and right side line segments of each preceding image respectively.

[0038] Respectively determine the minimum values of the edge distances in the four directions of the upper, lower, left, and right of the current image, and determine whether the minimum value of the corresponding edge distance is less than the edge preset value. If the minimum value of the edge distance is less than the edge preset value, then use the preceding image corresponding to the minimum value of the edge distance less than the edge preset value as the direction matching image for the corresponding side of the current image. For example, if the minimum value of the edge distance of the upper side of the current image is less than the edge preset value, then use the preceding image corresponding to the edge distance less than the edge preset value as the direction matching image for the upper side of the current image. If the minimum value of the edge distance is not less than the edge preset value, then there is no direction matching image for the corresponding side of the current image. Exemplarily, the value of the edge preset value is obtained by rounding to the nearest integer 20% of the distance represented by the short side pixel of the image field of view range (the range that the pod camera can capture).

[0039] It should be noted that the "at least part" here refers to the first quantity (which can be set to 10 - 20) of the previous images adjacent to the direction-matching image of the previous current image adjacent to the current image. Or it refers to the second quantity (which can be set to 10 - 20) of the previous images along the matching time sequence direction of the direction-matching image of the previous current image adjacent to the current image. Among them, the matching time sequence direction is the time sequence change direction (i.e., the matching time sequence direction) in which the previous images in the historical inspection image set are sequentially matched according to the matching situation between multiple previous current images in the current inspection image set and the previous images in the historical inspection image set.

[0040] In an embodiment of the present application, during the acquisition of inspection images, they are acquired in chronological order along the pipeline laying route, and the amount of image data is large. If the distance between the current image at the current moment in the current time period and all previous images in the historical inspection image set of the previous time period is calculated in real time, it will increase the overall time consumption of the method of the present application. Therefore, in the central region matching process, the first current image in the current time period is used to perform central region matching with all previous images in the historical inspection image set of the previous time period. The central distance between the central longitude and latitude of the first current image and each previous image is calculated, and the minimum value among multiple central distances is determined. By comparing the minimum value of the central distance with the central threshold, the central matching image of the first current image is determined. Find the position i of the central matching image of the first current image in the historical inspection image set of the previous time period. When performing central region matching on subsequent current images in the current inspection image set, only select a (10 - 20, which can be set according to actual needs) previous images in the historical inspection image set along the front and back two directions at the position i, then calculate the central distance between the subsequent current images and the central longitudes and latitudes of these a previous images, and then determine their respective central matching images according to the minimum value of the central distance, and update the position i according to the position of the central matching image in the historical inspection image set. The selection method of the previous images during direction matching can be the same as that of the central region matching, and only select a previous images at the position i in the historical inspection image set to calculate the direction matching image. By this method, part of the calculation amount can be reduced.

[0041] Specifically, the following method can also be used when performing the central region matching in step 301. First, obtain the central longitude and latitude of the current image in the current inspection image set, and obtain the central longitude and latitude of all previous images in the historical inspection image set. For the first current image in the current inspection image set, calculate the central distance between the central longitude and latitude of the first current image and the central longitude and latitude of each previous image in the historical inspection image set in turn. If the minimum value among the central distances is less than the central threshold, then use the previous image corresponding to the minimum value of the central distance as the central matching image of the first current image, and record its position i in the historical inspection image set. If the minimum value among the central distances of the first current image is not less than the central threshold, then the first current image has no central matching image. Continue to use the subsequent current images in the current inspection image set as the first current image, and execute the above steps until the position i is determined.

[0042] For the subsequent current images in the current inspection image set after determining the position i, the steps for determining the central matching image are as follows: Determine a adjacent previous images in the two directions before and after the position i as the first image set. The subsequent current images traverse the first image set, and calculate the central distance between the central longitude and latitude of the subsequent current images and the central longitude and latitude of the previous images in the first image set in turn. If the minimum value among the central distances is less than the central threshold, then use the previous image corresponding to the minimum value of the central distance as the central matching image, and update the position i according to the position of the central matching image in the historical inspection image set. If the minimum value among the central distances is not less than the central threshold, then this subsequent current image has no central matching image, continue to execute the above determination steps for the remaining subsequent current images, and use the position i at this time.

[0043] Perform the above determination steps on the subsequent current images in turn until the central region matching is performed on all current images in the current inspection image set.

[0044] Specifically, the following method can also be used when performing the direction matching in step 302. First, obtain the longitude and latitude coordinates of the four vertices of the current image in the current inspection image set, and obtain the longitude and latitude coordinates of the four vertices of all previous images in the historical inspection image set. According to the longitude and latitude coordinates of the four vertices of the current image / previous image, determine four edge line segments by combining the four vertices in pairs in the order of up, down, left, and right. That is, the longitude and latitude coordinates of the upper left corner and the upper right corner can determine the upper edge line segment, and the determination methods of the edge line segments in the other three directions are the same, so they will not be elaborated here one by one.

[0045] When determining the position i based on the last current image for central region matching, a pre-images adjacent to each other in both the front and back directions of the position i are determined as the second image set. The four side line segments of the current image are respectively used to calculate the side distances between the corresponding side line segments in the second data set. If the minimum value of the side distance in a certain direction of the current image is less than the side preset value, the direction matching image of the current image in this direction is obtained, and the position i is updated according to the position of the direction matching image in the historical inspection image set. The remaining current images in the current inspection image set are sequentially used to determine the corresponding direction matching images in the four directions according to the above method, and the position i is updated. If the minimum value of the side distance in a certain direction of the current image is not less than the side preset value, the corresponding position i is not updated. The subsequent current images continue to execute the above method until all the current images in the current inspection image set have been directionally matched.

[0046] Step 303: Obtain the matching image set of each current image in the current inspection image set according to the central matching image and the direction matching image. In the embodiment of the present application, the central matching image and the direction matching image of each current image are stored in the matching image set.

[0047] By means of combining the central longitude and latitude with the side line segments, the direction matching images in the four directions of up, down, left, and right and the central matching image of the central region are matched for the current image in the historical inspection image set, which can reduce the missed detection rate while ensuring the integrity of the detection area.

[0048] Step 102: Extract the overlapping regions in the current inspection image set and its corresponding matching image set to determine the alignment images. Among them, the matching point pairs include the current matching points and the pre-matching points. In the embodiment of the present application, the current images in the current inspection image set and the pre-images in the corresponding matching image set are segmented to obtain the current segmented images and the pre-segmented images. The current segmented images / pre-segmented images are scaled down, and the feature points therein are extracted for feature point matching to obtain the matching point pairs. Among them, the matching point pairs include the current matching points and the pre-matching points. The current matching points / pre-matching points are restored to the current segmented images / pre-segmented images, and the current matching points / pre-matching points in each current segmented image / pre-segmented image are merged. The overlapping regions are determined according to the merged current matching points and pre-matching points to obtain the alignment images.

[0049] Determine the overlapping region based on the merged current matching points and the previous matching points, including: determining the transformation matrix based on the merged current matching points and the previous matching points. Determine the current perspective point through the transformation matrix and the longitude and latitude coordinates of the four vertices of the current image, and determine the current overlapping region according to the current perspective point. Perform perspective transformation on the previous image using the transformation matrix to obtain the previous overlapping region. Determine the maximum inscribed rectangle within the current overlapping region according to the current perspective point. Intercept the current overlapping region and the previous overlapping region respectively through the maximum inscribed rectangle.

[0050] Specifically, after center region matching and direction matching, the current image and the corresponding previous image are basically in the same position. However, due to the fact that the pose information of the drone and the pod camera cannot be kept consistent at different times, there are non-overlapping regions in the collected inspection images, and the overlapping regions are not aligned, so it is impossible to directly perform change detection through time series change detection algorithms.

[0051] If the collected inspection images are high-resolution images, the number of feature points extracted is fixed, and the matched feature points are likely to be concentrated in a certain area, resulting in a small or incorrect overlapping region being extracted. To ensure that the area of the aligned overlapping region is the largest, the feature points must be correctly matched and evenly distributed. Therefore, this application adopts the method of segmenting the inspection images, extracting feature points, and merging them after matching to prevent the matched feature points from gathering in a certain area and ensure the accuracy of the overlapping region. In this process, to speed up the calculation, the images are downsized.

[0052] Specifically as Figure 6 shown, the current image and the previous image in the corresponding matching image set are respectively segmented into n segmented images. n is determined according to actual needs, and it is required that the square root of n is as close to an integer as possible to facilitate image segmentation and calculation. Exemplarily, n is set to 4, so the width and height of the segmented images after segmentation are half of the width and height of the current image / previous image. The segmented images are successively downsized, and exemplarily downsized to half of the original size, that is, the width and height of the downsized segmented images are one-fourth of the width and height of the current image / previous image. Successively detect and extract the feature points of the downsized segmented image j, and match the feature points of the current image with the feature points of the previous image in its corresponding matching image set to obtain matching point pairs. All the matching point pairs of all the segmented images are obtained by successively traversing all the segmented images. Among them, the matching point pairs include the current matching points and the previous matching points. As Figure 8 shown, where the left half is the previous image and the right half is the current image.

[0053] Restore the current matching points / previous matching points to the current segmented image / previous segmented image before downsizing, and then merge the current matching points / previous matching points in each current segmented image / previous segmented image. Determine the transformation matrix based on the merged current matching points and the previous matching points.

[0054] Specifically, the calculation method of the transformation matrix H is as follows: .

[0055] In the formula, represents the previous matching point, represents the current matching point. Exemplarily, the transformation matrix is as follows: .

[0056] In the formula, H represents the transformation matrix, , , , , , represent the rotation amount between the matching point pairs, , , represent the offset between the matching point pairs.

[0057] Determine four current perspective points according to the four vertices of the current image, and determine the current overlapping area according to the four current perspective points. Then determine a maximum inscribed rectangle within the current overlapping area, and the preferred embodiment is a maximum inscribed square rectangle. Then, by multiplying the previous image with the inverse transformation matrix, the previous overlapping area is obtained. Finally, use the maximum inscribed rectangle to intercept the current overlapping area and the previous overlapping area respectively to obtain the aligned images. As Figure 9 and Figure 10 shown are the current aligned image and the previous aligned image respectively.

[0058] Step 103: Use the trained change detection model to perform time series change detection on the aligned images to obtain the change detection results of the current images in the current inspection image set. In the embodiment of the present application, perform time series change detection on the aligned images of the template size to obtain a binary image. Restore the binary image of the template size to the original size of the aligned image. Superimpose the binary image of the original size with the corresponding aligned image to obtain a global grayscale image. Binarize the global grayscale image to obtain the change detection results of the current images.

[0059] Specifically, for high-resolution images (such as: 9552 6368, unit: pixel). Before performing change detection, it is necessary to cut the high-resolution image to ensure that the method of this application can run properly with a smaller video memory. After extracting the overlapping area and obtaining the aligned image, fix the size of the aligned image, and the length is equal to the height, and at the same time, the length and height are integer multiples of 32. This is to avoid the aligned image being compressed in an indeterminate ratio during direct change detection, which may lead to target deformation or large feature differences, affecting the accuracy of the change detection result. At the same time, it will cause information loss of the aligned image and affect the detection accuracy. In addition, due to hardware conditions, if the image size for change detection is too large, it will lead to an increase in time consumption or a situation where the video memory is not enough to run.

[0060] Therefore, before performing step 103, it is also possible to: judge whether the original size of the aligned image is larger than the set template size. If the original size of the aligned image is larger than the template size, divide the aligned image into several split images of the template size. When the width and / or height of the remaining image after splitting is less than the template size, make up for it within the range of the aligned image and then divide it into split images. If the original size of the aligned image is smaller than the template size, fill the aligned image to the size of the template size to obtain the filled aligned image. In addition, it is also possible to perform an inversion process on the split / filled aligned image.

[0061] Among them, the aligned image includes the current aligned image and the previous aligned image. The template size is exemplarily set to 512 512, unit: pixel.

[0062] Specifically, judge whether the original size of the aligned image is larger than the set template size. If the original size of the aligned image is larger than the template size, divide the aligned image into several split images of the template size. When the width and / or height of the remaining image after splitting is less than the template size, make up for it within the range of the aligned image and then divide it into split images. If the original size of the aligned image is smaller than the template size, fill the aligned image to the size of the template size to obtain the filled aligned image.

[0063] Furthermore, as Figure 7 shown, the entire black shaded part is the aligned image, and the solid red frame is the non-overlapping split image divided according to the size of the template. The black dashed frames below and on the right are the split images with overlapping areas divided when the remaining part of the aligned image is not enough to divide the template size and is moved up / left to completely divide the template size. If the size of the aligned image is smaller than the template size, uniformly fill black on the top, bottom, left, and right of the aligned image until the filled aligned image is equal to the size of the template size. That is, fill black with the height on the top and bottom of the aligned image, and fill black with the width on the left and right sides of the aligned image. Among them, and the width and height representing the template size and the width and height representing the original size of the aligned image.

[0064] Perform color inversion on the segmented / filled aligned image. Specifically, split the color of the segmented / filled aligned image into three channels: R, G, B (red, green, blue), subtract the color values in the R, G, and B channels from 255 respectively to obtain the color differences in the R, G, and B channels, and then combine the color differences in the R, G, and B channels to achieve color inversion. As Figure 11 and Figure 12 is a set of example images before and after color inversion for the aligned image.

[0065] In the embodiment of the present application, perform temporal change detection on the aligned image of the original size to obtain a black-and-white binary image. Restore the black-and-white binary image of the template size to the original size of the aligned image. Superimpose the black-and-white binary image of the original size with the corresponding aligned image to obtain a global grayscale image. Binarize the global grayscale image to obtain the change detection result of the current image.

[0066] Specifically, use a temporal change detection algorithm to perform temporal change detection on the segmented / filled aligned image, and the obtained detection result is a black-and-white binary image, as Figure 13 shown. Specifically, restore the black-and-white binary image of the template size to the original size of the aligned image. That is, for the black-and-white binary image of the aligned image (i.e., the segmented image) that has been segmented, splice it again to restore it to the original size. For the black-and-white binary image of the filled aligned figure, remove the black edges filled around it to obtain the black-and-white binary image of the original size. Superimpose the black-and-white binary image of the original size with the aligned image of the original size to obtain a global grayscale image, and then binarize the global grayscale image to obtain the change detection result of the current image.

[0067] In the embodiment of the present application, after obtaining the change detection result of the current image in the current inspection image set, it is also possible to: add at least part of the change detection results to the sample library of the change detection model for updating the change detection model. Input the to-be-inspected inspection image into the updated change detection model to obtain the change detection result of the to-be-inspected inspection image.

[0068] In the embodiments of the present application, before adding at least part of the change detection results to the sample library of the change detection model for updating the change detection model, the following operations can also be performed: converting the change detection results into labeled samples; screening the labeled samples, removing the misdetected contours therein, supplementing the undetected contours, and adjusting the labeled boxes until they fit the contours in the labeled samples. Among them, converting the change detection results into labeled samples includes: storing the change detection results in a set image format to obtain a change detection image; extracting the contours in the change detection image and writing the extracted contours into a file and storing them as labeled samples.

[0069] Specifically, the set image format includes the ".bmp" (a format of an image) or ".png" (a format of an image) format. Extracting the contours in the change detection image, that is, the contours of the white part, saving the extracted contours in a file in the ".json" (a text file written in JSON format) format to obtain labeled samples, counting the number of the extracted contours in the labeled samples, and marking the number of the contours in the labeled samples. If the number of the contours is 0 when storing in the labeled samples, use "null" (which means "empty" in common programming languages) as the marking result of the labeled samples. Then open the labeled samples with a marking tool for screening, and manually remove the misdetected contours therein, supplement the undetected contours, and adjust the labeled boxes until they fit the contours.

[0070] Then, use the labeled samples in combination with the historical labeled samples to train and update the change detection model. The historical labeled samples here can be samples manually labeled purely or the labeled samples obtained from the output processing in the previous rounds of the method of the present application.

[0071] Those skilled in the art should be aware that after obtaining the labeled samples converted from the change detection results, if directly merging them with the historical labeled samples for training and updating the change detection model, it will increase the training time of the change detection model. The change detection model should be updated irregularly according to the actual scenario. If adding labeled samples on the basis of the historical labeled samples without limitation, it will lead to an increasing number of samples, and at the same time, the training time of the change detection model will also become longer and longer, affecting the efficiency of updating the change detection model. Therefore, when updating the change detection model, the labeled samples and the historical labeled samples can be added or deleted according to the actual needs and then used for training and updating.

[0072] In the embodiments of the present application, the to-be-inspected patrol image can also be input into the updated change detection model to obtain the change detection result of the to-be-inspected patrol image.

[0073] Through the method of the present application, it is possible to detect the changed area based on the image features for the inspection images above the pipeline. For the inspection images of the new scenario, sample marking and model training and updating can be quickly achieved, and thus it can be more convenient and efficient in the application of the new scenario.

[0074] In addition, in the existing change detection methods, the data security risk is relatively high in the process of generating and distributing map tiles, and it cannot be applied to the pipeline inspection involving sensitive areas. However, the method of the present application adopts image cutting for the high-resolution image and then performs change detection, and then stitches the detection results. The change detection can be carried out in the offline state with limited device performance.

[0075] Although the present application provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on routine or non-creative labor. The step sequence listed in this embodiment is only one way among the execution sequences of numerous steps and does not represent the only execution sequence. When the actual device or client product is executed, it can be executed in the method sequence shown in this embodiment or the drawings or executed in parallel (such as in an environment of parallel processors or multi-threaded processing).

[0076] As Figure 14 shown, the embodiment of the present application also provides an image change detection device 1400 based on feature matching. The device includes: a matching module 1401, an extraction module 1402, and a detection module 1403, which are specifically as follows.

[0077] The matching module 1401 is used to perform image matching on the historical inspection image set of the previous time period by using multiple regions of each current image in the current inspection image set of the current time period to obtain a matching image set.

[0078] The extraction module 1402 is used to extract the overlapping regions in the current inspection image set and its corresponding matching image set to determine the aligned images; wherein, the aligned images include the current aligned image and the previous aligned image.

[0079] The detection module 1403 is used to perform time series change detection on the aligned images by using the trained change detection model to obtain the change detection results of the current images in the current inspection image set.

[0080] Some of the modules in the device described in this application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0081] The devices or modules illustrated in the above application embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. For the convenience of description, when describing the above devices, they are divided into various modules according to functions and described separately. When implementing the embodiments of this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the modules implementing a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0082] The methods, devices or modules described in this application can be implemented in the form of computer-readable program code. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or a processor, and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, the method steps can be logically programmed to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0083] The embodiments of this application also provide a device, which includes: a processor; a memory for storing processor-executable instructions; when the processor executes the executable instructions, the method described in the embodiments of this application is implemented.

[0084] In addition, in each embodiment of the present invention, each functional module can be integrated into one processing module, or each module can exist independently, or two or more modules can be integrated into one module.

[0085] The above storage medium includes but is not limited to Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0086] From the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or can also be reflected in the implementation process of data migration. This computer software product can be stored in a storage medium such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions to enable a computer device (which can be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments of this application.

[0087] The various embodiments in this specification are described in a progressive manner. For the same or similar parts between the various embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, small computers, large computers, distributed computing environments including any of the above systems or devices, and so on.

[0088] The above embodiments are only used to illustrate the technical solutions of this application, rather than to limit this application; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. An image change detection method based on feature matching, characterized in that, Including: Performing image matching on a historical inspection image set of its previous time period by using multiple regions of each current image in the current inspection image set of the current time period to obtain a matching image set; Extracting overlapping regions in the current inspection image set and its corresponding matching image set to determine aligned images; wherein, the aligned images include a current aligned image and a previous aligned image; Performing time series change detection on the aligned images by using a trained change detection model to obtain a change detection result of the current image in the current inspection image set.

2. The method according to claim 1, characterized in that, The performing image matching on a historical inspection image set of its previous time period by using multiple regions of each current image in the current inspection image set of the current time period to obtain a matching image set includes: Performing central region matching on the current images in the current inspection image set with previous images in the historical inspection image set respectively to obtain central matching images of the current images; Performing direction matching on the current images in the current inspection image set with previous images in the historical inspection image set respectively to obtain direction matching images of the current images; Obtaining the matching image set of each current image in the current inspection image set according to the central matching images and the direction matching images.

3. The method according to claim 2, characterized in that, The performing central region matching on the current images in the current inspection image set with previous images in the historical inspection image set respectively to obtain central matching images of the current images includes: Determining the central longitude and latitude of the current image in the current inspection image set and the previous image in the historical inspection image set; Respectively determining the central distances between the central longitude and latitude of the current image and the central longitude and latitude of at least part of the previous images in the historical inspection image set; Determining the minimum value of the central distance of the current image; If the minimum value of the central distance is less than a central threshold, using the previous image corresponding to the minimum value of the central distance as the central matching image of the current image.

4. The method according to claim 2, characterized in that, The performing direction matching on the current images in the current inspection image set with previous images in the historical inspection image set respectively to obtain direction matching images of the current images includes: Determining the longitude and latitude coordinates of the four vertices of the current image / previous image according to the image field of view range and the central longitude and latitude of the current image / previous image; Determining the edge line segments of the four sides of the current image / previous image according to the longitude and latitude coordinates of the four vertices of the current image / previous image; Respectively determining the edge distances between each edge line segment of the current image and the edge line segments corresponding to at least part of the previous images in the historical inspection image set; Respectively determining the minimum values of the multiple edge distances of the current image; If the minimum value of the edge distance is less than an edge preset value, using the previous image corresponding to the minimum value of the edge distance as the direction matching image of the corresponding edge of the current image.

5. The method according to claim 1, characterized in that, The extracting overlapping regions in the current inspection image set and its corresponding matching image set to determine aligned images includes: Segment the current image in the current inspection image set and its corresponding previous image in the matching image set to obtain a current segmented image and a previous segmented image; Shrink the current segmented image / previous segmented image, extract feature points therein for feature point matching to obtain matching point pairs; wherein, the matching point pairs include current matching points and previous matching points; Restore the current matching points / previous matching points to the current segmented image / previous segmented image, and merge the current matching points / previous matching points in each of the current segmented images / previous segmented images; Determine the overlapping area based on the merged current matching points and previous matching points to obtain the aligned image.

6. The method according to claim 1, characterized in that, Before using the trained change detection model to perform time series change detection on the aligned image, it further includes: Judge whether the original size of the aligned image is greater than the set template size; If the original size of the aligned image is greater than the template size, divide the aligned image into several segmented images of the template size. When the width and / or height of the remaining image after segmentation is less than the template size, make up for it within the range of the aligned image and then divide it into the segmented images; If the original size of the aligned image is less than the template size, fill the aligned image to the template size to obtain a filled aligned image.

7. The method according to claim 6, characterized in that, Using the trained change detection model to perform time series change detection on the aligned image to obtain the change detection result of the current image in the current inspection image set, including: Perform time series change detection on the aligned image of the template size to obtain a black and white binary image; Restore the black and white binary image of the template size to the original size of the aligned image; Overlay the black and white binary image of the original size with the corresponding aligned image to obtain a global grayscale image; Binarize the global grayscale image to obtain the change detection result of the current image.

8. The method according to claim 1, wherein After obtaining the change detection result of the current image in the current inspection image set, it further includes: Add at least part of the change detection results to the sample library of the change detection model for updating the change detection model; Input the inspection image to be measured into the updated change detection model to obtain the change detection result of the inspection image to be measured.

9. The method according to claim 8, wherein Before adding at least part of the change detection results to the sample library of the change detection model, it further includes: Convert the change detection result into a labeled sample; Screen the labeled sample, remove the misdetected contours therein, supplement the undetected contours, and adjust the bounding box until it fits the contours in the labeled sample; Among them, converting the change detection result into a labeled sample includes: Store the change detection result in a set image format to obtain a change detection image; Extract the contours in the change detection image and write the extracted contours into a file and store them as labeled samples.

10. An image change detection device based on feature matching, wherein It includes: A matching module, configured to perform image matching on the historical inspection image set of the previous period by using multiple regions of each current image in the current inspection image set of the current period to obtain a matching image set; An extraction module, configured to extract an overlapping region from the current inspection image set and its corresponding matching image set to determine an aligned image; wherein the aligned image includes a current aligned image and a previous aligned image. A detection module, configured to perform time-series change detection on the aligned image by using a trained change detection model to obtain a change detection result of the current image in the current inspection image set.

Images