Image change detection method and device based on feature matching

Abstract

The present application discloses a method and device for detecting image changes based on feature matching, which relates to the field of pipeline detection technology. The method comprises: using multiple regions of each current image in a current inspection image set in a current period to perform image matching on a historical inspection image set in a previous period to obtain a matching image set; extracting overlapping regions from the current inspection image set and its corresponding matching image set to determine an aligned image; wherein the aligned image includes the current aligned image and the previous aligned image; and using a trained change detection model to perform time series change detection on the aligned images to obtain a change detection result for the current image in the current inspection image set. This method solves the problem that existing change detection methods are complex, time-consuming, and difficult to detect changes above the pipeline in a timely manner. It can quickly respond to and prevent events that threaten pipeline transportation, thereby reducing the incidence of accidents and improving pipeline management efficiency.

Description

Technical Field

[0001] The present application relates to the field of pipeline detection technology, and in particular to a method and device for detecting image changes based on feature matching. Background Art

[0002] With rapid economic development, the demand for oil and gas resources is increasing, and the mileage of underground pipelines is also increasing. Accumulation, heavy vehicles, construction, or natural disasters such as landslides, collapses, and floods above and around the pipelines can exert pressure on the pipelines, causing them to deform, sink, rupture, and damage, leading to leaks. Therefore, the safety of the areas above and around the pipelines is crucial during transportation.

[0003] Change detection in pipeline areas can promptly identify changes in the pipeline's surrounding environment, prevent potential safety hazards and accidents, and ensure the efficiency and safety of pipeline transportation. Change detection is a method used to identify, analyze, and quantify changes in the surface or target objects over time. Existing change detection methods primarily use orthophotos to generate map tiles to detect and analyze change characteristics for the same area at different time points. However, during pipeline inspections at closely spaced intervals, map tile production is difficult and time-consuming, making it difficult to detect changes above the pipeline in a timely manner. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for detecting image changes based on feature matching, thereby solving the problem that the change detection methods in the prior art are complex and time-consuming, and are difficult to detect changes above the pipeline in a timely manner.

[0005] In the first aspect, an embodiment of the present application provides an image change detection method based on feature matching, including: using multiple areas of each current image in the current inspection image set of the current time period to perform image matching on the historical inspection image set of the previous time period to obtain a matching image set; extracting the overlapping areas in the current inspection image set and its corresponding matching image set to determine an aligned image; wherein the aligned image includes the current aligned image and the previous aligned image; using the trained change detection model to perform time series change detection on the aligned image to obtain a change detection result of the current image in the current inspection image set.

[0006] In combination with the first aspect, in a possible implementation method, the multiple areas of each current image in the current inspection image set of the current time period are used to perform image matching on the historical inspection image set of the previous time period to obtain a matching image set, including: performing center area matching on the current image in the current inspection image set with the previous image in the historical inspection image set to obtain a center matching image of the current image; performing direction matching on the current image in the current inspection image set with the previous image in the historical inspection image set to obtain a direction matching image of the current image; and obtaining the matching image set of each current image in the current inspection image set based on the center matching image and the direction matching image.

[0007] In combination with the first aspect, in a possible implementation method, the current image in the current inspection image set is respectively matched with the previous images in the historical inspection image set in the center area to obtain the center matching image of the current image, including: determining the center longitude and latitude of the current image in the current inspection image set and the previous images in the historical inspection image set; respectively determining the center distance between the center longitude and latitude of the current image and the center longitude and latitude of at least part of the previous images in the historical inspection image set; determining the minimum value of the center distance of the current image; if the minimum value of the center distance is less than the center threshold, then using the previous image corresponding to the minimum value of the center distance as the center matching image of the current image.

[0008] In combination with the first aspect, in a possible implementation method, the current image in the current inspection image set is directionally matched with the previous image in the historical inspection image set to obtain a direction-matched image of the current image, including: determining the longitude and latitude coordinates of the four vertices of the current image / previous image based on the image field of view and the center longitude and latitude of the current image / previous image; determining the edge segments of the four sides of the current image / previous image based on the longitude and latitude coordinates of the four vertices of the current image / previous image; respectively determining the edge distance between each edge segment of the current image and the edge segments corresponding to at least part of the previous images in the historical inspection image set; respectively determining the minimum value of the multiple edge distances of the current image; if the minimum value of the edge distance is less than the edge preset value, then the previous image corresponding to the minimum value of the edge distance is used as the direction-matched image of the edge corresponding to the current image.

[0009] In combination with the first aspect, in a possible implementation method, the extracting of the overlapping area in the current inspection image set and its corresponding matching image set to determine the aligned image includes: segmenting the current image in the current inspection image set and its corresponding previous image in the matching image set to obtain a current segmented image and a previous segmented image; reducing the current segmented image / previous segmented image, and extracting feature points therein for feature point matching to obtain a matching point pair; wherein the matching point pair includes a current matching point and a previous matching point; restoring the current matching point / previous matching point to the current segmented image / previous segmented image, and merging the current matching point / previous matching point in each of the current segmented image / previous segmented image; and determining the overlapping area based on the merged current matching point and previous matching point to obtain the aligned image.

[0010] In combination with the first aspect, in a possible implementation method, before using the trained change detection model to perform time series change detection on the aligned image, it also includes: judging whether the original size of the aligned image is larger than the set template size; if the original size of the aligned image is larger than the template size, then dividing the aligned image into several segmentation maps of the template size, and when the width and / or height of the remaining image after segmentation is less than the template size, it is padded within the range of the aligned image and then divided into the segmentation map; if the original size of the aligned image is smaller than the template size, then filling the aligned image to the template size to obtain the padded aligned image.

[0011] In combination with the first aspect, in a possible implementation method, the trained change detection model is used to perform time series change detection on the aligned image to obtain a change detection result of the current image in the current inspection image set, including: performing time series change detection on the aligned image of the template size to obtain a black and white binary image; restoring the black and white binary image of the template size to the original size of the aligned image; superimposing the black and white binary image of the original size with the corresponding aligned image to obtain a global grayscale image; binarizing the global grayscale image to obtain the change detection result of the current image.

[0012] In combination with the first aspect, in a possible implementation method, after obtaining the change detection result of the current image in the current inspection image set, it also includes: adding at least part of the change detection result to the sample library of the change detection model for updating the change detection model; inputting the inspection image to be tested into the updated change detection model to obtain the change detection result of the inspection image to be tested.

[0013] In combination with the first aspect, in a possible implementation method, before adding at least part of the change detection results to the sample library of the change detection model, it also includes: converting the change detection results into labeled samples; screening the labeled samples, removing falsely detected contours, supplementing missed contours, and adjusting the marking box until it fits the contour in the labeled sample; wherein, converting the change detection results into labeled samples includes: storing the change detection results in a set image format to obtain a change detection image; extracting the contours in the change detection image, and writing the extracted contours into a file to store as labeled samples.

[0014] In the second aspect, an embodiment of the present application provides an image change detection device based on feature matching, including: a matching module, used to use multiple areas of each current image in the current inspection image set of the current time period to perform image matching on the historical inspection image set of the previous time period to obtain a matching image set; an extraction module, used to extract the overlapping areas in the current inspection image set and its corresponding matching image set to determine an aligned image; wherein the aligned image includes the current aligned image and the previous aligned image; a detection module, used to use a trained change detection model to perform time series change detection on the aligned image to obtain a change detection result of the current image in the current inspection image set.

[0015] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0016] Through image matching, the embodiments of the present application can ensure the integrity of the inspection area during the inspection process, reducing missed targets. Aligning images ensures the accuracy of overlapping areas. This effectively addresses the problem of complex and time-consuming change detection methods in existing technologies, which make it difficult to detect changes above the pipeline in a timely manner. This avoids the time-consuming and complex production of map tiles, meets the requirements of closely spaced pipeline inspections, and allows for rapid response and prevention of events that threaten pipeline transportation. This can reduce the incidence of accidents, improve pipeline management efficiency, and ensure safe pipeline operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of an image change detection method based on feature matching provided in an embodiment of the present application;

[0019] Figure 2 An example diagram of image matching provided in an embodiment of the present application;

[0020] Figure 3 A flowchart of a method for obtaining a matching image set provided in an embodiment of the present application;

[0021] Figure 4 A flow chart of the central area matching method provided in an embodiment of the present application;

[0022] Figure 5 A flow chart of the direction matching method provided in an embodiment of the present application;

[0023] Figure 6 A flow chart of a method for determining alignment images provided in an embodiment of the present application;

[0024] Figure 7 An example diagram of the segmented and aligned images provided in an embodiment of the present application;

[0025] Figure 8 An example diagram of matching point pairs between the current image and the previous image provided in an embodiment of the present application;

[0026] Figure 9 An example diagram of the current alignment image provided in an embodiment of the present application;

[0027] Figure 10 This is an example diagram of a preamble alignment image provided in an embodiment of the present application;

[0028] Figure 11 This is an example diagram of an aligned image before color inversion processing provided in an embodiment of the present application;

[0029] Figure 12 This is an example diagram of an aligned image after color inversion processing provided in an embodiment of the present application;

[0030] Figure 13 This is an example of a black and white binary image provided in an embodiment of the present application;

[0031] Figure 14 A schematic structural diagram of an image change detection device based on feature matching provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] The following description of some of the technologies involved in the embodiments of this application is provided to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted from the following description.

[0034] Change detection in pipeline areas primarily involves periodic drone inspections, capturing images of the pipeline perimeter at different times. Comparing images of the same pipeline area from previous and subsequent inspection periods reveals areas where differences exist between the current inspection image (the current image) and the previous inspection image set (the previous image). During inspections, it is recommended that the angle between the drone and the pod camera be greater than 70 degrees. This means that the pod camera should maintain a vertical viewing angle of no more than 20 degrees from vertical. During pipeline inspections, objects detected for change include vehicles, groundbreaking works, accumulated water, accumulations, and buildings.

[0035] Those skilled in the art should realize that when the drone conducts an inspection in each time period (the time required for the drone to fly over the inspection area and complete an inspection), the time of the inspection images collected is different, that is, the current image / previous image in the current inspection image set / historical inspection image set can be stored in sequence according to their collection time.

[0036] Figure 1 This is a flowchart of an image change detection method based on feature matching provided by an embodiment of the present application, including steps 101 to 103. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of an image change detection method based on feature matching. Figure 1 The steps shown may be performed in parallel or reversed.

[0037] Step 101: Use multiple regions of each current image in the current inspection image set of the current time period to perform image matching on the historical inspection image set of the previous time period to obtain a matching image set. In this embodiment of the present application, the current inspection image set is inspection images of different areas of the pipeline area captured by the drone during the current time period (i.e., the current image) (inspection images are typically captured at a predetermined location). The historical inspection image set is inspection images of different areas of the pipeline area captured by the drone during the inspection period before the current time period (i.e., the previous image). It is understood that the current inspection image set and the multiple inspection images in the historical inspection image set are captured at the same or adjacent locations, but at different time periods. Furthermore, when performing image matching on the current image, image matching can be performed for each inspection image captured in real time during the current time period, or it can be performed for each inspection image captured in real time during the current time period, followed by sequential image matching.

[0038] Furthermore, each current image in the current inspection image set is traversed, and each current image is sequentially matched with each (or at least part of) the previous image in the historical inspection image set. Image matching includes center area matching and direction matching. Figure 2 As shown, the solid arrows indicate center area matching, and the dotted arrows indicate direction matching.

[0039] In an embodiment of the present application, during the inspection process of the drone, inspection images will be collected and saved at a given series of positioning points to obtain a historical inspection image set and a current inspection image set. During the inspection process at different time periods, due to the deviation of the drone positioning information, even if the inspection images are collected at the same positioning point, the latitude and longitude information carried by the inspection images are not equal, and the inspection images in the two inspection image sets cannot be completely matched. Moreover, after the inspection images are matched, due to the momentary changes in the drone's posture and the pod camera angle, the matched inspection images only have overlapping areas in the middle area. In order to ensure the integrity of the detection area during the detection process and reduce missed targets, Figure 3 The steps in perform one-to-many matching of the inspection images in the two inspection image sets.

[0040] The specific implementation method of step 101 is as follows Figure 3 As shown, it includes steps 301 to 303, which are specifically as follows.

[0041] Step 301: Match the center areas of the current image in the current inspection image set with the previous images in the historical inspection image set to obtain a center matching image of the current image. In an embodiment of the present application, the center longitude and latitude of the current image in the current inspection image set and the previous images in the historical inspection image set are determined. The center distances between the center longitude and latitude of the current image and the center longitude and latitude of at least some of the previous images in the historical inspection image set are determined respectively. The minimum value of the center distance of the current image is determined. If the minimum value of the center distance is less than the center threshold, the previous image corresponding to the minimum value of the center distance is used as the center matching image of the current image.

[0042] Specific as Figure 4 As shown, the center longitude and latitude of the current image and the previous image are determined respectively. Here, the center longitude and latitude of all current images and all previous images can be obtained in advance, or the center longitude and latitude of a current image / previous image can be obtained when it is used.

[0043] The center distances between the center longitude and latitude of each current image and those of at least some of the previous images in the historical inspection image set are calculated sequentially. The minimum center distance for the current image is determined and compared to see if it is less than a center threshold. If so, the previous image corresponding to the minimum center distance is considered the center matching image for the current image. Otherwise, no center matching image exists for the current image.

[0044] Setting the center threshold too high will result in too little common area between the current image and the corresponding matching image, affecting subsequent detection accuracy. For example, the center threshold is determined by rounding off 20% of the distance represented by the shortest pixel of the image's field of view (the range captured by the pod camera).

[0045] It should be noted that "at least partially" here refers to a first number (which can be set to 10-20) of preceding images adjacent to the center-matching image of the previous current image adjacent to the current image. Alternatively, it refers to a second number (which can be set to 10-20) of preceding images adjacent to the center-matching image of the previous current image adjacent to the current image along the matching time sequence direction. The matching time sequence direction is the direction of sequentially matched time changes in the historical inspection image set (i.e., the matching time sequence direction), determined based on the matching results of the previous multiple current images in the current inspection image set with the preceding images in the historical inspection image set.

[0046] Step 302: Directionally match the current image in the current inspection image set with the previous image in the historical inspection image set to obtain a direction-matched image of the current image. In an embodiment of the present application, the longitude and latitude coordinates of the four vertices of the current image / previous image are determined based on the image field of view and the center longitude and latitude of the current image / previous image. Based on the longitude and latitude coordinates of the four vertices of the current image / previous image, the edge segments of the four sides of the current image / previous image are determined. The edge distances between each edge segment of the current image and the edge segments corresponding to at least part of the previous images in the historical inspection image set are determined respectively. The minimum values ​​of the multiple edge distances of the current image are determined respectively. If the minimum value of the edge distance is less than the edge preset value, the previous image corresponding to the minimum value of the edge distance is used as the direction-matched image of the edge corresponding to the current image.

[0047] Specific as Figure 5 As shown, the latitude and longitude coordinates of the four vertices of the current image / previous image are determined based on the image field of view and the center longitude and latitude of the current image / previous image. Specifically, the actual width and height of the image field of view of the current image / previous image are determined based on the camera parameters, the image acquisition height, and the image pixel width and height. The actual width and height of the image field of view are then converted into longitude and latitude changes in the latitude and longitude coordinate system.

[0048] Furthermore, the longitude and latitude coordinates of the four vertices of the current image / previous image are as follows:

[0049] ,

[0050] ,

[0051] ,

[0052] .

[0053] Where, Indicates the latitude and longitude coordinates of the upper left corner of the current image / previous image. Indicates the latitude and longitude coordinates of the upper right corner of the current image / previous image. Indicates the latitude and longitude coordinates of the lower left corner of the current image / previous image. Indicates the latitude and longitude coordinates of the lower right corner of the current image / previous image. Indicates the center longitude and latitude, Indicates the latitude change of the current image / previous image. Indicates the longitude change between the current image and the previous image.

[0054] According to the longitude and latitude coordinates of the four vertices, the four vertices are combined in pairs in the order of up, down, left, and right to determine the four edge segments in turn. That is, the longitude and latitude coordinates of the upper left corner and the upper right corner can determine the upper edge segment. The edge segments in the other three directions are determined in the same way, so they will not be described here one by one.

[0055] Determine the edge distance between each upper edge segment of the current image and the upper edge segments of at least some of the previous images in the historical inspection image set. Specifically, determine the positional relationship (intersecting, parallel, skewed) between the two upper edge segments based on the longitude and latitude coordinates of the vertices corresponding to the upper edge segments of the current image and the previous image, and calculate the edge distance between the two based on the positional relationship between the two upper edge segments. How to calculate the distance between the two line segments when the endpoint coordinates and positional relationship of the two line segments are known is a conventional technical means in this field and will not be repeated here. Use the above method to calculate the edge distances between the edge segments in the three directions of the bottom, left and right of each current image and the edge segments in the three directions of the bottom, left and right corresponding to each previous image.

[0056] Determine the minimum edge distances in the four directions of the top, bottom, left, and right of the current image, and determine whether the corresponding minimum edge distances are less than the edge preset value. If the minimum edge distance is less than the edge preset value, the previous image corresponding to the minimum edge distance less than the edge preset value is used as the direction matching image for the corresponding edge of the current image. For example, if the minimum edge distance above the current image is less than the edge preset value, the previous image corresponding to the edge distance less than the edge preset value is used as the direction matching image for the top of the current image. If the minimum edge distance is not less than the edge preset value, no direction matching image exists for the corresponding edge of the current image. Exemplarily, the edge preset value is obtained by rounding off 20% of the distance represented by the short side pixels of the image field of view (the range that the pod camera can capture).

[0057] It should be noted that "at least partially" here refers to a first number (which can be set to 10-20) of preceding images adjacent to the direction-matching image immediately preceding the current image. Alternatively, it refers to a second number (which can be set to 10-20) of preceding images along the matching time sequence direction. The matching time sequence direction is the direction of sequentially matched time changes in the historical inspection image set (i.e., the matching time sequence direction), determined based on the matching results between the previous multiple current images in the current inspection image set and the preceding images in the historical inspection image set.

[0058] In one embodiment of the present application, during the acquisition of inspection images, the images are acquired in chronological order along the laying route of the pipeline, and the amount of image data is large. If the distance between the current image at the current moment in the current time period and all the previous images in the historical inspection image set of the previous time period is calculated in real time, the overall time consumption of the method of the present application will increase. Therefore, in the process of center area matching, the first current image of the current time period is matched with all the previous images in the historical inspection image set of the previous time period for center area matching, the center distance between the center longitude and latitude of the first current image and each previous image is calculated, and the minimum value of the multiple center distances is determined. By comparing the minimum value of the center distance with the center threshold, the center matching image of the first current image is determined. Find the position i of the first current image's center matching image in the previous period's historical inspection image set. When performing center region matching for subsequent current images in the current inspection image set, simply select a (10-20, set as needed) preceding images in both the forward and backward directions at position i in the historical inspection image set. Calculate the center distances between the subsequent current image and the center longitude and latitude of these a preceding images. Determine the respective center matching images based on the minimum center distance, and update position i based on the center matching image's position in the historical inspection image set. The selection of preceding images for directional matching can be performed in the same manner as for center region matching: simply select a preceding image at position i in the historical inspection image set to calculate the directional matching image. This method can reduce some computational complexity.

[0059] Specifically, the following method can also be used when performing the center area matching of step 301. First, obtain the center longitude and latitude of the current image in the current inspection image set, and obtain the center longitude and latitude of all previous images in the historical inspection image set. For the first current image in the current inspection image set, calculate the center distances between the center longitude and latitude of the first current image and each previous image in the historical inspection image set in turn. If the minimum value of the center distances is less than the center threshold, the previous image corresponding to the minimum value of the center distance is used as the center matching image of the first current image, and its position i in the historical inspection image set is recorded. If the minimum value of the center distance of the first current image is not less than the center threshold, the first current image does not have a center matching image. Continue to use the subsequent current images in the current inspection image set as the first current image and execute the above steps until the position i is determined.

[0060] For the subsequent current images in the current inspection image set after determining position i, the steps for determining the center matching image are as follows: determine a number of adjacent previous images in the front and back directions of position i as the first image set. The subsequent current images are traversed in the first image set, and the center distances between the center longitudes and latitudes of the subsequent current images and the previous images in the first image set are calculated in sequence. If the minimum value in the center distance is less than the center threshold, the previous image corresponding to the minimum value of the center distance is used as the center matching image, and the position i is updated according to the position of the center matching image in the historical inspection image set. If the minimum value in the center distance is not less than the center threshold, there is no center matching image for this subsequent current image, and the above determination steps are continued for the remaining subsequent current images, and the position i at this time is used.

[0061] The above determination steps are performed on subsequent current images in sequence until all current images in the current inspection image set have been matched with the central regions.

[0062] Specifically, the following method can also be used when performing the direction matching in step 302. First, the longitude and latitude coordinates of the four vertices of the current image in the current inspection image set are obtained, and the longitude and latitude coordinates of the four vertices of all previous images in the historical inspection image set are obtained. Based on the longitude and latitude coordinates of the four vertices of the current image / previous image, the four vertices are combined in pairs in the order of top, bottom, left, and right to determine four edge segments. That is, the longitude and latitude coordinates of the upper left corner and the upper right corner can determine the upper edge segment. The edge segments in the other three directions are determined in the same manner and are not described in detail here.

[0063] According to the position i determined when the center area matching is performed on the last current image, a number of adjacent previous images are determined in the two directions before and after the position i as the second image set. The edge distances between the four edge segments of the current image are calculated in the second data set. If the minimum value of the edge distance in a certain direction of the current image is less than the edge preset value, the direction matching image of the current image in that direction is obtained, and the position i is updated according to the position of the direction matching image in the historical inspection image set. The remaining current images in the current inspection image set determine the corresponding direction matching images in the four directions in turn according to the above method, and update the position i. If the minimum value of the edge distance in a certain direction of the current image is not less than the edge preset value, the corresponding position i is not updated. The above method is continued for subsequent current images until all current images in the current inspection image set have been direction matched.

[0064] Step 303: Obtain a matching image set for each current image in the current inspection image set based on the center matching image and the direction matching image. In the embodiment of the present application, the center matching image and the direction matching image of each current image are stored in the matching image set.

[0065] By combining the central longitude and latitude with the edge segments, the current image is matched with directional matching images in the four directions of up, down, left and right, as well as the center matching image of the center area in the historical inspection image set, which can reduce the missed detection rate while ensuring the integrity of the detection area.

[0066] Step 102: Extract the overlapping area in the current inspection image set and its corresponding matching image set to determine the aligned image. The matching point pair includes the current matching point and the previous matching point. In an embodiment of the present application, the current image in the current inspection image set and the previous image in the corresponding matching image set are segmented to obtain the current segmented image and the previous segmented image. The current segmented image / previous segmented image is reduced, and the feature points therein are extracted for feature point matching to obtain the matching point pair. The matching point pair includes the current matching point and the previous matching point. The current matching point / previous matching point is restored to the current segmented image / previous segmented image, and the current matching point / previous matching point in each current segmented image / previous segmented image is merged. The overlapping area is determined based on the merged current matching point and the previous matching point to obtain the aligned image.

[0067] Determining an overlapping area based on the merged current matching point and the previous matching point includes: determining a transformation matrix based on the merged current matching point and the previous matching point; determining a current perspective corner point using the transformation matrix and the longitude and latitude coordinates of four vertices of the current image; and determining a current overlapping area based on the current perspective corner point. Performing a perspective transformation on the previous image using the transformation matrix to obtain the previous overlapping area. Determining a maximum inscribed rectangle within the current overlapping area based on the current perspective corner point. Using the maximum inscribed rectangle, the current overlapping area and the previous overlapping area are respectively intercepted.

[0068] Specifically, after center region matching and orientation matching, the current image and the corresponding previous image are essentially in the same position. However, because the pose information of the drone and pod camera cannot be consistent from moment to moment, the collected inspection images have non-overlapping areas and the overlapping areas are not aligned, making it impossible to directly perform change detection using the time series change detection algorithm.

[0069] If the inspection image collected is a high-resolution image, the number of feature points extracted is fixed, and the feature points after matching tend to be concentrated in a certain area, resulting in a smaller overlapping area or extraction errors. In order to ensure that the area of ​​the aligned overlapping area is the largest, the feature points must be correctly matched and evenly distributed. Therefore, this application adopts the method of extracting feature points after segmenting the inspection image and merging them after matching to prevent the matching feature points from gathering in a certain area and ensure the accuracy of the overlapping area. In this process, in order to speed up the calculation speed, the image is reduced.

[0070] Specific as Figure 6As shown, the current image and the corresponding previous image in the matching image set are respectively divided into n segmented images. n is determined according to actual needs, and it is required that n should be an integer after square root as much as possible to facilitate image segmentation and calculation. Exemplarily, n is set to 4, so the width and height of the segmented image after segmentation are half of the width and height of the current image / previous image. The segmented images are successively reduced, exemplarily reduced to half of the original, that is, the width and height of the reduced segmented image are one quarter of the width and height of the current image / previous image. The feature points of the reduced segmented image j are detected and extracted in sequence, and the feature points of the current image are matched with the feature points of the previous image in its matching image set to obtain matching point pairs. All segmented images are traversed in sequence to obtain matching point pairs of all segmented images. Among them, the matching point pair includes the current matching point and the previous matching point. As shown Figure 8 As shown, the left half is the previous image and the right half is the current image.

[0071] The current matching point / previous matching point is restored to the current segmented image / previous segmented image before reduction, and then the current matching point / previous matching point in each current segmented image / previous segmented image is merged. The transformation matrix is ​​determined based on the merged current matching point and the previous matching point.

[0072] Specifically, the calculation method of the transformation matrix H is as follows:

[0073] .

[0074] Where, Represents the preorder matching point, Represents the current matching point. For example, the transformation matrix is ​​as follows:

[0075] .

[0076] Where H represents the transformation matrix, 、 、 、 、 、 represents the rotation between the matching point pairs, , , Indicates the offset between pairs of matching points.

[0077] Determine the four current perspective corners based on the four vertices of the current image, and determine the current overlapping area based on the four current perspective corners. Then determine a maximum inscribed rectangle within the current overlapping area. The preferred embodiment is a maximum inscribed regular rectangle. Then, multiply the previous image by the inverse transformation matrix to obtain the previous overlapping area. Finally, use the maximum inscribed rectangle to intercept the current overlapping area and the previous overlapping area respectively to obtain the aligned image. Figure 9 and Figure 10 Shown are the current aligned image and the previous aligned image respectively.

[0078] Step 103: Use the trained change detection model to perform time-series change detection on the aligned images to obtain a change detection result for the current image in the current inspection image set. In this embodiment of the present application, time-series change detection is performed on the aligned images of the template size to obtain a black-and-white binary image. The black-and-white binary image of the template size is restored to the original size of the aligned images. The original-size black-and-white binary image is superimposed with the corresponding aligned image to obtain a global grayscale image. The global grayscale image is binarized to obtain a change detection result for the current image.

[0079] Specifically, for high-resolution images (such as 9552 6368, unit: pixel), before performing change detection, the high-resolution image needs to be cut to ensure that the method of the present application can operate normally when the video memory is small. After the overlapping area is extracted and the aligned image is obtained, the size of the aligned image is fixed, and the length and height are equal, and the length and height are integer multiples of 32. This is to prevent the aligned image from being compressed at an indefinite proportion when change detection is performed directly, resulting in target deformation or large feature differences, which affects the accuracy of the change detection results. At the same time, the information of the aligned image is lost, affecting the detection accuracy. In addition, due to hardware constraints, if the image size of the change detection is too large, it will lead to increased time consumption or insufficient video memory to run.

[0080] Therefore, before executing step 103, it is also possible to determine whether the original size of the aligned image is larger than the set template size. If the original size of the aligned image is larger than the template size, the aligned image is segmented into several segmented images of the template size. If the width and / or height of the remaining image after segmentation is less than the template size, it is padded within the aligned image and then segmented into segmented images. If the original size of the aligned image is smaller than the template size, the aligned image is padded to the template size to obtain a padded aligned image. Furthermore, the segmented / padded aligned image can be color-inverted.

[0081] The alignment images include the current alignment image and the previous alignment image. The template size is exemplarily set to 512 512, unit: pixel.

[0082] Specifically, a determination is made as to whether the original size of the aligned image is larger than the set template size. If the original size of the aligned image is larger than the template size, the aligned image is segmented into several segments of the template size. If the width and / or height of the remaining image after segmentation is less than the template size, it is padded within the aligned image and then segmented into segments. If the original size of the aligned image is smaller than the template size, the aligned image is padded to the template size to obtain the padded aligned image.

[0083] Furthermore, if Figure 7 As shown in the figure, the entire black shadow part is the alignment image, and the red solid line frame is the non-overlapping segmented image divided according to the size of the template. The black dotted frame at the bottom and right is the segmented image with overlapping areas when the remaining part of the alignment image is not enough to split the template size and moves up / left to complete the segmentation template size. If the size of the alignment image is smaller than the template size, the top, bottom, left and right of the alignment image are evenly filled with black until the filled alignment image is equal to the size of the template. That is, fill the top and bottom of the alignment image with black. High black fill on the left and right sides of the aligned image Width of black. and Indicates the width and height of the template size, and Indicates the width and height of the original size of the aligned image.

[0084] Perform inversion processing on the aligned image after segmentation / filling. Specifically, the color of the aligned image after segmentation / filling is split into three channels: R, G, and B (red, green, and blue). Subtract the color values ​​of the three channels from 255 to obtain the color difference of the three channels. Then, the color difference of the three channels is merged to achieve inversion processing. Figure 11 and Figure 12 A set of example images before and after the color inversion process is performed on the aligned images.

[0085] In this embodiment of the present application, temporal change detection is performed on the aligned images at their original size to obtain a black-and-white binary image. The black-and-white binary image at the template size is restored to the original size of the aligned images. The original-size black-and-white binary image is superimposed with the corresponding aligned images to obtain a global grayscale image. The global grayscale image is binarized to obtain the change detection result for the current image.

[0086] Specifically, the temporal change detection algorithm is used to detect temporal changes in the segmented / filled aligned images, and the detection result is a black and white binary image, such as Figure 13Specifically, the black-and-white binary image of the template size is restored to the original size of the aligned image. Specifically, the black-and-white binary image of the segmented aligned image (i.e., the segmented image) is reassembled and restored to its original size. The black borders surrounding the filled aligned image are removed to obtain a black-and-white binary image of the original size. The original-size black-and-white binary image is then superimposed with the original-size aligned image to obtain a global grayscale image. This global grayscale image is then binarized to obtain the change detection result for the current image.

[0087] In an embodiment of the present application, after obtaining a change detection result for a current image in the current inspection image set, at least a portion of the change detection result may be added to a sample library of a change detection model for updating the change detection model. The inspection image to be tested is input into the updated change detection model to obtain a change detection result for the inspection image to be tested.

[0088] In an embodiment of the present application, before adding at least part of the change detection results to the sample library of the change detection model for use in updating the change detection model, the change detection results may be converted into labeled samples. The labeled samples are screened to remove falsely detected contours, supplement missed contours, and adjust the marking frame until it fits the contours in the labeled sample. Converting the change detection results into labeled samples includes: storing the change detection results in a set image format to obtain a change detection image. Extracting contours from the change detection image, and writing the extracted contours into a file to store them as labeled samples.

[0089] Specifically, the set image format includes ".bmp" (an image format) or ".png" (an image format). Contours in the change detection image, i.e., the contours of the white portion, are extracted and saved in a file in ".json" (a text file written in JSON format) format to obtain a labeled sample. The number of extracted contours in the labeled sample is counted, and the number of contours in the labeled sample is marked. If the number of contours is 0, "null" (commonly used in coding languages ​​to represent "empty") is used as the labeling result of the labeled sample when storing the labeled sample. The labeled sample is then opened with a labeling tool for screening, and falsely detected contours are manually removed, missed contours are supplemented, and the marking frame is adjusted until it fits the contour.

[0090] Then, the change detection model is trained and updated by combining the labeled samples with the historical labeled samples. The historical labeled samples here can be purely manually labeled samples or labeled samples obtained by the output processing of the previous rounds of the method of this application.

[0091] Those skilled in the art should be aware that if labeled samples converted from change detection results are directly combined with historical labeled samples to train and update the change detection model, the training time of the change detection model will increase. The change detection model should be updated periodically based on actual scenarios. If labeled samples are added to historical labeled samples without restriction, the number of samples will increase, and the training time of the change detection model will also increase, affecting the efficiency of the change detection model update. Therefore, when updating the change detection model, labeled samples and historical labeled samples can be added or deleted according to actual needs before being used for training and updating.

[0092] In the embodiment of the present application, the inspection image to be tested may also be input into the updated change detection model to obtain a change detection result of the inspection image to be tested.

[0093] The method of this application can detect changed areas in inspection images above pipelines based on image features. For inspection images of new scenes, sample labeling and model training and updating can be quickly achieved, making application in new scenes more convenient and efficient.

[0094] Furthermore, existing change detection methods pose high data security risks during the generation and distribution of map tiles, making them unsuitable for pipeline inspections in sensitive areas. However, the method proposed in this application uses high-resolution images for change detection after image segmentation and then stitches the detection results together, enabling offline change detection even with limited device performance.

[0095] Although this application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in this embodiment is only one way of executing the steps among many, and does not represent the only execution order. When an actual device or client product executes, the method shown in this embodiment or the accompanying drawings may be executed sequentially or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0096] like Figure 14 As shown, the embodiment of the present application further provides an image change detection device 1400 based on feature matching. The device includes: a matching module 1401, an extraction module 1402 and a detection module 1403, as follows.

[0097] The matching module 1401 is configured to perform image matching on a historical inspection image set of a previous period using multiple regions of each current image in the current inspection image set of the current period to obtain a matching image set.

[0098] The extraction module 1402 is used to extract the overlapping area in the current inspection image set and its corresponding matching image set to determine an aligned image; wherein the aligned image includes the current aligned image and the previous aligned image.

[0099] The detection module 1403 is configured to perform time series change detection on the aligned images using the trained change detection model to obtain a change detection result of the current image in the current inspection image set.

[0100] Some modules in the apparatus described herein may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0101] The devices or modules described in the above application embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function in various modules. When implementing the embodiments of this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0102] The methods, devices, or modules described herein can be implemented in the form of computer-readable program code. The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the memory control logic. Those skilled in the art will also appreciate that, in addition to implementing the controller in pure computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for realizing various functions may be considered to be both a software module for realizing the method and a structure within a hardware component.

[0103] An embodiment of the present application further provides a device comprising: a processor; a memory for storing processor-executable instructions; and when the processor executes the executable instructions, the method described in the embodiment of the present application is implemented.

[0104] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist independently, or two or more modules may be integrated into one module.

[0105] The above-mentioned storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. Such memory can be used to store computer program instructions.

[0106] Through the description of the above implementation methods, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, or can be embodied through the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments.

[0107] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0108] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A method for detecting image changes based on feature matching, characterized in that: include: Performing image matching on a plurality of areas of each current image in a current inspection image set of a current time period with a historical inspection image set of a previous time period to obtain a matching image set, including: performing center area matching on the current image in the current inspection image set with a preceding image in the historical inspection image set to obtain a center matching image of the current image; performing direction matching on the current image in the current inspection image set with a preceding image in the historical inspection image set to obtain a direction matching image of the current image; obtaining the matching image set of each current image in the current inspection image set based on the center matching image and the direction matching image; wherein the matching image set of each current image includes a center matching image and a direction matching image of the current image; Extracting overlapping areas between the current inspection image set and its corresponding matching image set to determine an aligned image; wherein the aligned image includes a current aligned image and a previous aligned image; The trained change detection model is used to perform time series change detection on the aligned images to obtain a change detection result of the current image in the current inspection image set.

2. The method according to claim 1, characterized in that The performing center area matching of the current image in the current inspection image set with the previous images in the historical inspection image set to obtain a center matching image of the current image includes: Determining the center longitude and latitude of the current image in the current inspection image set and the previous image in the historical inspection image set; respectively determining the center distances between the center longitude and latitude of the current image and the center longitude and latitude of at least part of the previous images in the historical inspection image set; determining a minimum value of the center distance of the current image; If the minimum value of the center distance is less than the center threshold, the previous image corresponding to the minimum value of the center distance is used as the center matching image of the current image.

3. The method according to claim 1, characterized in that The step of performing direction matching on the current image in the current inspection image set and the previous images in the historical inspection image set to obtain the direction matching image of the current image includes: Determining the longitude and latitude coordinates of four vertices of the current image / previous image based on the image field of view and the center longitude and latitude of the current image / previous image; Determining edge segments of four sides of the current image / previous image according to the longitude and latitude coordinates of the four vertices of the current image / previous image; respectively determining edge distances between each edge segment of the current image and edge segments corresponding to at least some of the previous images in the historical inspection image set; respectively determining minimum values ​​of distances between the plurality of edges of the current image; If the minimum value of the edge distance is less than the edge preset value, the previous image corresponding to the minimum value of the edge distance is used as the direction matching image of the edge corresponding to the current image.

4. The method according to claim 1, wherein The extracting the overlapping area between the current inspection image set and the corresponding matching image set to determine the aligned image includes: Segmenting a current image in the current inspection image set and a corresponding previous image in the matching image set to obtain a current segmented image and a previous segmented image; Reducing the current segmented image / previous segmented image, extracting feature points therein, performing feature point matching, and obtaining a matching point pair; wherein the matching point pair includes the current matching point and the previous matching point; Restoring the current matching point / previous matching point to the current segmented image / previous segmented image, and merging the current matching point / previous matching point in each of the current segmented image / previous segmented image; An overlapping area is determined according to the merged current matching point and the previous matching point to obtain the aligned image.

5. The method according to claim 1, characterized in that Before performing time series change detection on the aligned images using the trained change detection model, the method further includes: Determining whether the original size of the aligned image is larger than the set template size; If the original size of the aligned image is larger than the template size, the aligned image is segmented into several segmented images of the template size. If the width and / or height of the remaining image after segmentation is less than the template size, it is padded within the range of the aligned image and then segmented into the segmented images; If the original size of the aligned image is smaller than the template size, the aligned image is padded to the template size to obtain a padded aligned image.

6. The method according to claim 5, characterized in that The performing time series change detection on the aligned images using the trained change detection model to obtain a change detection result of the current image in the current inspection image set includes: Performing time series change detection on the aligned images of the template size to obtain a black and white binary image; Restoring the black and white binary image of the template size to the original size of the aligned image; Superimposing the black and white binary image of the original size with the corresponding aligned image to obtain a global grayscale image; The global grayscale image is binarized to obtain a change detection result of the current image.

7. The method according to claim 1, characterized in that After obtaining the change detection result of the current image in the current inspection image set, the method further includes: adding at least part of the change detection results to a sample library of the change detection model for use in updating the change detection model; The inspection image to be tested is input into the updated change detection model to obtain a change detection result of the inspection image to be tested.

8. The method according to claim 7, characterized in that Before adding at least part of the change detection results to the sample library of the change detection model, the method further includes: Converting the change detection results into labeled samples; Screening the marked sample, removing falsely detected contours, supplementing missed contours, and adjusting the marking frame until it fits the contours in the marked sample; The step of converting the change detection result into a labeled sample includes: Storing the change detection result in a set image format to obtain a change detection image; Contours in the change detection image are extracted, and the extracted contours are written into a file and stored as labeled samples.

9. An image change detection device based on feature matching, characterized in that: include: A matching module is used to use multiple areas of each current image in the current inspection image set of the current time period to perform image matching with the historical inspection image set of the previous time period to obtain a matching image set, including: performing center area matching on the current image in the current inspection image set with the previous image in the historical inspection image set to obtain a center matching image of the current image; performing direction matching on the current image in the current inspection image set with the previous image in the historical inspection image set to obtain a direction matching image of the current image; obtaining the matching image set of each current image in the current inspection image set based on the center matching image and the direction matching image; wherein the matching image set of each current image includes a center matching image and a direction matching image of the current image; An extraction module, configured to extract an overlapping area between the current inspection image set and its corresponding matching image set to determine an aligned image; wherein the aligned image includes a current aligned image and a previous aligned image; The detection module is used to perform time series change detection on the aligned images using the trained change detection model to obtain a change detection result of the current image in the current inspection image set.

Images